# web-23signs `web.23signs.org`: web2md as a service for agents. Any public page as clean markdown, a JSON block tree or semantic HTML; search over papers (arXiv, Crossref), news (GDELT) and Wikipedia; and change notifications delivered in-band with HTTP Events Query (draft-gupta-httpapi-events-query). Modelled on tinyfish.ai's Fetch, Search and Monitor APIs, without the account: no keys, free while in beta, 60 requests a minute per address. Live at `https://web.23signs.org` (Cloudflare Worker on the Free plan, custom domain in `wrangler.toml`). Stateless: each request is processed on demand, nothing is stored beyond a one-day edge cache. A watch is an open request; when the client disconnects, it ends. The web page is one minimal landing (heading, tagline, one field that fetches a URL or searches) with Watch and For agents sections below the fold, built with React, shadcn/ui and Tailwind in the 23signs look, as ytj.23signs.org. Design and requirements: `openspec/changes/web-service-v1/`. ## Architecture ``` client --GET/POST/QUERY--> src/worker.js (Hono) --fetch--> origin site | extract.js (HTML -> tree -> markdown/json/html) | fetcher.js (routes: arXiv, X, PDF via Workers AI, browser fallback) | search.js (arXiv, Crossref, GDELT, Wikipedia) | events.js (Events Query: carriers, long poll, stream loop) +-- public/ (UI built from web/, agent docs) ``` - **`src/worker.js`**: Hono app. `POST /v1/fetch`, `GET /{url}`, `QUERY /v1/fetch` and `QUERY /{url}` (page watches), `GET /v1/search`, `QUERY /v1/search` (topic watches), `GET /v1/status`, the discovery documents under `/.well-known/`, `POST /mcp`, the agent docs with `Cache-Control` and the discovery `Link` header, and content negotiation on `/`. Every JSON body carries `_links`; every error is RFC 9457 problem+json with `type` `https://web.23signs.org/errors/`. A per-IP rate limit (Workers `ratelimits` binding, 60 a minute) guards the API paths. - **`src/extract.js`**: the web2md HTML path in JavaScript. A regex tokenizer builds a small tree (HTMLRewriter's per-element callbacks alone cost about 10 ms of CPU on an ordinary page, the Free plan's whole budget), then the serializer follows web2md's `_node_to_md`: main-content container with the 30% rule, boilerplate pruning in place of readability, hidden-text markers, tables, lists, code, math as LaTeX. Also the JSON block tree and the clean-HTML renderer. HTMLRewriter is still used for CSS `include_selectors` / `exclude_selectors`, in a pre-pass that fires only on matches. - **`src/fetcher.js`**: one URL in, one result out. Identifies as `web2md/1.0`, retries 403/429 with a Chrome user agent, then a headless browser (Browser Rendering, `render: auto`). PDFs and office files go through Workers AI `toMarkdown`, whose metadata block becomes the result's title, author, date and description. arXiv prefers the HTML version; X posts use oembed; Google Patents keep the full DOM. Private, local and metadata addresses are rejected. - **`src/search.js`**: TinyFish's result shape over free indexes. arXiv and Crossref interleaved for `research_paper`, GDELT DOC 2.0 for `news` (it throttles shared Cloudflare addresses; one retry after its 5 s window), Wikipedia. `web` needs `SEARXNG_URL` or `BRAVE_API_KEY` and answers 501 without one. DuckDuckGo and Mojeek challenge datacenter addresses; Google News RSS is personal, non-commercial only. - **`src/events.js`**: Events Query. Carrier negotiation and framing from `camara` (multipart, json-seq, ndjson) plus SSE; `Events: duration=N` parsing; the single-notification long poll (Section 8) and the stream loop (Sections 9 and 10) with heartbeats every 50 s. A watch is capped at 40 checks because Workers Free allows 50 subrequests per invocation, and the interval is raised to fit. - **`src/discovery.js`**, **`src/mcp.js`**: the catalogs, cards, tool definitions and the stateless MCP server (`web_fetch`, `web_search`, `web_watch`, `web_status`), as in ytj. - **`web/`**: the frontend (Vite, React 19, TypeScript, Tailwind 4, shadcn/ui on Radix, TanStack Query). `npm run build` writes `public/index.html` and `public/assets/`; the output is committed so a deploy needs no build step. First-screen JavaScript is about 111 KB gzipped (budget 180 KB); the Watch and Agents sections are separate chunks loaded as they scroll in. Interface text in `web/src/i18n/{en,fi,sv}.json`. ## Limits and costs Workers Free: 100k requests a day, 10 ms CPU per request (ordinary pages take 9 to 20 ms and are tolerated so far; a run of 1102 errors means the Paid plan), 50 subrequests per invocation, Browser Rendering about 10 minutes a day (hence `render: never` inside watches), Workers AI `toMarkdown` free for non-image files. Nothing else is billed. ## Structure ``` src/worker.js Hono API, discovery, MCP mount, docs, root negotiation, /{url} shortcut src/extract.js HTML tokenizer and renderers (markdown, JSON blocks, clean HTML) src/fetcher.js One URL -> result: UA fallbacks, browser, documents, arXiv, X, patents src/search.js Search backends and parameter validation src/events.js Events Query: negotiation, framing, long poll, stream loop src/discovery.js _links, status document, catalogs, cards, MCP tool definitions src/mcp.js Stateless MCP server (Streamable HTTP) src/problem.js RFC 9457 helper web/ Frontend source (Vite + React + shadcn/ui), builds into public/ public/index.html Built page (from web/); public/assets/ the hashed bundles public/*.md, llms.txt, openapi.*, robots.txt, sitemap.xml, webmcp.js Agent documents test/ vitest in workerd: extractor, worker routes (fetchMock), discovery, MCP scripts/ Playwright checks for the page (check_web.py, check_a11y.py) openspec/ Spec-driven development: changes/web-service-v1 ``` ## Development ```bash npm install && (cd web && npm install) npm test # vitest, workers pool (outbound fetches mocked) cd web && npm run build # tsc + vite -> ../public npx wrangler dev -c wrangler.toml --port 8799 # serves the page; outbound fetch fails behind the corporate proxy ``` Local `wrangler dev` cannot reach origin sites through the corporate TLS proxy, so live checks run against the deployment: ```bash uv run --with playwright python -I scripts/check_web.py https://web.23signs.org uv run --with playwright python -I scripts/check_a11y.py https://web.23signs.org ``` ## Deploy From this directory only (never from `~/code/apps`), per the app-deploy skill: ```bash cd web && npm run build && cd .. env -u NODE_EXTRA_CA_CERTS -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u ALL_PROXY -u all_proxy npx wrangler deploy -c wrangler.toml ``` The API token now covers the 23signs.org zone, so the custom domain attaches without a dashboard step. Optional secrets: `SEARXNG_URL` (+ `SEARXNG_TOKEN`) or `BRAVE_API_KEY` for general web search; `WATCH_MIN_INTERVAL` is a test-only override. ## Agent discovery | Document | URL | |---|---| | llms.txt | https://web.23signs.org/llms.txt | | Skill | https://web.23signs.org/SKILL.md | | OpenAPI 3.2 (QUERY operations) | https://web.23signs.org/openapi.yaml, /openapi.json | | API catalog (RFC 9727) | https://web.23signs.org/.well-known/api-catalog | | ARD manifest | https://web.23signs.org/.well-known/ai-catalog.json, /ard.json | | MCP server, card | https://web.23signs.org/mcp, /.well-known/mcp/server-card.json | | A2A agent card | https://web.23signs.org/.well-known/agent-card.json | | Agent skills index | https://web.23signs.org/.well-known/agent-skills/index.json | | WebMCP shim | https://web.23signs.org/webmcp.js | ## Not yet - Payments: a paid tier over x402 after a daily free allowance, reusing the `mppx` setup from ytj. The allowance counter will be the first stored state (a Durable Object). - General web search backend (SearXNG on the NAS, or Brave). - YouTube and video transcripts (yt-dlp and whisper stay in local web2md). - Notifications after the client disconnects (webhooks, email): by design out of scope for a stateless service; keep a stream open from a process you control instead.