feat(site-crawl): find broken routes and SEO issues after deployments

[dviejokfs]

Successful deployments currently provide no route-level evidence of broken links or SEO regressions. This adds Site Crawl: a native TypeScript plugin with an administrator-only Temps sidebar UI, persistent reports, and automatic crawling after successful deployments.

Production-only automation is enabled by default once an administrator grants events_read. Administrators can include other environments, exclude projects, pause automation, clear queued work, and cancel active crawls. Reports identify the project, environment, and deployment that triggered them.

Behavior

• Crawl same-origin links and sitemaps, respect robots.txt/nofollow, and retain referring-page evidence for broken routes, redirects, and metadata issues.
• Persist a bounded 20-job queue, suppress the last 200 duplicate deployment identities, and run one crawl at a time. Retain 30 reports with JSON export.
• Recheck live Events read permission before dequeueing. Revocation pauses queued work; active crawls require separate cancellation.
• Enforce public-IP DNS validation/pinning, same-origin redirects, standard ports, request/response/page limits, and a 20-minute deadline. Native plugins still execute with the host OS account's permissions and are not sandboxed.
• Add a dedicated CI job; no changes to the host feature or ESLint worktrees.

Verification

• bun run check: passed.
• bun test src: 44 passed, 146 assertions.
• Compiled native plugin test exercises protocol-v2 handshake, signed Unix-socket channel, admin/reader/spoofed caller access, embedded UI, capability discovery, duplicate deployment events, deployment-attributed report creation, and private-address rejection.
• Automation tests cover successful crawls using mock HTTP responses, production/preview filtering, project exclusions, persistence/restart, denied event permission, queue bounds, settings pruning, and serialized requests.
• bun run build: passed. bun audit: no vulnerabilities across 170 packages.
• Browser: real public example.com crawl completed; settings saved across reload; mobile 390px layout had no horizontal overflow. Desktop and mobile inspected.
• Independent security review: PASS, no blocking findings.

Remaining limits

This has not been installed and triggered by a real deployment through a full Temps instance. Deployment integration evidence uses the compiled plugin with a local host protocol harness; successful automatic crawl content uses HTTP mocks. The separate browser crawl used a real public URL. No AI provider calls are used.

This PR does not publish a registry entry or release binary. The self-contained plugin folder can be published through the normal reviewed installation flow. Older hosts without permission discovery support manual crawls only. Crawls inspect server HTML, not JavaScript-rendered pages, and inspect the live URL at crawl time rather than an immutable deployment snapshot. Private/authenticated sites and external link checking are outside this version.

Follow-up verification

• Integrated the actual Temps design-system source components as a portable, attributed snapshot, with shared Radix controls, tables, tabs, dialog confirmations, and light/dark tokens.
• Verified 390px, 768px, and 1440px layouts, light/dark themes, report filters, and cancellation of a deletion confirmation without deleting user data.
• The reported CLI page returned HTTP 200 and 1,343,812 bytes. After raising the bounded response cap to 8 MiB, a real browser-triggered one-page crawl completed with HTTP 200, zero route errors, and zero SEO warnings. Oversized responses now retain HTTP status and receive an incomplete-inspection warning.
• Repeated independent security review: PASS. Audited 170 packages with no vulnerabilities; Vite and esbuild are pinned to patched releases.

Tokenizer migration

Replaced Cheerio for HTML and sitemap parsing with bounded htmlparser2 callbacks. Added seven focused regression tests for metadata/entities, ignored subtrees, robots/nofollow, malformed HTML, field/link caps, sitemap namespaces, and excessive nesting. HTTP buffering stays bounded to 8 MiB; parsing does not construct a DOM. Real CLI documentation returned HTTP 200, 1,343,812 bytes, the correct title and 129 unique links. Final security review PASS after adding HTML/XML depth limits. Full suite: 44 passed, 146 assertions; native build/typecheck/audit passed.