Skip to content

feat: migrate crawler to Fly and Tigris - #28

Merged
duncanleo merged 4 commits into
mainfrom
codex/migrate-crawler-to-fly
Oct 14, 2026
Merged

duncanleo merged 4 commits into
mainfrom
codex/migrate-crawler-to-fly

Conversation

@duncanleo

Copy link
Copy Markdown
Member

Move crawler execution from Cloudflare Workers to a Node service on Fly, with independently signed QStash jobs for five-minute evidence/realtime collection and daily GTFS schedule capture. Use separate Tigris S3 clients and credentials for private and public snapshots while preserving object keys and GitHub dispatch contracts.

Add the production container, gated GitHub Actions deployment, deterministic schedule provisioning, and operator setup/cutover instructions. Cloudflare Builds has been disabled by the operator; initial Fly deployment keeps jobs disabled until health checks pass.

Validation: 56 deterministic tests, type checking, lint, Node build, Fly configuration validation, Docker build, and container smoke checks (health 200, unsigned job 401, signed disabled job 503). Production acquisition and scheduling are verified during cutover.

duncanleo and others added 4 commits October 12, 2026 22:16
Co-Authored-By: Codex <noreply@openai.com>
Replace scheduled Worker execution with independent QStash job endpoints and preserve evidence and GTFS contracts.

Co-Authored-By: Codex <noreply@openai.com>
Add a scale-to-zero container, disabled deployment gates, and operator setup and cutover instructions.

Co-Authored-By: Codex <noreply@openai.com>
Use separate authenticated S3 clients for private and public snapshots and document R2 data and domain cutover.

Co-Authored-By: Codex <noreply@openai.com>
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
mrtdown-data-crawler a6dacd8 Oct 14 2026, 06:32 AM

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-14T06:36:08.156979Z a6dacd8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@duncanleo
duncanleo merged commit 7f9f861 into main Oct 14, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant