Skip to content

feat(pipeline): replace run_daily.sh with local control-plane - #2590

Merged
twoimo merged 1 commit into
datafrom
develop
Aug 17, 2026
Merged

twoimo merged 1 commit into
datafrom
develop

Conversation

@twoimo

@twoimo twoimo commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Summary

  • Replace backend/run_daily.sh / backend/run_local_heavy.sh as operator and CI entrypoints with python3 -m backend.pipeline_control.worker.
  • Add local-first async control-plane API + worker. Jobs/locks/audit live in local schema pipeline_control plus a process-shared FileStore. Overlay compose has no Postgres.
  • Admin BFF and dashboard talk to the control-plane API. Grafana iframe stays out.
  • Legacy shims remain only as regression fixtures under backend/utils/tests/fixtures/*.legacy.

Promotion path

Serialized PR: develop -> data. Do not merge without external approval and branch-protection checks.

This source tree does not claim hosted pipeline_control was applied. Hosted project writes remain forbidden.

Test plan

  • python -m unittest backend.pipeline_control.tests.test_slice0_control_plane backend.pipeline_control.tests.test_manifest_parity backend.pipeline_control.tests.test_live_and_cutover (31 OK)
  • CI on this PR
  • External review before merge to data

Operator and CI entry is now python3 -m backend.pipeline_control.worker.
Jobs, locks, and audit live in local schema pipeline_control plus a
process-shared FileStore; overlay compose has no Postgres. Admin BFF
and dashboard talk to the API, not Grafana. Legacy shims stay only as
regression fixtures.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@vercel

vercel Bot commented Aug 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
tzudong Error Error Aug 17, 2026 6:13pm

handler.send_response(status)
handler.send_header("Content-Type", "application/json")
handler.send_header("Cache-Control", "no-store")
handler.send_header("X-Request-Id", request_id)
@gitguardian

gitguardian Bot commented Aug 17, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
33514022 Triggered Generic Password e6c7f2a .github/workflows/daily-crawler.yml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@twoimo
twoimo merged commit 53eecd0 into data Aug 17, 2026
23 of 36 checks passed

This branch had an error being deployed

1 failed deployment
Preview — e6c7f2a6 Deployed Aug 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants