Skip to content

feat(crawler): hybrid crawling mode and publisher digest fix - #2556

Merged
twoimo merged 1 commit into
datafrom
develop
Aug 16, 2026
Merged

twoimo merged 1 commit into
datafrom
develop

Conversation

@twoimo

@twoimo twoimo commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Separates heavy compute (local) from lightweight data ops (Actions), and fixes publisher artifact digest validation.

…ght Actions

Separate crawler pipeline into two modes:
- GitHub Actions (lightweight): URL collection, metadata, transcript enrichment,
  and publication artifact generation. Skips frame extraction and Chunk Multimodal
  via RUN_DAILY_SKIP_HEAVY_COMPUTE=true.
- Local (heavy-compute): Runs full pipeline or heavy steps via backend/run_local_heavy.sh,
  committing data directly to the shared data branch.

Fixes artifact-digest output format (prepend sha256: prefix) so publisher validation succeeds.
@vercel

vercel Bot commented Aug 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
tzudong Ready Ready Preview Aug 16, 2026 3:43pm

@twoimo
twoimo merged commit 064e7f1 into data Aug 16, 2026
19 of 20 checks passed
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

This branch was successfully deployed

1 active deployment
Preview — 039a861c Deployed Aug 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant