Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 86 additions & 2 deletions .github/workflows/gtfs-capture.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,16 @@ permissions:
jobs:
inspect:
runs-on: ubuntu-latest
outputs:
sha256: ${{ steps.acquire.outputs.sha256 }}
retrieved-at: ${{ steps.acquire.outputs.retrieved-at }}
independent-outcome: ${{ steps.validate.outcome }}
inspection-outcome: ${{ steps.report.outcome }}
artifact-url: ${{ steps.artifact.outputs.artifact-url }}
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: actions/setup-node@v6
with:
node-version: 24
Expand All @@ -30,7 +38,10 @@ jobs:
GTFS_SNAPSHOT_BASE_URL: ${{ vars.GTFS_SNAPSHOT_BASE_URL }}
GTFS_CAPTURE_KEY: ${{ github.event.client_payload.capture_key || inputs.capture_key }}
GTFS_INPUT_DIR: ${{ runner.temp }}/gtfs-input
run: node scripts/acquire-gtfs-capture.mjs
run: |
node scripts/acquire-gtfs-capture.mjs
jq -r '"sha256=\(.sha256)", "retrieved-at=\(.retrieved_at)"' \
"$GTFS_INPUT_DIR/capture.json" >> "$GITHUB_OUTPUT"
- uses: actions/setup-java@v5
with:
distribution: temurin
Expand All @@ -40,24 +51,97 @@ jobs:
url=$(node -p "JSON.parse(require('fs').readFileSync('scripts/gtfs-validator-lock.json')).url")
curl --fail --location --max-time 120 "$url" -o "$RUNNER_TEMP/gtfs-validator.jar"
- name: Independently validate verified snapshot
id: validate
if: ${{ !cancelled() && steps.acquire.outcome == 'success' }}
run: >-
node scripts/validate-gtfs-independent.mjs
"$RUNNER_TEMP/gtfs-validator.jar"
"$RUNNER_TEMP/gtfs-input/schedule.zip"
"$RUNNER_TEMP/gtfs-independent"
- name: Inspect verified local snapshot
id: report
if: ${{ !cancelled() && steps.acquire.outcome == 'success' }}
run: >-
node packages/cli/dist/index.js gtfs inspect
--file "$RUNNER_TEMP/gtfs-input/schedule.zip"
--capture "$RUNNER_TEMP/gtfs-input/capture.json"
--write "$RUNNER_TEMP/gtfs-report.json"
- uses: actions/upload-artifact@v7
id: artifact
if: always()
with:
name: gtfs-inspection-${{ github.run_id }}
# Reruns keep their own reports and preserve links from earlier attempts.
name: gtfs-inspection-${{ github.run_id }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/gtfs-report.json
${{ runner.temp }}/gtfs-independent/
if-no-files-found: ignore

review:
needs: inspect
if: >-
${{ !cancelled() && needs.inspect.outputs.sha256 != '' &&
github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
concurrency:
group: gtfs-review-${{ needs.inspect.outputs.sha256 }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.event.repository.default_branch }}
- name: Open snapshot review
env:
GH_TOKEN: ${{ github.token }}
BASE_BRANCH: ${{ github.event.repository.default_branch }}
SNAPSHOT_ORIGIN: ${{ vars.GTFS_SNAPSHOT_BASE_URL }}
SNAPSHOT_SHA256: ${{ needs.inspect.outputs.sha256 }}
RETRIEVED_AT: ${{ needs.inspect.outputs.retrieved-at }}
INDEPENDENT_OUTCOME: ${{ needs.inspect.outputs.independent-outcome }}
INSPECTION_OUTCOME: ${{ needs.inspect.outputs.inspection-outcome }}
ARTIFACT_URL: ${{ needs.inspect.outputs.artifact-url }}
run: |
review_branch="gtfs-review/$SNAPSHOT_SHA256"
review_ref="refs/heads/$review_branch"
existing_pr=$(gh api --method GET "repos/$GITHUB_REPOSITORY/pulls" \
-f state=all -f head="$GITHUB_REPOSITORY_OWNER:$review_branch" \
--jq '.[0].html_url // empty')
if [[ -n "$existing_pr" ]]; then
echo "Snapshot review already exists at $existing_pr"
exit 0
fi

title="chore(gtfs): review schedule snapshot ${SNAPSHOT_SHA256:0:12}"
# A previous attempt may have pushed the branch before PR creation failed.
existing_ref=$(git ls-remote --refs origin "$review_ref")
if [[ -z "$existing_ref" ]]; then
run_url="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
origin="${SNAPSHOT_ORIGIN%/}"
record_path="docs/gtfs-captures/$SNAPSHOT_SHA256.md"
mkdir -p docs/gtfs-captures
cat > "$record_path" <<EOF
# LTA train GTFS snapshot

- Retrieved at $RETRIEVED_AT
- ZIP SHA-256 \`$SNAPSHOT_SHA256\`
- [Capture receipt]($origin/captures/schedule/$SNAPSHOT_SHA256.json) and [source ZIP]($origin/blobs/schedule/$SNAPSHOT_SHA256.zip)
- Independent validator result \`$INDEPENDENT_OUTCOME\`
- MRTDown inspection result \`$INSPECTION_OUTCOME\`
- [GitHub Actions run]($run_url)
EOF
if [[ -n "$ARTIFACT_URL" ]]; then
printf '%s\n' "- [Download available reports]($ARTIFACT_URL)" >> "$record_path"
fi
git add -- "$record_path"
git -c user.name='github-actions[bot]' \
-c user.email='41898282+github-actions[bot]@users.noreply.github.com' \
commit -m "$title" -m 'Co-Authored-By: Codex <noreply@openai.com>'
git push --force-with-lease="$review_ref:" origin "HEAD:$review_ref"
fi

gh pr create --repo "$GITHUB_REPOSITORY" --base "$BASE_BRANCH" \
--head "$review_branch" --title "$title" \
--body "Review the new LTA train GTFS capture. The added record links the source ZIP and inspection results. Check validation findings, service-date coverage, and mapping changes before preparing a timetable release."
21 changes: 13 additions & 8 deletions docs/gtfs-public-handoff.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ Only schedule hashes trigger GitHub `repository_dispatch` event `gtfs_capture`:
The crawler records notification state only in private Tigris. Failed notifications
remain pending and are retried on subsequent enabled DataMall polling runs.
A bounded batch of 20 avoids unbounded catch-up. Successful hashes are not
notified again. A crash after dispatch can cause duplicate delivery; the receiver
only inspects and uploads a report, so replay performs no canonical writes.
notified again. A crash after dispatch can cause duplicate delivery. The
receiver may repeat inspection, but opens at most one review PR per ZIP hash.
Published alerts/trip updates do not dispatch to the schedule workflow.

## Provisioning
Expand All @@ -75,12 +75,17 @@ then downloads the ZIP without following redirects. Transfers have byte limits
and request timeouts. Both files are hash/size verified before materialization
in a fresh temporary directory.

Acquisition is in the CLI layer, separate from the offline `@mrtdown/fs`
inspector. The workflow invokes that inspector and retains its JSON report as
a GitHub Actions artifact, including reports with structural errors. It has
read-only repository permissions. It does not commit, merge, run models or
publish a scheduled-arrivals artifact. Trusted delivery does not replace
independent GTFS validation or reviewed canonical mappings.
The read-only inspection job runs the independent validator and the offline
`@mrtdown/fs` inspector and uploads their available reports as a GitHub Actions
artifact, including findings with errors.

After the receipt and ZIP are verified, a separate job on the default branch
opens a review PR for that ZIP hash even if validation reports errors. The PR
adds `docs/gtfs-captures/<sha256>.md` with the capture time, source links,
validation outcomes, and links to the run and available reports. Existing PRs,
including closed ones, are reused on replay. The job uses the repository's
GitHub Actions permission to create PRs. Canonical mapping review and timetable
release approval remain separate steps.

For a local smoke test after publication:

Expand Down
Loading