Skip to content

fix(ci): authenticate the federated externals sync — for QUOTA, not for access (#1143) - #1144

Merged
avrabe merged 1 commit into
mainfrom
fix/federated-ratelimit-1143
Sep 4, 2026
Merged

fix(ci): authenticate the federated externals sync — for QUOTA, not for access (#1143)#1144
avrabe merged 1 commit into
mainfrom
fix/federated-ratelimit-1143

Conversation

@avrabe

@avrabe avrabe commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

The federated-graph job has validated nothing since 2026-09-02

Sync externals (network — clones the 7 sibling repos)   failure
Validate the federated graph                            skipped
Non-vacuity guard — resolution must actually have run   skipped
error: IO error: git clone failed: Cloning into './.rivet/repos/gale'...
fatal: could not read Username for 'https://github.com': No such device or address

That error reads like a permissions failure and is not one

All seven siblings are public (gale, scry, loom, meld, kiln, jess, spar — all private=false), so an anonymous clone needs no credentials at all.

git only prompts for a username after the transport refuses it. GitHub returned a throttled response to an anonymous request, and on a tty-less runner the fallback to interactive auth surfaces as No such device or address.

The cause is a rate limiter on unauthenticated traffic, which bites hardest from datacenter address space — exactly where the self-hosted fleet lives. Timing fits: failures begin 2026-09-02, this repo's heaviest CI period. Independently corroborated — the same limiter hit unrelated infrastructure elsewhere in the same window, which is stronger than my inference from the log.

Fix: authenticate for quota, not access

GITHUB_TOKEN is repo-scoped and that's fine — we need the authenticated rate limit, not cross-repo permission. The url.insteadOf rewrite is applied inside the step and removed on exit rather than persisted on a shared self-hosted runner.

Two of my own claims are retracted

I said my v0.62 scoping commit broke this (a YAML commit cannot affect a git clone; the correlation was coincidental), and I told the org-wide review it was "standing-red on ~40 dangling traces-to" — that's my local rivet, a different version against a different cache. The job never reaches validation. Both came from describing a failure from memory instead of reading its step list.

What was right about the job, and stays

It is advisory, so nothing was blocked; and it fails closed — the non-vacuity guard is skipped rather than passing vacuously, which is the #1012 lesson working. What remains true is that an advisory job red on every commit since a release is indistinguishable from one that is merely noisy.

Same class as #1062: GitHub's own limits are the binding constraint, not our configuration.

Refs #1143, #1062

…or access (#1143)

The federated-graph job has validated nothing since 2026-09-02. It fails before
it validates:

  Sync externals (network — clones the 7 sibling repos)   failure
  Validate the federated graph                            skipped
  Non-vacuity guard — resolution must actually have run   skipped

  error: IO error: git clone failed: Cloning into './.rivet/repos/gale'...
  fatal: could not read Username for 'https://github.com': No such device or address

THAT ERROR READS LIKE A PERMISSIONS FAILURE AND IS NOT ONE. All seven siblings
are PUBLIC (verified: gale, scry, loom, meld, kiln, jess, spar all
private=false), so an anonymous clone needs no credentials whatsoever. git only
prompts for a username after the transport refuses it — GitHub returned a
throttled/denied response to an ANONYMOUS request, and on a tty-less runner the
fallback to interactive auth surfaces as "No such device or address".

The cause is a RATE LIMITER on unauthenticated traffic, which bites hardest from
datacenter address space — exactly where the self-hosted fleet lives. The timing
fits: failures begin 2026-09-02, during this repo's heaviest CI period (a
release wave plus ~9 PRs, all on those runners). Independently corroborated —
the same limiter hit unrelated infrastructure elsewhere in the same period,
which is stronger evidence than my inference from the log alone.

FIX: authenticate the clone even though it needs no permission. GITHUB_TOKEN is
repo-scoped and that is FINE — what we need is the authenticated rate limit, not
cross-repo access. The url.insteadOf rewrite is applied inside the step and
removed on exit rather than persisted on a shared self-hosted runner.

TWO OF MY OWN CLAIMS ABOUT THIS JOB WERE WRONG AND ARE RETRACTED IN #1132/#1143.
I said my v0.62 scoping commit broke it (a commit adding YAML artifacts cannot
affect a git clone; the correlation was coincidental), and I told the org-wide
review it was "standing-red on ~40 cross-repo dangling traces-to" — that is what
my LOCAL rivet reports, a different version against a different cache. The job
never reaches validation at all. Both came from describing a failure from memory
instead of reading its step list.

WHAT WAS RIGHT ABOUT THE JOB, AND STAYS: it is advisory, so nothing was blocked;
and it FAILS CLOSED — the non-vacuity guard is skipped rather than passing
vacuously, which is the #1012 lesson working exactly as designed. What remains
true is that an advisory job red on every commit since a release is
indistinguishable from one that is merely noisy, which is how I came to
mis-describe it twice without checking.

SAME CLASS AS THE CI-CAPACITY FINDING (#1062): GitHub's own limits are the
binding constraint, not our configuration. There the self-hosted fleet sat idle
while the GitHub-hosted quota queued for hours; here public repos are unclonable
from that same fleet.

Refs #1143, #1062

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L
@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@avrabe
avrabe merged commit cf64b67 into main Sep 4, 2026
60 checks passed
@avrabe
avrabe deleted the fix/federated-ratelimit-1143 branch September 4, 2026 06:37
avrabe added a commit that referenced this pull request Sep 4, 2026
…ommit exactly as designed (#1147)

Merging #1144 turned main red on R10:

  FAIL R10: delivery-shaped commit in the release window is attributable to NO
  release artifact: 'fix(ci): authenticate the federated externals sync ...' —
  no known artifact id or issue number in the subject, and no artifact's
  `landed:`/`verified-by:` names its PR (#1119: work landed, every artifact
  silent).

That rule shipped ten commits ago in #1124, and this is the first time it has
fired on real work. It fired on MINE, which is the right direction: I merged a
CI fix with no artifact, and the gate refused to let the release plan stay
silent about it. Fixed by creating the artifact, not by exempting the commit.

RQ-62-FEDAUTH records the finding, which is worth more than the fix:

THE ERROR MESSAGE ACTIVELY MISLEADS. "could not read Username for
https://github.com" reads as a permissions failure. All seven siblings are
PUBLIC, so an anonymous clone needs no credentials — git only prompts AFTER the
transport refuses, and a throttled anonymous response on a tty-less runner
surfaces as "No such device or address". The cause is a rate limiter on
unauthenticated traffic, hardest from datacenter address space, which is where
the self-hosted fleet lives.

THE PASSES DO NOT PROVE THE FIX, recorded so nobody later claims they did.
#1141 and #1142 passed WITHOUT it at 21:25; #1140 failed without it at 06:14;
#1144 passed with it at 21:29. Every failure sits in one contiguous window and
everything outside succeeds — the limiter eased on its own. The fix is still
right for a reason those passes do not show: it removes the dependence on which
side of a window a run lands.

AN INTERMITTENT LIMITER IS WORSE FOR SIGNAL THAN A PERMANENT ONE. A job that
flaps teaches maintainers to stop reading it — which is precisely what happened:
I described this job's failure from memory twice in the org-wide review and was
wrong both times.

What the job got right and keeps: advisory so nothing was blocked, and it FAILS
CLOSED — the non-vacuity guard was skipped rather than passing vacuously, the
#1012 lesson working as designed.

ARTIFACT_FLOOR re-derived at 519 with `rivet list`.

Refs #1143, #1119, #1062


Claude-Session: https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant