test(handler): fix TestRelayRoutes flake on /hex/packages and /gem/info - #395
Merged
Merged
Conversation
…ndons TestRelayRoutes fails intermittently on CI with "write: connection reset by peer" at the upstream handler's final Flush, on /hex/packages/demo and /gem/info/demo. Those two routes are the only ones in the table that fan out a second upstream request: with cooldown enabled, the hex and gem handlers fetch version timestamps concurrently with the artifact. The test server answers that sidecar request with the same 64 KiB chunked 502, but the handler returns as soon as it sees a non-200 and closes the body unread, so the proxy drops the connection while the test server is still writing to it. Whether the reset lands before or after the Flush is a scheduling race, which is why it only surfaces on loaded runners. The flush error says nothing about the response under test, which is asserted on the downstream side, so stop reporting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TestRelayRoutesfails intermittently on CI withwrite tcp ...: write: connection reset by peeratinternal/handler/relay_test.go:86, the upstream test server's finalrw.Flush().mainis red on this right now, and it has taken five of the six analytics PRs down with it. The failing subtest is alwaystruncated=falseon either/hex/packages/demoor/gem/info/demo.Cause
Those two routes are the only entries in the table that fan out a second upstream request. With cooldown enabled — and the test sets
proxy.Cooldown = &cooldown.Config{Default: "3d"}—HexHandler.fetchPackageAndVersionsandGemHandler.fetchIndexAndVersionsfetch version timestamps concurrently with the artifact itself:/hex/packages/demo/packages/demoand/api/packages/demo/gem/info/demo/info/demoand/api/v1/versions/demo.jsonThe test server has a single handler, so the sidecar request is answered with the same 64 KiB chunked 502 as the artifact request.
fetchFilteredVersionsreturns as soon as it sees a non-200 and its deferredresp.Body.Close()abandons the body unread, so the transport tears that connection down while the test server is still writing to it. The server's next write gets the reset.Whether the reset arrives before or after the
Flushis a scheduling race between two goroutines and the kernel socket buffers, which is why it only shows up on loaded runners and never locally. Reproduced in isolation — a client that reads the headers of a chunked response and then callsBody.Close(), against a hijacking server that keeps writing — the error string is identical once the body outgrows the receive buffer:On a Linux CI container the threshold sits far lower than on a developer macOS loopback, so 64 KiB is enough there.
Fix
The upstream handler cannot tell which of the two connections it is on, so a flush error there says nothing about the response under test — that is asserted on the downstream side, which is where a genuinely broken relay shows up. Report the write, drop the assertion.
Verification
go build ./...,go vet ./...,go test -race ./...andgo tool golangci-lint run ./...all clean onmain.The test keeps its teeth. Two mutations of
relayResponseare still caught:relayTrailerscall →trailers not relayed: map[X-Checksum:[]]io.Copy→io.CopyN(w, resp.Body, 1024)→body length = 1024, want 65536Not in scope
Abandoning the sidecar body also costs a reusable upstream connection on every cooldown-filtered hex/gem request that upstream answers with a non-200. Draining it under a byte cap before closing would be a small, separate improvement to the handlers; this PR only stops the test from asserting on a connection it does not own.
Context
Unblocks the pipelines on #388–#393, the six-part split of #381.
🤖 Generated with Claude Code