Confirm request forwarding caused the 27 missed responses in the external-kill migration test #47

Context

CI run 40125789293, on PR #35 commit 761163c, failed test_migration_ft_external_kill_during_migration with 27 missed responses: requests that got no output at all. The state check and the other migration tests passed.

PR #35 (commit c7f8697) later removed the likely cause: forwarding requests for migrated keys into the owning worker's epoch.

• Before the fix: the consuming worker forwarded such a request to the owner as a WrongPartitionRequest, which sequenced it in its own epoch. Two workers then advanced the same ingress partition's offset, each taking the max, and forwarded requests could run out of order over the socket pool. A snapshot, or the Phase B stop, could therefore record an offset past requests that hadn't run yet. Recovery or the rebuilt consumer then skipped them.
• Now: the request stays in the consuming worker's epoch and runs on the owner as a proxied call (RunFunProxy / ProxyResult), so only the consuming worker advances that partition's offset.

What supports it

• The layout fits. In that run's external-kill test, recovery reverted to the old layout and put shadow partition 7 on worker 2 and partition 3 on worker 1; under the old layout, partition 3 holds partition 7's keys. The client that submitted the migration routes by the new graph from the moment it submits, so every request it sent to partition 7 crossed workers, which is the forwarding path. The logs are in that run's e2e-migration-artifacts, kept until 2027-01-22.
• A regression test fails on the forwarding code. TestOffsetNeverPassesAnUnrunRequest in tests/unit/worker/test_proxied_requests.py fails on the commit before the fix: offset 41 is recorded while request 40 is still waiting in the owner's sequencer.
• Proxied calls run in the same scenario. In a local migration E2E run, 236 of the 239 proxied calls came from the external-kill test. CI runs on e629945 since then show 199–926 proxied calls in the revert path, with no missed responses.

What's missing

• No direct evidence. That run predates uploading the unanswered request ids (missed_requests.csv, added in e8c0564), and the old forwarding handler didn't log at the default WARNING level. So the 27 requests can't be matched to forwarded requests or to the partition 7 / partition 3 split.
• Never reproduced. The failure never showed up locally, before or after the fix.

To close this

Either:

• run test_migration_ft_external_kill_during_migration on the commit before the fix (c8bf919 has the proxy change, so use e8c0564) until it misses responses, and check that the ids in missed_requests.csv are requests sent to a shadow partition whose keys belong to a partition on another worker; or
• close it once the external-kill test has passed on CI enough times. With the fix it has passed on every run so far (four as of e629945 and its rerun).

Once this is settled, lower the Proxied calls this epoch: N log line in worker/transactional_protocols/aria.py from WARNING to INFO. It's kept at WARNING so CI artifacts show whether the proxy path ran.