Raised by @julianrubino during review of #4618.
The gap
e2e/e2etests/test_keygen_reset_signing.go reproduces the incident state (keygen record blanked) and proves the signers keep signing through it. It never restarts a zetaclient, so it does not exercise the actual mainnet failure: a cold start against a blanked record, where the whitelist has to come from TssParticipantList because the grantee list is gone.
That path is covered by unit tests on resolveWhitelist, and it was verified by hand during development — a docker restart of a signer against the blanked state comes up logging TSS key already finalized; whitelisting its participants grantees_in_keygen_record=0 followed by TSS service created. But nothing automated covers it.
Why it cannot be fixed in the test today
The e2e runs inside the orchestrator container, which has no way to control its siblings:
contrib/localnet/docker-compose.yml mounts /var/run/docker.sock into promtail only, and only under the monitoring profile. The orchestrator service has no such mount.- There is no docker client dependency anywhere in
go.mod, cmd/zetae2e/ or e2e/.
So Setup only ever runs once per container in an e2e, at process start, before any test body executes. Any test that wants to assert on startup behaviour has no way to trigger it.
What it would take
Give the orchestrator a docker socket and a small restart helper in e2e/runner, so a test can stop and start a named zetaclient container and wait for it to come back healthy.
That unlocks more than this one test — anything about zetaclient startup, config reload, or crash recovery is currently untestable for the same reason.
Deliberately not done in #4618: it is a hotfix headed for two release branches, and adding a docker socket to the orchestrator is an e2e infrastructure change that deserves its own review.
Raised by @julianrubino during review of #4618.
The gap
e2e/e2etests/test_keygen_reset_signing.goreproduces the incident state (keygen record blanked) and proves the signers keep signing through it. It never restarts a zetaclient, so it does not exercise the actual mainnet failure: a cold start against a blanked record, where the whitelist has to come fromTssParticipantListbecause the grantee list is gone.That path is covered by unit tests on
resolveWhitelist, and it was verified by hand during development — adocker restartof a signer against the blanked state comes up loggingTSS key already finalized; whitelisting its participants grantees_in_keygen_record=0followed byTSS service created. But nothing automated covers it.Why it cannot be fixed in the test today
The e2e runs inside the
orchestratorcontainer, which has no way to control its siblings:contrib/localnet/docker-compose.ymlmounts/var/run/docker.sockintopromtailonly, and only under themonitoringprofile. Theorchestratorservice has no such mount.go.mod,cmd/zetae2e/ore2e/.So
Setuponly ever runs once per container in an e2e, at process start, before any test body executes. Any test that wants to assert on startup behaviour has no way to trigger it.What it would take
Give the orchestrator a docker socket and a small restart helper in
e2e/runner, so a test can stop and start a named zetaclient container and wait for it to come back healthy.That unlocks more than this one test — anything about zetaclient startup, config reload, or crash recovery is currently untestable for the same reason.
Deliberately not done in #4618: it is a hotfix headed for two release branches, and adding a docker socket to the orchestrator is an e2e infrastructure change that deserves its own review.