Continuation of #44's diagnosis work: tikv/client-rust#516 (txn_cleanup_2pc_locks hang-timeout) is now root-caused end to end, and the client-side fix is in flight upstream.
The diagnosis (from the CI phase logs #553 gave us, the cluster-logs artifacts of two failing runs, and a local docker pause reproduction): a hypervisor-level pause of the runner VM (all three TiKVs logged monotonic time jumped back ~65s; PD lost its election lease and closed its TSO allocator, recovering ~15s later) leaves the client's TSO stream frozen-open — and nothing on the client's PD path bounded a wait: not the TSO receive loop, not stream creation, not the dial (Connection::connect even ignored its _timeout parameter). An in-flight current_timestamp() therefore hangs until nextest's 600s kill. The trigger is any virtualization infrastructure, so this is a production hang wearing a CI-flake costume.
Layer 2 — in flight: tikv/client-rust#558 bounds every PD-path wait (receive-loop deadline in Config::timeout windows with a [1,2]-window stall rule; creation watchdog armed only while a batch is pending; worker-death watch signal observed on both the enqueue and response sides; connect_with_timeout bounding dials, get_members bounded). Oracle: client-go's per-batch deadlineWatcher (defaultPDTimeout = 3s), checkStreamTimeout, and bounded dials. Verified by 10 unit tests plus the preserved docker pause repro (~/.cache/client-rust-test-toolchains/tso-repro/): before, a silent 44.5s hang; after, stall detected in ~4s, bounded 2s reconnect errors, recovery ~3s after thaw.
Layer 1 — queued behind #558's review: gRPC keepalive parity. client-go dials TiKV with keepalive.ClientParameters Time=10s/Timeout=3s, config-exposed (config/client.go, conn_pool.go); client-rust sets neither (its tcp_keepalive is inert against a paused peer whose kernel still ACKs). Honest benefit today is connection-level health and smoothing the first request after an incident; becomes load-bearing if client-rust grows long-lived KV streams. File as its own single-behavior PR once #558 settles.
Expected effect on the flake: the unbounded hang is gone — short freezes recover and pass; freezes longer than the retry budget fail fast (~3 min red with a descriptive error, cheap retrigger) instead of an opaque 600s timeout. Green-under-long-pause would additionally need CI retry policy (nextest retries with backoff) — offer upstream if they want #516 closeable in the strong sense.
Done when: #558 merged; layer-1 keepalive PR filed and merged (or explicitly declined upstream); #44 and upstream #516 updated/closed against the outcome.
Roadmap §3 / Phase 0 — landing what is already in flight.
Continuation of #44's diagnosis work: tikv/client-rust#516 (
txn_cleanup_2pc_lockshang-timeout) is now root-caused end to end, and the client-side fix is in flight upstream.The diagnosis (from the CI phase logs #553 gave us, the cluster-logs artifacts of two failing runs, and a local
docker pausereproduction): a hypervisor-level pause of the runner VM (all three TiKVs loggedmonotonic time jumped back~65s; PD lost its election lease and closed its TSO allocator, recovering ~15s later) leaves the client's TSO stream frozen-open — and nothing on the client's PD path bounded a wait: not the TSO receive loop, not stream creation, not the dial (Connection::connecteven ignored its_timeoutparameter). An in-flightcurrent_timestamp()therefore hangs until nextest's 600s kill. The trigger is any virtualization infrastructure, so this is a production hang wearing a CI-flake costume.Layer 2 — in flight: tikv/client-rust#558 bounds every PD-path wait (receive-loop deadline in
Config::timeoutwindows with a [1,2]-window stall rule; creation watchdog armed only while a batch is pending; worker-death watch signal observed on both the enqueue and response sides;connect_with_timeoutbounding dials,get_membersbounded). Oracle: client-go's per-batchdeadlineWatcher(defaultPDTimeout= 3s),checkStreamTimeout, and bounded dials. Verified by 10 unit tests plus the preserveddocker pauserepro (~/.cache/client-rust-test-toolchains/tso-repro/): before, a silent 44.5s hang; after, stall detected in ~4s, bounded 2s reconnect errors, recovery ~3s after thaw.Layer 1 — queued behind #558's review: gRPC keepalive parity. client-go dials TiKV with
keepalive.ClientParametersTime=10s/Timeout=3s, config-exposed (config/client.go,conn_pool.go); client-rust sets neither (itstcp_keepaliveis inert against a paused peer whose kernel still ACKs). Honest benefit today is connection-level health and smoothing the first request after an incident; becomes load-bearing if client-rust grows long-lived KV streams. File as its own single-behavior PR once #558 settles.Expected effect on the flake: the unbounded hang is gone — short freezes recover and pass; freezes longer than the retry budget fail fast (~3 min red with a descriptive error, cheap retrigger) instead of an opaque 600s timeout. Green-under-long-pause would additionally need CI retry policy (nextest retries with backoff) — offer upstream if they want #516 closeable in the strong sense.
Done when: #558 merged; layer-1 keepalive PR filed and merged (or explicitly declined upstream); #44 and upstream #516 updated/closed against the outcome.
Roadmap §3 / Phase 0 — landing what is already in flight.