JCloudsCleanupThread stalls with HTTP/2 REFUSED_STREAM (OVH Public Cloud); failed termination resets the idle clock, making agents immortal

[pilere]

Environment

• Jenkins core 2.576 (official container image, K3D/Kubernetes)
• openstack-cloud 1465.vd3b_a_103e31fd (latest), okhttp-api 5.3.2-200.vedb_720a_cf1f8 (latest)
• Provider: OVH Public Cloud (Keystone v3 at https://auth.cloud.ovh.net/v3/, region GRA11, Nova at compute.gra11.cloud.ovh.net)
• One cloud, one template, JNLP launcher, retentionTime: 30, instanceCap: 3

Symptoms

1. The controller log fills with OpenStack slave cleanup thread is still running. Execution aborted. — bursts of 13-19 entries over ~30 min windows, so each cleanup pass runs long enough to collide with the next tick.
2. Passes regularly die with:

SEVERE  j.p.o.c.JCloudsCleanupThread#execute: Unable to perform the cleanup
org.openstack4j.api.exceptions.ConnectionException: stream was reset: REFUSED_STREAM
caused by okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
	at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:166)
	at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:105)
	...

3. Net effect: ephemeral agents are never reaped. We had idle agents alive (and billing) for 48h+ with retentionTime: 30.

The part that makes it self-sustaining

While watching an idle agent that should have been reaped, its Computer.getIdleStartMilliseconds() jumped forward every ~30-36 minutes with zero builds on the node (checked per-executor: both executors' idleStart reset, at slightly different times). That matches the retention period: every time the retention check decides to terminate and the termination path fails on the OpenStack API, the node's executors appear to be recycled, which resets the idle clock — so the next attempt is again 30 minutes away, forever. The agent is effectively immortal.

Ruled out

• The OVH API itself is healthy: from the controller host, Keystone answers 200 in ~55 ms consistently; Nova responds normally. Provisioning (few API calls) works fine throughout — only the chatty cleanup path degrades, which is what points at HTTP/2 stream exhaustion on a long-lived connection (OVH's API gateway refusing new streams, REFUSED_STREAM).
• A full controller restart does not help: the very first cleanup pass on the fresh JVM already collides with the next tick, and the SEVERE reappears within the hour.
• Plugin and okhttp-api are the latest released versions.

Workaround

Jenkins.get().getComputer(name).deleteSlave() from the script console works instantly and reliably (node + Nova server cleanly destroyed), which confirms single destroy calls are fine — it is the bulk cleanup/listing burst that chokes.

Suggestions

• Retrying REFUSED_STREAM on a fresh connection (or forcing HTTP/1.1 for the openstack4j connector, or capping concurrent streams) would likely fix the SEVERE.
• Independently, a failed termination should not reset the node's idle accounting — that turns a transient API failure into permanently leaked (billed) instances.

Happy to provide more logs or test a patched build against OVH.