fix(k8s): pin kubernetes client below 36 and self-heal on 401

[aron-muon]

Summary

Background: #72 reported continuous 401s from the Kubernetes API after upgrading to 3.6.0 on AKS, surviving pod restarts, with the only auth-relevant change being the kubernetes client bump 34.1.0 -> 36.0.0. I went looking for the actual regression instead of just pinning blind, and it turned out to be worse than a stale-token bug - client 36.0.0 can't authenticate in-cluster at all. The regenerated Configuration.auth_settings() gates on api_key['BearerToken'], while load_incluster_config's loader (byte-identical across 34/35/36) still writes api_key['authorization'] - so auth_settings() returns {} and every request goes out with no Authorization header. The reported "ran fine for ~3.5 days then broke" was likely the old ReplicaSet still serving until something forced the new image to actually take over.

• pinned kubernetes>=34.1.0,<36 (resolves to 35.0.0, which I verified behaves like 34.1.0)
• added regression tests that exercise the real installed client library: auth_settings() must be non-empty for in-cluster config, and a kubelet-rotated token must be served after the cached one expires. these fail on 36.0.0, so a future lockfile bump can't silently reintroduce this class of breakage
• added throttled reset-on-401 recovery: ApiException handlers in pool.py/job_executor.py now call handle_unauthorized(e.status), which drops the cached CoreV1Api/BatchV1Api so the next call reloads config and re-reads the on-disk SA token instead of failing until restart. throttled to once per 30s so a persistent RBAC problem doesn't turn into config-reload thrash

Testing

I reproduced the regression directly against all 3 client versions with fake in-cluster token files (rewrite token, expire the cache, re-read auth settings):

version: 34.1.0 | auth keys: ['BearerToken'] | after rotation: bearer TOKEN_V2
version: 35.0.0 | auth keys: ['BearerToken'] | after rotation: bearer TOKEN_V2
version: 36.0.0 | auth keys: []              | after rotation: NO AUTH SETTINGS

Full unit suite passes with the pin in place: 1529 passed. The new tests/unit/test_kubernetes_client_auth.py covers the loader regression plus the 401 reset/throttle behavior.

Review Strategy

The pin is the actual fix; the 401 recovery is belt-and-braces for future stale-token failures - if you'd rather ship the pin alone, the recovery commit is separable.

Fixes #72

🤖 Generated with Claude Code

[github-actions]

🎉 This PR is included in version 3.6.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀