[Isaac Lee (Isaac-Lee-msft)]

What this adds

A benchmark that measures how many Agent Sandbox claims per second an AKS control plane on the H8 hyperscale profile serves within its latency objectives. Claims are served from pre-warmed Sandboxes whose pods run on simulated KWOK nodes, so the load falls on the control plane: API server, etcd, kube-controller-manager's garbage collector, kube-scheduler and the agent-sandbox controller.

• modules/python/agent_sandbox
  • main.py configure installs pinned KWOK v0.8.0 and agent-sandbox v1.0.7 releases (SHA-256 checked) and fills 4 warm pools of 750 Sandboxes. The controller runs as one instance with throughput settings.
  • main.py execute runs an in-cluster load generator through a ladder of claim rates and copies its results back. The default ladder is 50–300 claims/s, 300 s measured per rate after a 60 s warmup.
  • main.py collect writes one JSON line per rate: latency p50/p90/p99, error rate, warm hit, teardown, generator lateness, and two verdicts (p99 ≤ 1 s, and p90 ≤ 200 ms, each with the error, warm-hit, teardown, drift and lateness checks).
  • generator.py uses only the Python standard library and runs from mcr.microsoft.com/azurelinux/base/python:3.12, so no new image is needed.
• steps/engine/agent-sandbox, plus execute-agent-sandbox.yml and collect-agent-sandbox.yml in the existing hyperscale-cluster topology. Its validate step already checks the H8 API server cores.
• scenarios/perf-eval/agent-sandbox: H8 cluster inputs, using the azapi module like ccp-provisioning, with a 4 × Standard_D16s_v5 system pool. Also a README describing the metrics and verdicts.
• pipelines/perf-eval/Hyperscale Cluster Benchmark/agent-sandbox-H8.yml: eastasia, manual trigger for now.

No Terraform module changes.

Testing

• New unit tests: 56 in modules/python/tests/agent_sandbox. The full Python suite passes locally (877 passed, 1 skipped) with 84.7% total coverage; the new modules are at 94–100%.
• Linters: pylint 3.3.4 with the repo's .pylintrc scores 10.00/10 on the new files. yamllint and terraform fmt -check (1.10.0) are clean.
• End to end on a kind cluster: configure → execute (5 and 10 claims/s) → collect.
• The generator alone on an AKS H8 test cluster: 2,972 claims at 50 claims/s, all Ready and served warm. p50/p90/p99 21/23/38 ms after warmup, generator lateness p99 0.6 ms, teardown p99 0.3 s.
• Not yet run in Telescope. It needs a New Pipeline Test run and /run-tf-integration.

Needs from maintainers

• A Kusto table and ingestion mapping for the agent-sandbox-H8 results (one line per rate).
• Quota for 4 × Standard_D16s_v5 in eastasia in the Telescope subscription, or a different size.
• A schedule (for example weekly), once the first runs are validated.

[github-actions]

For reviewers only: reply /run-tf-integration to trigger the terraform integration pipeline before approving the PR.