[Liyu Ma (liyu-ma)]

Summary

• right-size the Scheduler Service Churn AKS system pool from 5 x Standard_D8_v3 to 3 x Standard_D2_v3
• right-size the in-cluster ClusterLoader2 Prometheus pool from 1 x Standard_D64_v3 to 1 x Standard_D4_v3
• right-size all 1,000 workload nodes from Standard_D4_v3 to Standard_D2_v3 while preserving the benchmark's 1,000-node / 20,000-pod topology
• change cpu_per_node from 4 to 2 so the generated 20 pods/node remain schedulable on D2_v3
• add scenario-specific Prometheus scale factors, generating a 2-core CPU request and 8 GiB memory request/limit for this run
• update ready-node validation from 1,006 to 1,004 after reducing the system pool by two nodes
• extend deletion TTL from 8h to 12h so this long-running benchmark is not deleted before post-test collection

Utilization evidence

Baseline build 79343 provided 39-40 live samples per pool. It ultimately failed after the 8-hour TTL deleted AKS before post-test cloud-info collection; the utilization samples below were captured successfully before deletion.

Pool	Baseline capacity	Captured peak CPU	Captured peak memory
default (5 x D8_v3)	39.10 cores / 131.57 GiB	3.78 cores	8.08 GiB
prompool (1 x D64_v3)	63.26 cores / 249.58 GiB	2.21 cores	7.17 GiB
userpool0 (300 x D4_v3)	1,158 cores / 3,996.55 GiB	257.32 cores	447.11 GiB
userpool1 (300 x D4_v3)	1,158 cores / 3,996.55 GiB	187.15 cores	444.67 GiB
userpool2 (400 x D4_v3)	1,544 cores / 5,328.74 GiB	250.09 cores	592.75 GiB

Verification build 79379 succeeded and supplied 52-54 healthy live samples per pool:

Pool	Resized capacity	Average / peak CPU	Peak CPU	Peak memory
default (3 x D2_v3)	5.70 cores / 16.97 GiB	44.53% / 61.04%	3.48 cores	6.42 GiB (37.84%)
prompool (1 x D4_v3)	3.86 cores / 13.32 GiB	28.73% / 39.33%	1.52 cores	5.56 GiB (41.75%)
userpool0 (300 x D2_v3)	570 cores / 1,696.90 GiB	24.00% / 37.67%	214.71 cores	434.21 GiB (25.59%)
userpool1 (300 x D2_v3)	570 cores / 1,696.90 GiB	24.63% / 38.94%	221.97 cores	434.87 GiB (25.63%)
userpool2 (400 x D2_v3)	760 cores / 2,262.53 GiB	25.15% / 46.21%	351.19 cores	578.31 GiB (25.56%)

No CPU sample exceeded 80%. The workload pools retain all 1,000 scheduler candidate nodes and the same deployment/pod scale. D2_v3 is already the smallest practical workload SKU; reducing workload node count would change the benchmark's scheduler-scale semantics.

Validation

• python3 -m unittest tests.test_large_cluster -v (41 tests passed)
• git diff --check main...HEAD
• verification pipeline build 79379 succeeded with the intended 1,004-node topology

[github-actions]

For reviewers only: reply /run-tf-integration to trigger the terraform integration pipeline before approving the PR.

[Copilot]

🟢 Approval recommended

Changes are consistent across pipeline, step templates, Python wiring, and tests, and the updated validation count matches the resized node pool topology.

Pull request overview

This PR right-sizes the AKS node pools and ClusterLoader2 Prometheus resource requests for the perf-eval/pod-churn-50k-sched scheduler benchmark, while preserving the 1,000-node / 20,000-pod topology and adjusting validation/TTL to match the resized cluster.

Changes:

• Resize AKS default, Prometheus, and workload node pools (D8→D2, D64→D4, D4→D2) and extend resource deletion TTL (8h→12h).
• Plumb scenario-specific Prometheus scaling factors through the pipeline → execution step → ClusterLoader2 override generation (with defaults preserved).
• Update cluster ready-node validation to match the new total node count (1006→1004) and adjust cpu_per_node (4→2) to keep pods schedulable on D2_v3.

File summaries

File	Description
steps/topology/service-churn/validate-resources.yml	Updates desired ready node validation count to match resized node pools.
steps/engine/clusterloader2/large-cluster/execute.yml	Adds CLI wiring for Prometheus scaling factors (env-driven with defaults).
scenarios/perf-eval/pod-churn-50k-sched/terraform-inputs/azure.tfvars	Right-sizes AKS node pools and extends deletion TTL for this scenario.
pipelines/perf-eval/Scheduler Benchmark/pod-churn-50k-sched.yml	Adjusts workload CPU-per-node and sets scenario-specific Prometheus scaling factors.
modules/python/tests/test_large_cluster.py	Adds/updates unit tests validating new Prometheus factor plumbing.
modules/python/clusterloader2/large_cluster/large_cluster.py	Extends override generation and CLI args to support configurable Prometheus scaling factors.

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Liyu Ma (liyu-ma)]

For reviewers only: reply /run-tf-integration to trigger the terraform integration pipeline before approving the PR.

[Liyu Ma (liyu-ma)]

/run-tf-integration