[Xu Xue (xuexu6666)]

Problem

The GPU Cluster CRUD pipeline (pipelines/perf-eval/GPU Benchmark/k8s-gpu-cluster-crud.yml, ADO defId 61) fails on most scheduled runs. Node-pool scale/create steps hit the per-step timeout while A100/H100 nodes are still coming up:

• ARM reports the scale succeeded, but the K8s readiness wait — driven by step_time_out (was 600s / 10 min via operation_timeout_minutes) — gives up first with Only N nodes are ready.
• On timeout the harness advances to the next CRUD step and collides with the still-running ARM op → (OperationNotAllowed) ... in-progress scale node pool operation, cascading to a hard failure.

A first attempt at step_time_out: 1800 (30 min) still failed (build 79914): the 1→2 A100 scale-up needed more than 30 min, and — critically — step_time_out alone cannot exceed 30 min because the ARM long-running-operation poll was hardcoded to TIMEOUT_SECONDS = 1800.

Fix

• step_time_out 600 → 2700 (45 min) for all GPU stages. This drives the K8s readiness wait (operation_timeout_minutes = step_time_out / 60).
• Make the ARM poll timeout configurable. begin_create_or_update_with_retry gains a timeout_seconds arg (defaults to TIMEOUT_SECONDS), threaded through prepare_create_operation / prepare_scale_operation.
• AKSClient derives arm_operation_timeout_seconds = max(step_timeout_seconds, 1800) so the ARM ceiling tracks the per-step timeout for GPU while non-GPU callers keep the historical 1800s floor.

Together these let a GPU step run the full 45 min on both the readiness and ARM sides, so the op is no longer abandoned mid-flight and the OperationNotAllowed cascade disappears. Well within the per-stage timeout_in_minutes: 180 budget.

Testing

• Unit tests added for the configurable timeout (begin_create_or_update_with_retry, prepare_* forwarding, AKSClient.arm_operation_timeout_seconds floor/track).
• Full modules/python suite: 824 passed, 1 skipped.

[github-actions]

For reviewers only: reply /run-tf-integration to trigger the terraform integration pipeline before approving the PR.

[Copilot]

🟡 Changes recommended

Workload readiness waits share the increased timeout, potentially exceeding the unchanged 180-minute job budget.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Raises GPU CRUD pipeline step timeouts from 600 to 1800 seconds to accommodate slow node readiness.

Changes:

• Updates all eight GPU stages to use a 1800-second timeout.
• Keeps the existing 180-minute job timeout.

File summaries

File	Description
pipelines/perf-eval/GPU Benchmark/k8s-gpu-cluster-crud.yml	Increases timeout values for every GPU stage.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.