Repository navigation
Add node initialization validation benchmark - #1357
Merged
Merged
Conversation
Measure node readiness and NoSchedule taint clearance across sequential 250-, 500-, and 1000-node worker pool stages, polling every minute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid jq's reserved end keyword and preserve the last valid node snapshot and summary when parsing fails. Report result-writing failures explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep one-minute polling and update pipeline labels and recorded timeout metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use JobConfig with Standard_D2s_v3 system and worker pools. Run westus2 and eastasia 500-node stages in parallel after the westus2 baseline, and require both before the 1000-node stage. Qualify all stage IDs by region. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Set an explicit node resource group name and record it in benchmark metadata to avoid AKS's auto-generated name exceeding 80 characters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce benchmarkJob with sequential westus2 jobs and an independent eastasia job. Preserve PlanNodePools-based provisioning and include job identity in resource and artifact names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run two 1000-node worker pools in East Asia and add a keep_cluster boolean parameter that skips resource-group deletion when enabled, defaulting to cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Derive resource and artifact names from explicit region and node-count configuration instead of System.JobName. Rename keep_cluster to keep_failed_cluster and preserve cleanup for successful jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the westus2 250-node job and add a 15-minute operator pause after cluster creation in each remaining job. Preserve the westus2 500-to-1000 dependency chain. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the manual scaling pause, configuration recording, diagnostics, and artifact publication. Reduce validation to ten minutes and simplify the waiter to console progress with a temporary snapshot, preserving strict node checks and tolerating missing readiness conditions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract focused Bash and jq helpers, validate only requested pool_name:expected_count pairs, and derive pipeline arguments from the provisioned worker pools. Remove the fixed pool sizes and dependency availability checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the single-use nodeResourceGroup variable without changing generated pipeline YAML. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Liyu Ma (liyu-ma)
requested review from
Leonard Du (LeonardCareer),
Vittoria (vittoriasalim),
Lei Yao (wonderyl) and
xinWeiWei24
as code owners
October 26, 2026 13:49
|
For reviewers only: reply |
Relocate the pipeline and waiter to kcl/ccp_team/node-validation and regenerate YAML with the updated script path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Liyu Ma (liyu-ma)
changed the title
Remove readiness validation and reporting while retaining exact requested pool counts and NoSchedule taint checks. Reduce the polling deadline to 300 seconds and update generated pipeline labels. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
xinWeiWei24
reviewed
Oct 27, 2026
xinWeiWei24
reviewed
Oct 27, 2026
xinWeiWei24
reviewed
Oct 27, 2026
xinWeiWei24
reviewed
Oct 27, 2026
Reduce fetched node data before persistence, validate selected-node specs and taints, and remove unused kind/name checks. Separate the jq projection into named filters and distinguish fetch failures from malformed responses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Inline sleep handling and shorten the last interval to start a final check with five seconds remaining. Retain the static 30-second request timeout and accept successful in-flight responses after the polling window. Update generated pipeline labels. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Format KCL calls, rename predecessors, remove redundant stage conditions, and log UTC validation boundaries while preserving exit status during cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the westus2 500-node job and job predecessor wiring. Replace JobConfig with a region parameter and use a shared 1000-node-per-pool constant for both regions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Liyu Ma (liyu-ma)
force-pushed
the
maliyu/node-validation
branch
from
October 29, 2026 14:27
690ad5e to
7894487
Compare
xinWeiWei24
approved these changes
Oct 30, 2026
Lei Yao (wonderyl)
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a benchmark pipeline to validate worker nodes clear NoSchedule taints after node-pool creation.
Validation