feat(breakfix): BFX break-fix suite (BFX01-BFX06) with NICo coverage - #562
Draft
abegnoche wants to merge 5 commits into
Draft
feat(breakfix): BFX break-fix suite (BFX01-BFX06) with NICo coverage#562abegnoche wants to merge 5 commits into
abegnoche wants to merge 5 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
abegnoche
force-pushed
the
cursor/bfx-remediation-suite-4f18
branch
from
July 30, 2026 15:23
85d9bae to
52bc942
Compare
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Introduce suites/remediation.yaml covering all open M8 break-fix issues (BFX01-BFX06): validation classes, my-isv demo stubs, and NICo provider wiring. Move BFX03-01 (HardwareSerialCheck) out of bare_metal into the new plain suite with capability gating per PR #561. NICo implements read-only observability where the tenant REST API exposes signals (maintenance events, repair history, serial inventory). Mutating lifecycle actions, GPUd/Sentinel/Maestro agents, NV switch firmware, retirement notices, and tenant notification channels emit structured skips with documented gap IDs. Stacks on PR #561 (capability/requires model). Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Alexandre Begnoche <abegnoche@users.noreply.github.com>
abegnoche
force-pushed
the
cursor/bfx-remediation-suite-4f18
branch
from
August 4, 2026 20:10
52bc942 to
4a7e1b9
Compare
Break-fix is mostly bare-metal plus k8s-only cordon/GPU reset, so a plain remediation suite is not justified the way storage is. Wire BM BFX into bare_metal, k8s actions into k8s, and rename scripts/modules to breakfix. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
Both nico configs imported suites/bare_metal.yaml, so `--provider nico --suite bare_metal` could not resolve and demanded an explicit -f. This was the only such collision in the repo. Fold key_access.yaml's setup/teardown steps into bare_metal.yaml and delete the file. The test step was already identical in both; only the throwaway-key provisioning was unique. Those two steps carry `skip: true` so a routine run stays read-only -- setup_key_access mutates the site (POST sshkey + sshkeygroup, best-effort PATCH of the SSH-key SOL flag), which should not be a side effect of ordinary bare-metal validation. Unskip both to make AUTH-XX-03 pass without a pre-synced key. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
The test-plan coverage guardrail requires each plan entry's labels to equal the union of labels across the suite wirings declaring that test_id. Folding the BFX checks into bare_metal.yaml and k8s.yaml added platform labels to the wiring but not to docs/test-plan.yaml, leaving `make test` red on all 13 BFX entries. Add bare_metal to the eleven bare-metal entries, and kubernetes to BFX01-01 and BFX01-04, which live in the k8s suite. Also apply the pre-commit import ordering and whitespace fixes the BFX commits missed. Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
… checks
Against a live NICo site both checks passed while proving nothing.
events_queryable and history_queryable are set by the provider script the
moment the machine-list call succeeds, so MaintenanceEventsCheck passed with
zero events -- it would pass identically against a site with no maintenance
capability at all.
Skip instead of passing when the evidence list is empty. An empty list cannot
distinguish a working query API from one that returns nothing, and a provider
should not clear BFX02 without demonstrating the capability. A failed or
absent API still fails, so the stricter reading only affects the vacuous case.
Also stop counting InUse as repair history. _REPAIR_STATUSES is
{Maintenance, Reset, Error, Repairing}, but the filter admitted InUse too --
a normal lifecycle state -- so machines that had never been repaired produced
repair records.
Verified on a NICo dev site: BFX02-01 now skips (site has no maintenance
events), BFX02-03 still passes on genuine Error history across all 6 machines.
Signed-off-by: Alexandre Begnoche <abegnoche@nvidia.com>
abegnoche
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the break-fix validation suite covering all 13 M8 tests (BFX01–BFX06), wired into
the existing
bare_metalandk8ssuites rather than a standalone suite.isvtest/validations/breakfix.py— 12 new validation classes (BFX03-01 reuses theexisting
BmHardwareSerialCheck)suites/bare_metal.yaml— 11 checks;suites/k8s.yaml— GPU reset + cordonmy-isv— demo stubs for every step, exercised bymake demo-testnico— 4 read-only implementations; the rest emit a structured skip with agap: BFXxx-yymarker rather than a vacuous passValidated against live NICo
Run on two independent NICo deployments on
az51-dev3(management cluster, 6 CPU-onlymachines) and
az51-dev3-dh1(18 GPU hosts).Errorhistory on 18/18 machinesThe 8 gaps reproduced with identical reasons on both deployments, so they are platform
gaps rather than one site's configuration. BFX02-01 is a missing-fixture problem, not a
NICo gap — it belongs in a different bucket.
Every break-fix step reads only the
machineresource, so the suite runs withinfrastructure-scope access and needs no tenant-level API permissions.
Honest coverage note
On NICo, 9 of the 13 checks are placeholders that cannot pass or fail — they document
missing API surface. Only BFX03-01 and BFX02-03 currently prove anything about NICo. The
gapmarkers make that visible instead of hiding it behind green checkmarks.Also in this PR
nico/config/key_access.yamlfolded intobare_metal.yaml— both imported the samesuite, which made
--provider nico --suite bare_metalunresolvable. Thekey-provisioning steps carry
skip: trueso a routine run stays read-only.make test).InUseno longer counts as repair history.Testing
make test(124 passed) ·make demo-test·make lint·uvx pre-commit run -a·scripts/validate_suite_wiring.py --checkNew checks are unreleased; run with
ISVTEST_INCLUDE_UNRELEASED=1.Follow-ups
maestro-systemis deployed onaz51-dev3— may unblock the BFX01 mutating workflowsnico/scripts/breakfix/reset_gpus.py(GPU reset moved to the k8s suite; NICo has no k8s config)k8s.yamland have notbeen exercised end-to-end —
make demo-testdoes not run the k8s suite and NICo has nok8s config
Closes
Closes #207
Closes #208
Closes #210
Closes #211
Closes #212
Closes #213
Closes #214
Closes #215