Skip to content

test(bdd): prototype composable smoke suite - #1193

Draft
sbaum1994 wants to merge 3 commits into
mainfrom
test/bdd-composable-live-smoke
Draft

test(bdd): prototype composable smoke suite#1193
sbaum1994 wants to merge 3 commits into
mainfrom
test/bdd-composable-live-smoke

Conversation

@sbaum1994

@sbaum1994 sbaum1994 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Why

The current BDD suite couples stack installation with product smoke behavior.
That makes every smoke pay the install cost and prevents the same feature from
running against an existing local, remote, or production-connected target.

This POC separates those lifecycles and supports both intended entry points:

  • Nightly CI selects a reviewed collection from a committed plan.
  • A developer selects one or more safe feature files directly against an
    existing target.

What changed

  • Split suite ownership into tests/bdd/install/ and tests/bdd/smoke/.
  • Kept command execution, state restoration, compensation, logs, DSL helpers,
    and target-neutral steps in shared packages.
  • Added smoke-only target loading, committed plan expansion, repeatable direct
    feature selection, consent checks, and isolated CLI sessions.
  • Added separate step registrars. Smoke features cannot resolve local topology
    bootstrap steps.
  • Moved ncp-local cleanup policy into the install suite. The shared harness
    exposes only the lifecycle hook; the smoke suite rejects BDD_CLEANUP_MODE.
  • Moved the cluster-maintenance POC into the reviewed operational smoke area.

Organization before and after

Arrows show execution or dependency direction. Neither suite depends on the
other.

flowchart LR
    subgraph "Before: one install-oriented suite"
        BRunner["godog_test.go<br/>entry points and wiring"]
        BFeatures["features/<br/>provisioning and product checks"]
        BCore["harness/, steps/, dsl/<br/>shared implementation"]

        BRunner -->|runs| BFeatures
        BRunner -->|uses| BCore
        BFeatures -->|uses vocabulary| BCore
    end

    subgraph "After: two suites around one shared core"
        direction TB
        Install["install/<br/>features, runner, cleanup policy"]
        Smoke["smoke/<br/>features, runner, selection"]
        Nightly["Nightly CI<br/>committed plan"]
        Developer["Developer one-off<br/>repeatable feature flags"]
        Target["smoke/targets/<br/>non-secret coordinates"]
        Steps["steps/<br/>shared visible DSL"]
        Harness["harness/<br/>execution and state mechanics"]
        DSL["dsl/<br/>pure helpers"]

        Nightly -->|selects collection| Smoke
        Developer -->|selects features| Smoke
        Target -->|configures| Smoke
        Install -->|registers install catalog| Steps
        Smoke -->|registers smoke catalog| Steps
        Install -->|executes through| Harness
        Smoke -->|executes and restores through| Harness
        Steps -->|runs commands through| Harness
        Steps -->|uses| DSL
    end

    BRunner -.->|separate ownership| Install
    BRunner -.->|separate ownership| Smoke
    BCore -.->|preserved| Steps
    BCore -.->|preserved| Harness
    BCore -.->|preserved| DSL
Loading

Component responsibilities

Package or directory Reason to change
install/ Install workflows, topology provisioning, install wiring, or ncp-local cleanup policy changes.
smoke/features/ Product behavior exercised against a ready target changes. Top-level files are safe for direct selection; operations/ requires a reviewed plan.
smoke/targets/ A target's non-secret execution coordinates change. Targets do not select code or grant consent.
smoke/plans/ A nightly collection, ordered operation, tag filter, or consent requirement changes.
smoke/ Go package Target, selection, consent, or smoke orchestration mechanics change.
harness/ Process execution, artifacts, state isolation, restoration, retry, or compensation mechanics change.
steps/ The shared operator-visible Gherkin vocabulary changes. Install-only bootstrap registration remains explicit.
dsl/ A deterministic interpolation, command-building, or structured-data helper changes.
fixtures/ An input is genuinely shared by both suites. Suite-specific fixtures stay with their owner.

Blast radius and extension checks

  • Adding a function-lifecycle smoke that uses the existing vocabulary and
    target coordinates adds one file under smoke/features/. Recursive wiring,
    direct selection, and a nightly *.feature collection pick it up without a
    Go change.
  • Adding a new target coordinate changes the feature precondition and the
    applicable target YAML, not a central Go struct or environment switch.
  • Running the same smokes on a multi-cluster topology selects a multi-cluster
    target. No runner or feature change is required.
  • Adding a reviewed operational workflow adds the nested feature and plan
    entry. Direct selection intentionally cannot bypass that plan.

Customer Release Notes

Not customer visible.

Plan Summary

No runtime resources change by default. An explicitly invoked maintenance plan
creates one temporary sample function and temporarily cordons the approved
compute cluster. Compensation is persisted before mutation.

Usage

Nightly or reviewed operation:

cd tests/bdd
go test ./smoke -run '^TestLive$' -timeout 60m -v -args \
  -bdd-target tests/bdd/smoke/targets/local-multi.yaml \
  -bdd-plan tests/bdd/smoke/plans/cluster-maintenance.yaml

Developer one-off selection uses repeatable flags. The function and task files
below are planned lifecycle smoke names:

cd tests/bdd
go test ./smoke -run '^TestLive$' -timeout 60m -v -args \
  -bdd-target path/to/target.yaml \
  -bdd-feature function-lifecycle.feature \
  -bdd-feature task-lifecycle.feature

See tests/bdd/smoke/README.md for the schemas and safety contract.

Testing

  • cd tests/bdd && go test -short ./...
  • cd tests/bdd && scripts/lint.sh
  • Mermaid diagram rendered successfully with Mermaid CLI.

The destructive maintenance feature was not run. QA is still required against
a ready target after the product behavior tracked by #1186 is available.

Notes

This remains a draft architecture POC. It does not auto-detect capabilities,
skip incompatible features, store credentials in targets, duplicate product
validation, or let direct selection reach cluster-wide operational features.

References

Relates to #1186

Related Pull Requests

None.

Dependencies

None. Existing Go modules are reused. No license review or NOTICE update is
required.

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Add a target-driven live suite with explicit provider and smoke phases, bounded retry and compensation primitives, and a portable cluster-maintenance example.

Relates to #1186

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Use open target environment maps and committed ordered plans so new smokes do not require Go schema or wiring changes. Isolate CLI and command state between phases, centralize Godog execution, and persist pending compensation for crash recovery.

Relates to #1186

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
@sbaum1994 sbaum1994 changed the title test(bdd): prototype portable live smoke composition test(bdd): prototype portable smoke composition Aug 26, 2026
Relates to #1186

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>
@sbaum1994 sbaum1994 changed the title test(bdd): prototype portable smoke composition test(bdd): prototype composable smoke suite Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant