Skip to content

Validation pipeline foundation (P1+P2): validate floor + trusted lane as shadow checks - #866

Merged
brandom-msft merged 16 commits into
mainfrom
brandom-msft-merge-stack-to-main
Jul 31, 2026
Merged

Validation pipeline foundation (P1+P2): validate floor + trusted lane as shadow checks#866
brandom-msft merged 16 commits into
mainfrom
brandom-msft-merge-stack-to-main

Conversation

@brandom-msft

@brandom-msft brandom-msft commented Jul 31, 2026

Copy link
Copy Markdown
Member

Context

This merges the P1+P2 foundation of the public-first validation pipeline into main. The whole stack was built as a linear feature-branch stack and has never landed on public main. Merging it here turns on the two PR checks — validate (the L3 "floor" matrix) and validate / trusted (the fork-gated L4 lane) — so they start running against real incoming PRs.

These go live as SHADOW / non-required checks. Nothing in this PR marks any check as required and no branch ruleset is touched. Flipping to a required gate is a later, separate step (P3). Tracked as ADO 5486308 — https://msdata.visualstudio.com/Vienna/_workitems/edit/5486308 (under Feature 5449671).

What changed

  • Purely additive — 29 files, 1926 insertions, zero deletions/modifications vs the merge-base. No existing file is altered.
  • New workflows: .github/workflows/validate.yml (detect → validate floor matrix → single trusted job) and .github/workflows/scripts-selftest.yml (script self-test harness, real-triggered).
  • New shared scripts: validate-sample.sh (single-sample L3 validator), detect-changed-samples.sh (in-job, data-driven change detection), plus CLASSIFICATION.md and detect-changed-samples.README.md.
  • New test harness + fixtures under .github/scripts/test/ (go / python / typescript / csharp / java / csharp-yaml — good, broken, and deps variants).
  • main was merged into this branch (merge commit 2ab3c5c) to bring it current; the merge was clean (ort, no conflicts) because main has zero changes under .github/ since the merge-base. The stack's .github/ tree is byte-identical to its pre-merge tip.

Why this matters

Turning these on as shadow checks lets us observe real signal (pass/fail/error classification) on live PRs — including fork PRs — with zero risk of blocking contributors, and establishes the precondition for making the floor a required gate in P3.

Design properties (frozen — unchanged by this PR)

  • validate.yml is on: pull_request (never pull_request_target) with permissions: contents: read — fork-safe by construction: fork code executes on a barren, secret-less runner.
  • The trusted L4 lane is a single job (never a matrix), so its check context is exactly validate / trusted; it is draft-skipped and fork-gated.
  • Protected public paths (redirect-pull-requests.yml, mirror-back.yml, run-setup.yml, mirror-back.sh) are untouched by this merge.
  • This PR is a merge, not a code change — no trust/fork/gate logic, redirect bot, or scripts were modified.

Static-analysis notes (for reviewer transparency)

Local actionlint + shellcheck are clean apart from two pre-existing, benign false positives that predate this merge (the .github/ tree is byte-identical to the pre-merge tip) and are intentionally left as-is under the merge-only scope:

  • validate.yml:270 — SC1007: the GITHUB_OUTPUT= prefix is an intentional per-invocation empty-env assignment (documented inline).
  • validate-sample.sh:349 — SC2329: cleanup_python_venv is invoked indirectly via trap ... EXIT (line 393).

Notes & links

brandom-msft and others added 15 commits July 24, 2026 09:50
@brandom-msft
Add public-first single-sample validator (P1.1) + inert self-test har…
585afca 
…ness

Re-home validate-sample.sh into public .github/scripts/ — a verbatim logic
port of .azure-pipelines/validation.yml Stage 2, whose 5 duplicated
per-language bash jobs collapse into one function set. Net-new vs. ADO: the
3-way classifier (pass=0 / fail=1 / error=2, ERROR never quarantines) and
--results-dir / $GITHUB_OUTPUT emit.

Adds real per-language good/broken fixtures, a generalized 5-language exit
gate (run-tests.sh, with a scrubbed-PATH ERROR proof), and an inert
contents:read workflow that runs the gate on a real runner. The harness
triggers only on workflow_dispatch (inert stub) + pull_request into the
integration branch — never on public main traffic. Credential-free: no
secrets, no OIDC.

ADO Task 5015439 (P1.1): https://msdata.visualstudio.com/Vienna/_workitems/edit/5015439

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b970b82c-3e04-4c0a-970c-e2aef97c714b
@brandom-msft
Add P1.2 in-job path detection (detect-changed-samples.sh) + inert se…
bef230e 
…lf-test

Ports the private ADO DetectChanges stage to a GH-Actions-native detector: changed files under samples/ resolve to their nearest ancestor sample.yaml dir, deduped into a set exposed as job outputs (has_changes/count/samples) for needs.*.outputs consumption. Fail-loud on git diff error (ADO 5247751); empty set = docs-only short-circuit. Single deduped set instead of per-language has_<lang> booleans (language derived from path at fan-out). Sibling hermetic harness (git-only) + inert selftest workflow (contents:read, no secrets/OIDC, no on:paths).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 441d830f-ce1e-4ebf-bdbf-cdf84cafef1f
@brandom-msft
Keep detect harness hermetic: unset GITHUB_OUTPUT so it never pollute…
dc669c3 
…s the step's output file

The unit harness invokes the detector ~10x and asserts on stdout; on a runner each call was appending has_changes/count/samples to the real \, which GitHub then rejected. Unset it in the harness — the GITHUB_OUTPUT emission is proven separately by the detect->consume plumbing job.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 441d830f-ce1e-4ebf-bdbf-cdf84cafef1f
@brandom-msft
Fix empty-result GITHUB_OUTPUT corruption on docs-only detection
41a6249 
grep -c on an empty file prints 0 AND exits 1, so the '|| echo 0' fallback double-printed, making COUNT=\
@brandom-msft
docs(detect): document base-ref known limitations (two-dot, HEAD~1, b…
f7cc98d 
…est-effort fetch)

Non-behavioral: adds a Known-limitations comment block above resolve_base_ref
capturing the three faithful-port characteristics surfaced in the P1.2
Chapter-1 walkthrough (two-dot PR diff, HEAD~1 final-transition-only on push,
best-effort fetch staleness). No logic change to the frozen DetectChanges port.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 441d830f-ce1e-4ebf-bdbf-cdf84cafef1f
@brandom-msft
docs(detect): add feature-team README for the changed-sample detector
7a6f222 
One-page companion to detect-changed-samples.sh: the five-move flow, the
fail-loud rule, the input/output contract, the needs.*.outputs wiring note,
and the two tracked known-characteristics follow-ups (ADO 5473511, 5473651).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 441d830f-ce1e-4ebf-bdbf-cdf84cafef1f
@brandom-msft
P1.3: classify infra/transport dependency failures as error not failure
a441769 
pip/npm install failures with positive transport evidence (DNS / connection refused-reset / timeout / registry 5xx) now classify as error (exit 2) instead of failure (exit 1); unaccompanied resolution errors stay failure. Transport evidence wins even when pip prints its misleading 'No matching distribution' line after exhausting retries against an unreachable index.

- dep_infra_signature() narrow transport allow-list, routed on the two dedicated install steps only (merged build tools stay failure - documented limit)
- CLASSIFICATION.md: exit contract, taxonomy, ambiguity-bias, honest limits
- run-tests.sh: real-runner blackhole cases (pip/npm -> error) + hermetic no-index resolution failure (-> fail); new python-deps / typescript-deps fixtures

ADO 5449687: https://msdata.visualstudio.com/Vienna/_workitems/edit/5449687

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6b0ffec6-9f36-4309-80ef-adb79f1de81a
@brandom-msft
P1.3 fix: don't mute npm errors (--loglevel=error not --silent)
391ba22 
npm install --silent suppresses error output too, so the captured log was empty and dep_infra_signature saw no transport evidence -> blackhole misclassified as fail. Real-runner proof (run 30309241864) caught this: pip cases passed, npm registry-unreachable got fail instead of error. Switch to --no-audit --no-fund --loglevel=error: quiet on success, ECONNREFUSED still emitted and captured.

ADO 5449687: https://msdata.visualstudio.com/Vienna/_workitems/edit/5449687

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6b0ffec6-9f36-4309-80ef-adb79f1de81a
@brandom-msft
docs(classification): note proxy-403/blocked-registry honest limit
01cdde1 
Microsoft Security is blocking direct public-npm access on managed devices (CFS-protected feed); a held package can return 403/blocked, which the 5xx-only allow-list doesn't promote to error. Documented as a v1 honest limit + deferred follow-up. Doc-only; no classifier change (runner-proven behavior is commit 391ba22).

ADO 5449687: https://msdata.visualstudio.com/Vienna/_workitems/edit/5449687

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6b0ffec6-9f36-4309-80ef-adb79f1de81a
@brandom-msft
P1.4: audit + retirement mapping for ADO task-wrapper -> GH actions swap
1ce5dfa 
Documents that the public-first validation path is already fully GitHub-Actions-native (zero ADO-isms): every UseDotNet@2/UsePythonVersion@0/NodeTool@0/JavaToolInstaller@0/GoTool@0 setup task maps to a pinned setup-* action; cross-job data flows via needs.*.outputs; CheckMailmap + Report/status-posting intentionally dropped; artifact fan-out + aggregation deferred to P2.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a7d3b36b-7970-4500-888e-af959a5a5066
@brandom-msft
P2.1: add validate (L3 floor) lane for public fork PRs
3287cec 
Real PR-triggered floor lane (distinct from the inert self-tests): on: pull_request (never pull_request_target), permissions contents:read only, no environment/secrets. detect-changed-samples.sh finds changed samples in-job; a matrix fans out one runner per sample and validates to L3 via validate-sample.sh, deriving --language from the samples/<dir>/ path. Untrusted fork code runs on a barren runner by design.

ADO 5449695

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: c4e21e67-bf10-42a8-9e21-98dde95aa89f
@brandom-msft
P2.2: add validate / trusted L4 OIDC lane (single job, fork-gated)
b7a3dd8 
Adds the credentialed trusted counterpart to the P2.1 floor lane as a third job (trusted) in validate.yml, producing required-check context 'validate / trusted'. Single job (no matrix); fork+draft gated via job-level if; docs-only short-circuit before any secret step; in-job parallel L3; L4 via warm project (SKIP_PROVISION=true) with OIDC login to env L4-validation. Reuses shared detect + validate-sample.sh scripts unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 446e89c2-fad1-4300-bdc7-fcd2599deb24
@brandom-msft
CI: consolidate script self-tests into one real-triggered workflow
47c5416 
Replace the two inert *-selftest.yml dev harnesses (which only fired on PRs

based on a throwaway dev branch, so never on real traffic) with one lean

.github/workflows/scripts-selftest.yml that triggers on pull_request +

paths: ['.github/scripts/**'] (plus workflow_dispatch).

Jobs (contents:read only, non-required, separate from validate.yml):

- detect-harness: hermetic run-detect-tests.sh

- validate-harness: 5 toolchain setups + yq, run-tests.sh (must reach exit 0)

- detect -> consume: trimmed needs.*.outputs isolation proof (non-empty case;

  docs-only empty-result contract already pinned by run-detect-tests.sh case h)

Repoint now-dangling references to the deleted workflows in CLASSIFICATION.md

and detect-changed-samples.README.md; add a post-consolidation banner to the

historical P1.4 WRAPPER-MAPPING.md audit.

Validated: actionlint + shellcheck clean; hermetic detect harness 23/23 GREEN

locally; validate harness PARTIAL locally with 0 failures (toolchain-gated

cases skip, provided by the runner).
@brandom-msft
docs: drop point-in-time P1.4 WRAPPER-MAPPING audit (record lives in …
94984d7 
…ADO 5449688 + git history)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3417ac9a-6d98-4124-b452-44410f3c3960
@brandom-msft
Merge branch 'main' into brandom-msft-merge-stack-to-main
2ab3c5c 
Merge-forward to bring public main into the P1+P2 validation-pipeline stack ahead of the shadow go-live PR (ADO 5486308). Stack is purely additive vs merge-base; no .github changes on main since merge-base.
@brandom-msft
brandom-msft requested a review from lindazqli as a code owner July 31, 2026 16:45
@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

👋 Thanks for your contribution, @brandom-msft!

This repository is read-only. If you are contributing on behalf of Microsoft, please submit your PR to the private staging repository instead:

👉 foundry-samples-pr

See CONTRIBUTING.md for full instructions.

@github-actions github-actions Bot closed this Jul 31, 2026
@brandom-msft brandom-msft reopened this Jul 31, 2026
@brandom-msft
brandom-msft changed the base branch from brandom-msft-scripts-selftest-ci to main July 31, 2026 16:56
@brandom-msft
brandom-msft removed the request for review from lindazqli July 31, 2026 16:58
@brandom-msft
Correct stale/contradictory comments in validation pipeline foundation
5469a61 
Comment/doc-only cleanup on the composed P1+P2 change set. No YAML logic, script behavior, or trust/gate/fork logic changed.

- validate.yml: scope the fork-safety 'no environment/no secrets' bullet to the fork-reachable detect/validate jobs so it no longer contradicts the trusted L4 lane described below; name scripts-selftest.yml directly instead of the stale *-selftest.yml glob.

- run-detect-tests.sh: header case list now documents case (h) (GITHUB_OUTPUT hygiene) plus the --output-dir artifact assertion.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 348d520e-aad2-4b98-ba83-7c157597212d
@brandom-msft
brandom-msft merged commit 3cf6ffe into main Jul 31, 2026
13 checks passed
@brandom-msft
brandom-msft deleted the brandom-msft-merge-stack-to-main branch July 31, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@brandom-msft