Skip to content

Admit only complete canonical source-bound benchmark reports #142

Description

@flyingrobots

Problem and observable outcome

T-09.1's source-bound benchmark evidence has an incomplete ingress validator. xtask/src/benchmark_baseline/artifact.rs::validate searches for expected metadata lines and counts scenario/profile rows; it does not admit the complete canonical report grammar. A report containing both the expected git-commit and a conflicting git-commit is accepted. Existing unit fixtures also demonstrate that bare scenario<TAB>index and profile<TAB>index rows pass admission without headers or metrics.

Outcome: only one complete, unambiguous keep.streaming-cas-baseline/v1 report bound to the captured environment can reach artifact publication.

Reproduced RED

On original main-equivalent source in copy-isolated Docker, Rust 1.96.0, dedicated target: append metadata<TAB>git-commit<TAB>ffffffffffffffffffffffffffffffffffffffff<LF> to the accepted existing report fixture. validate returns success. The law conflicting_duplicate_source_metadata_refuses_before_publication fails at conflicting source coordinates were admitted. The probe was restored after execution; no production change or bad artifact was published.

Scope

Implement bounded canonical report admission, validated state before publication, precise typed admission failures, and permanent corruption/fuzz regression witnesses. Validate schema, complete unique metadata, exact headers, ordered unique scenario/profile catalogs, field widths, canonical numeric fields, configured sample/warmup coordinates, threshold posture, and required counter relationships. Bind captured source/compiler/host coordinates without accepting conflicting duplicates. Reject unknown/trailing rows and noncanonical framing.

Acceptance checks

  • The committed c529c07 baseline remains accepted with matching captured coordinates; the single-pass baseline from PR Fix: authenticate reference chunks once per read #134 also remains compatible after its prerequisite lands.
  • Identical and conflicting duplicate metadata both refuse with exact typed evidence before publication.
  • Missing/unknown metadata, wrong headers, duplicate/missing/reordered/unknown scenario or profile rows, wrong widths, invalid/overflow/noncanonical numbers, mismatched sample/warmup metadata, and trailing data refuse precisely.
  • Required semantic counter relationships agree with the report schema; legitimate historical two-pass accounting is not reinterpreted as current one-pass accounting.
  • Seeded bounded parser fuzz target and permanent mutation/model witnesses cover the new admission boundary.
  • Refusal never overwrites the last admitted artifact or leaves a published ambiguous report; existing publication recovery/exclusion laws remain green.
  • Docker debug/release, fmt, Clippy with -D warnings, source/documentation and relevant policy checks pass.

Prerequisites and safe merge boundary

No prerequisite on #71/#74, GC, migration or performance optimization: the current valid report format can be admitted strictly on main. PR #140 fixes unrelated hardware-dependent test setup; it is needed to clear that full-workspace Docker limitation, not to make report grammar correct. Shared files do not create semantic dependency edges. The fix must preserve the currently valid report format and leave main working independently.

Exclusions

No new timing measurements, thresholds, CI performance gates, benchmark scenarios, wire format/version change, forged environment evidence, or native test execution. T-09.2 and T-09.3 were originally unfinished and are excluded from this completed-task audit.

Refs #131 and #132. This is one executable issue for one coherent admission boundary, not a tracking container. The fixing PR must include regression evidence, issue linkage and resulting integration traceability.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions