Skip to content

Integrity counts every default-branch workflow run as verification, bots and scheduled jobs included; a dormant repository with external CI is UNINSTRUMENTED #40

Open

Description

@drevendev

Found by the review of 2026-09-30. Two questions about what Integrity counts as evidence; both need a judgement, because the accepted contracts do not say.

1. Every workflow run on the default branch is "verification", whatever triggered it

_collect_ci asks GET /actions/runs?branch=<default> and keeps every run whose head_sha is a default-branch commit of the window; the run's event is never looked at. GitHub runs schedule, workflow_dispatch, issues, issue_comment, pull_request_target, label and similar workflows with the default branch as ref and its newest commit as head_sha, so they arrive in that listing as parents of the revision.

This repository shows it live. On 2026-09-30 the listing for master since 2026-09-01 holds 40 runs: 12 push, 27 schedule and 1 workflow_dispatch, and every schedule run is on a8726f5:

schedule  a8726f5  Released pins resolve  success  2026-09-30T14:13:59Z
schedule  a8726f5  Self-observation       success  2026-09-28T14:09:22Z

So the daily check that documented tags exist, and the self-observation job itself, are counted as verification of the code at a8726f5. It cuts both ways, and both were reproduced against the fake transport:

  • a failing triage bot on issue_comment becomes a VERIFY_FAIL of the revision, and the failure is sticky for 14 days. This is exactly what happened here: the one failed run master ever had is Released pins resolve on push at a8726f5, 72 seconds before v0.1.8 was tagged, and the self-observations of 2026-09-14 and 2026-09-21 reported Integrity FAILING, attention integrity CRITICAL, while every CI run was green (the review PR removes that workflow's push trigger, which removes this one instance, not the rule);
  • a labeler that succeeds becomes a VERIFY_PASS of a revision nothing tested (in the probe it turned FAILING into SPARSE_MIXED);
  • a bot run also counts as an Actions run seen, which suppresses the check-suite surface, so an external CI (CircleCI, Buildkite) that failed is never sampled: FAILING became SPARSE.

The accepted texts do not settle it: integrity-ci.md defines a parent as "one logical verification execution of a revision" and github-adapter.md keeps "only Actions runs whose head_sha is a default-branch commit of the window", which says nothing about the event.

Question: which events make a run a verification of the revision? A defensible reading: push and merge_group are, event types that react to issues, comments, labels or pull-request metadata are not, and schedule / workflow_dispatch need a decision (a nightly test run on the head is verification; a nightly pins or stale-bot job is not, and nothing in the payload tells them apart). Whatever the answer, it is an outcome-map-like versioned table, and adopting it versions the Integrity rule.

2. A dormant repository with external CI is exactly UNINSTRUMENTED

ci.configured is false "only when there are zero workflows and no check suite on every sampled revision of the window". When no commit landed in the 14-day window, nothing is sampled and the condition holds vacuously: a repository whose head carries a failing CircleCI suite, and which simply had no commit for two weeks, is UNINSTRUMENTED / AVAILABLE / EXACT ("No automated verification is configured"). Reproduced against the fake transport: zero check-suite requests, ci.configured AVAILABLE false, Integrity UNINSTRUMENTED EXACT.

That contradicts observations.md ("zero, empty and false are only ever positively observed values"). The review PR already stops ci.configured from being a false over a revision the budget refused; the vacuous case needs a decision: UNKNOWN (NO_REVISION_SAMPLED), or sample the default-branch head's suites even outside the window.

Related, for calibration

The same repository reads integrity NO_RECENT_RUNS since 2026-09-22, correctly under the spec (no revision in the window), while 27 green scheduled runs sit on its head. A quiet but green mainline and a mainline whose CI never runs look alike; if the answer to question 1 excludes scheduled runs, that is the right reading, and the band name is worth a sentence in vitals.md either way.

🤖 Generated with Claude Code

Activity

  1. added
    for:researcherSpec disagreement or calibration finding to be read by the research process
    on Sep 30, 2026
  2. drevendev commented on Sep 30, 2026

    @drevendev
    OwnerAuthor

    Numbers from the 0.2.0 verification run (#53; a read-only run of the fleet into a copy of the store, 2026-09-30 22:00 UTC). drevendev/trade_simulation's default branch carried, on 12 revisions in 14 days: 834 workflow_dispatch, 89 workflow_run, 24 push, 23 schedule, 19 issues and 11 dynamic runs. Under 0.2.0 each of them is a verification parent of its revision and is carried from bundle to bundle as durable history (stored grouped by shape, 91 KB of snapshot for that project), and the listing now reaches the provider's 1,000-result cap (#51). The answer to question 1 here decides both how much that history holds and whether the cap is reached at all.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    for:researcherSpec disagreement or calibration finding to be read by the research process

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions