Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

### Changes

- **`checks.ignore` no longer turns a configured advisory gate green while the decision says `blocked`.** A suppression cannot waive a mandatory current-surface control (the four dedicated control families, such as `SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING`, and engine-raised control-pack obligations), so the release decision kept such a finding as a blocker; the exit code dropped it with every other suppressed finding before matching `fail_on`. `samples/support_refund_agent` in advisory mode with `fail_on: [critical]` and three `checks.ignore` entries printed `Decision: blocked`, `would_fail_ci=false` and exited `0`; without the entries it exited `20`. The exit code now also reads the decision's `blockers[]`: a blocker whose severity is in `fail_on` (including `verify`'s base `ci.fail_on` floor) exits `20` in either mode, for `scan`, `verify` and the Action, and `fail_policy.exit_code` / `would_fail_ci` in `report.json` and `verifier.json` say the same. Strict-mode exit codes are unchanged, because a blocker already meant a `blocked` decision, which strict mode fails; advisory mode without `fail_on` still exits `0`; and a suppression still waives every other finding. A report whose blocker in `fail_on` would not fail CI is refused when it is written. Separately, `verify-run.json`'s outcome, the verification receipt and the safety qualification script now read the decision-to-merge-verdict projection from `schemas/verifier.py` instead of carrying their own copies of its table, which would have refused every verify run the day that table changed; a test refuses a new copy. Migration: [STABILITY.md](STABILITY.md#release-decision-truth-table). (#1029)
- **A Claude Code setting that runs a command, or an `env` entry that redirects requests or loads code, is a host-grant row, and a settings change no row shows is no longer "no change".** Adding `"statusLine": {"command": "curl … | sh"}`, `"apiKeyHelper": "curl … | sh"`, `env.ANTHROPIC_BASE_URL` or `env.NODE_OPTIONS` to `.claude/settings.json` gave no row: `diff` (text and `--json`), `verify`'s host comparison and the PR comment headlined `No static host-grant changes detected. No verdict is implied.`, and `audit --host` reported Claude Code coverage `complete`. Host-grants `0.9` (extended in place) now publishes `command_setting` grants for `statusLine`, `subagentStatusLine`, `fileSuggestion`, `apiKeyHelper`, `otelHeadersHelper`, `awsAuthRefresh` and `awsCredentialExport` (`high`, the command shown as a hook's is: name, digest and shape, never text) and `env` grants for endpoint, proxy, certificate, search-path and loader names (`*_BASE_URL`, OpenTelemetry exporter endpoints, `HTTPS_PROXY` and the other proxies, `NODE_EXTRA_CA_CERTS`, `NODE_TLS_REJECT_UNAUTHORIZED`, `SSL_CERT_FILE`, `NODE_OPTIONS`, `BASH_ENV`, `PATH`, `PYTHONPATH`, `LD_PRELOAD`, `GIT_SSH_COMMAND` and the other names `docs/host-boundary-support.md` lists, read without regard to case; `high`, the value digested and never published), so adding one is a `⚠ high added` row and editing one a `changed` row naming the two digests. A settings file's other keys are published as display-only `unmodelled_keys` with digests; when one changes with no row, the coverage item names it in `detail` and the headline reads `Changed, not compared: .claude/settings.json (keys this entry does not compare: …)`, as it does, without keys, for any settings, MCP or hook file that changed with no row. A key no reader models that is shaped as a command (an object with a `command`, or a `…Helper` string) is a blocking limit, so `audit --host` reports coverage `partial`, as for `policyHelper`. The cold-start benchmark's oracle moves those seven settings and those `env` names from unsupported to supported. Migration: [STABILITY.md](STABILITY.md#host-commands-env-1027). (#1027)
- **An n8n MCP Client Tool that exposes every tool of its server is read as doing so.** n8n's node selects its tools with `include` (`all`, `selected` or `except`) and leaves it out of an exported workflow at its default, `all`; the reader recognised only `toolSelection`, `toolsToInclude` and `toolSelectionMode`, which n8n does not write. So the default export, `include: all` and `include: except` each entered the catalog as one concretely named tool, with no wildcard, no dynamic surface and no `SHIP-N8N-MCP-CLIENT-TOOLSET-UNFILTERED`. Each is now one `<node>.*` action, which raises `SHIP-INVENTORY-WILDCARD-TOOLS` and, with no local inventory declared, `SHIP-N8N-DYNAMIC-TOOL-SURFACE-NOT-ENUMERABLE` and the unfiltered finding (`high` in `production_like` and `production`, `medium` otherwise); an `except` node's `excludeTools` are listed in the finding's `excluded_tools`; `selected` names its `includeTools`, and a `selected` node naming none is read as `all`, as n8n runs it. The earlier spellings remain aliases, and `include` decides when both are present. A workflow whose MCP Client Tool keeps n8n's default now reports these findings where it passed; select explicit tools in the node, or declare a local MCP inventory. Migration: [STABILITY.md](STABILITY.md#mcp-reader-conformance-1025). (#1025)
- **`SHIP-MCP-ENV-SECRET-PASSTHROUGH` counts a secret an MCP server forwards by name, not only an `env` entry.** A secret-like variable in Codex's `env_vars` list or `env_http_headers`, any `bearer_token_env_var`, and a `${NAME}` reference in a `headers` value (`.mcp.json`) now fire the finding (an editor's `${env:NAME}` is read as `NAME` there and in `env`, and a prompted `${input:…}` names no variable) and add to the server's risk score as an `env` entry always did. A credential-named header (`Authorization`) counts the variable it carries whatever that variable is called, and a value written where a variable name belongs is redacted in the evidence, not echoed. (#1025)
Expand Down
9 changes: 6 additions & 3 deletions STABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -5166,7 +5166,7 @@ These are **intentionally different signals**, kept apart for backwards compatib

#### Release decision truth table

The classification below is the contract for how every active finding lands in `release_decision.{blockers, review_items}[]` and which `contribution_rules[].rule` (v0.17+) fires for it. Starting in v0.33, a finding with typed `support` is considered active for release contribution only when `support.policy_eligible=true`; otherwise it is excluded with `rule="unsupported_evidence"` before severity or `blocks_release` is consulted. Suppressed findings are excluded with `rule="suppressed"` after that evidence-eligibility check. Legacy findings without typed support retain the table's established behavior.
The classification below is the contract for how every active finding lands in `release_decision.{blockers, review_items}[]` and which `contribution_rules[].rule` (v0.17+) fires for it. Starting in v0.33, a finding with typed `support` is considered active for release contribution only when `support.policy_eligible=true`; otherwise it is excluded with `rule="unsupported_evidence"` before severity or `blocks_release` is consulted. Suppressed findings are excluded with `rule="suppressed"` after that evidence-eligibility check, except a mandatory current-surface control (the four dedicated control families and engine-raised control-pack obligations): `checks.ignore` records it but cannot waive it, so it stays a blocker with `rule="policy_block_new"` whatever its baseline status (the last row). Legacy findings without typed support retain the table's established behavior.

Notation: `fail_on` is `release_decision.fail_policy.fail_on` after `ci_mode` resolution (advisory → empty, strict → `["critical"]`, plus any explicit `--fail-on` override). `blocker_severities` = `{critical} ∪ fail_on`. `review_tier` = `{critical, high, medium}` (or any severity when `requires_human_review=true`).

Expand All @@ -5182,14 +5182,17 @@ Notation: `fail_on` is `release_decision.fail_policy.fail_on` after `ci_mode` re
| false | any | new / null | no | yes | review_item | `review_required` | 0 |
| false | any | matched | no | yes | review_item | `review_required` | 0 |
| false | any | new / null / matched | no | no | excluded | `sub_threshold` | 0 |
| suppressed mandatory current-surface control | any | new / null / matched | n/a | n/a | **blocker** | `policy_block_new` | 20 |

**Why baseline-matched policy findings drop to `review_items`, not `blockers`.** `blocks_release=true` represents an explicit *policy* decision (Action Surface Diff rule, `action_surface:` manifest entry, or policy-pack rule with `block: true`) that the finding must block release **on first appearance**. A baseline accepts technical debt that already passed prior review — the project agreed to ship with that finding present. Treating baselined policy debt as a hard blocker would defeat the purpose of `baseline save`. The baseline-aware drop is symmetric for severity-driven blockers and policy blockers: both land in `review_items` once accepted into the baseline, both become hard blockers if newly introduced.

**Why `severity ∈ blocker_severities + matched + below review_tier` lands in `excluded`, not `review_items`.** A finding whose severity isn't in `{critical, high, medium}` (and which doesn't carry `requires_human_review=true`) has nothing for a human reviewer to act on per the v0.8 contract — it's been baselined and isn't severe enough to warrant attention. v0.17 records this in the audit so the (rare) edge case isn't silently invisible, but the `blockers[]`/`review_items[]` lists themselves are unchanged.

**Why exit code 20 depends on `--baseline-mode`.** `release_decision.{blockers, review_items}[]` always include the full set computed against `report.findings` (with suppressed excluded). The strict-mode exit code, however, is computed from `baseline_filtered_active(report, new_findings_only=...)` — when `--baseline-mode new-findings` is set (the default for the GitHub Action when `baseline:` is provided), baseline-matched policy and severity blockers are filtered out before the exit check, so exit is `0`. With `new_findings_only=False`, a matched policy blocker still triggers exit 20. The `release_decision` block remains baseline-aware in all cases; only the exit-code path changes mode.
**Why exit code 20 depends on `--baseline-mode`.** `release_decision.{blockers, review_items}[]` always include the full set computed against `report.findings` (with suppressed findings excluded, except a mandatory current-surface control, which stays a blocker). The exit code's finding check, however, is computed from `baseline_filtered_active(report, new_findings_only=...)` — when `--baseline-mode new-findings` is set (the default for the GitHub Action when `baseline:` is provided), baseline-matched policy and severity blockers are filtered out before the exit check, so exit is `0`. With `new_findings_only=False`, a matched policy blocker still triggers exit 20. The `release_decision` block remains baseline-aware in all cases; only the exit-code path changes mode.

**`verify`'s base `fail_on` floor (#1024).** With a base and without `--fail-on`, `verify` passes the base manifest's declared `ci.fail_on` to the exit computation only. The exit column then also reads `20` for any row whose finding is active after the same baseline filtering and whose severity is in that list, in either CI mode. The category and `rule` columns do not move, because `blocker_severities` stays `{critical} ∪ fail_policy.fail_on`, the change's own list.
**The exit code reads the decision's blockers (#1029).** Besides that finding check, the exit code is `20` whenever a `release_decision.blockers[]` entry has a severity in `fail_on`, in either CI mode. Suppression removes a finding from the finding check, so this is what keeps a suppressed mandatory current-surface control failing a configured `fail_on`: advisory mode with `fail_on: [critical]` exits `20` on the blocked decision such a control produces, as strict mode already did through its `blocked` rule. In strict mode the rule adds nothing: a non-empty `blockers[]` is a `blocked` decision, which already exits `20`. A suppression still waives every other finding, in both modes.

**`verify`'s base `fail_on` floor (#1024).** With a base and without `--fail-on`, `verify` passes the base manifest's declared `ci.fail_on` to the exit computation only. The exit column then also reads `20` for any row whose finding is active after the same baseline filtering, or is a blocker, and whose severity is in that list, in either CI mode. The category and `rule` columns do not move, because `blocker_severities` stays `{critical} ∪ fail_policy.fail_on`, the change's own list.

Concretely: a scan with one baseline-matched critical and zero new findings produces `summary.status = "release_blockers_detected"` AND `release_decision.decision = "review_required"`. Both are correct under their respective contracts. New consumers should read `release_decision.decision`.

Expand Down
11 changes: 4 additions & 7 deletions scripts/run_safety_qualification.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
tier_for_requirements,
)
from agents_shipgate.schemas.verification_identity import VerificationReceipt
from agents_shipgate.schemas.verifier import VerifierArtifact
from agents_shipgate.schemas.verifier import VerifierArtifact, map_merge_verdict
from agents_shipgate.schemas.verify_run import VerifyRunArtifact

if __package__:
Expand All @@ -83,12 +83,9 @@
"blocked",
)
QUALIFYING_ORIGINS = {"real_history", "rejected_or_reverted", "design_partner"}
EXPECTED_MERGE_VERDICT = {
"passed": "mergeable",
"review_required": "human_review_required",
"insufficient_evidence": "insufficient_evidence",
"blocked": "blocked",
}
# The verifier's projection, read from its one authority rather than copied:
# a copy would reject every receipt the day that authority changes (#1029).
EXPECTED_MERGE_VERDICT = {decision: map_merge_verdict(decision) for decision in DECISIONS}

# Match the receipt reader's per-artifact bound; wheels have their own bound
# because they contain the whole distribution. These are input limits, never
Expand Down
30 changes: 28 additions & 2 deletions src/agents_shipgate/ci/exit_policy.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
from __future__ import annotations

from collections.abc import Sequence

from agents_shipgate.schemas.common import ReleaseDecisionStatus, Severity
from agents_shipgate.schemas.report import (
Finding,
ReadinessReport,
ReleaseDecisionItem,
)

GATE_FAILURE_EXIT_CODE = 20
Expand All @@ -25,6 +28,10 @@ def baseline_filtered_active(report: ReadinessReport, *, new_findings_only: bool
Findings with typed support therefore contribute to process policy only
when that support is policy-eligible. Legacy findings without support keep
their established behavior for frozen-report compatibility.

Suppression drops a finding here, but it does not drop a blocker the
decision engine keeps non-waivable: those reach the exit code through the
``blockers`` argument of :func:`exit_code_for_report` (#1029).
"""
active = [
finding
Expand All @@ -41,13 +48,22 @@ def exit_code_for_report(
report: ReadinessReport,
ci_mode: str,
*,
blockers: Sequence[ReleaseDecisionItem],
fail_on: list[Severity] | None = None,
new_findings_only: bool = False,
release_decision: ReleaseDecisionStatus | None = None,
has_semantic_gaps: bool = False,
) -> int:
# Summary counts exclude suppressed findings, so strict mode intentionally
# passes when every critical finding has an explicit suppression reason.
"""The process exit code for a scan, read from the one decision engine.

``blockers`` and ``release_decision`` are what ``build_release_decision``
computed for this report; they are required so that no caller can derive
an exit code beside the decision instead of from it. The exit code is not
a projection of the decision alone (see STABILITY.md "Exit codes"): in
advisory mode it is 20 only when the configured ``fail_on`` matches. But
when it does match, it matches what the decision blocks on, including a
blocker no suppression can waive.
"""
fail_on_resolved = effective_fail_on(ci_mode, fail_on)
active = baseline_filtered_active(report, new_findings_only=new_findings_only)
# v0.29: the semantic gate is not represented as a Finding on purpose —
Expand All @@ -63,6 +79,16 @@ def exit_code_for_report(
return GATE_FAILURE_EXIT_CODE
if not fail_on_resolved:
return 0
# A suppression explains a waivable finding, so a suppressed finding
# leaves `active` and no longer matches `fail_on`. It cannot waive a
# mandatory current-surface control: the decision engine keeps that one
# as a blocker, and the exit code reads the engine's blockers rather than
# restating which suppressions count (#1029). Without this, advisory mode
# with `fail_on` exited 0 on a report whose decision was `blocked` by a
# blocker in that very `fail_on`. Strict mode already exits above on any
# `blocked` decision, so this changes no strict exit code.
if any(item.severity in fail_on_resolved for item in blockers):
return GATE_FAILURE_EXIT_CODE
if any(finding.severity in fail_on_resolved for finding in active):
return GATE_FAILURE_EXIT_CODE
return 0
Expand Down
8 changes: 6 additions & 2 deletions src/agents_shipgate/ci/release_decision.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,10 +206,13 @@ def build_release_decision(
# blockers/review_items consider the full findings set, NOT
# new_findings_only: baseline-matched criticals must remain visible
# as accepted debt in review_items. The new_findings_only filter
# only affects fail_policy.exit_code (via exit_code_for_report).
# only affects fail_policy.exit_code (via exit_code_for_report, which
# also reads these blockers, so a blocker no suppression can waive
# still fails a matching fail_on — #1029).
# v0.17: iterate report.findings directly so the contribution_rules
# audit row set is exhaustive (suppressed findings get an audit row
# too, classified as excluded/suppressed).
# too, classified as excluded/suppressed, or as a blocker when the
# suppression cannot waive a mandatory current-surface control).
blockers: list[ReleaseDecisionItem] = []
review_items: list[ReleaseDecisionItem] = []
contribution_rules: list[ContributionRule] = []
Expand Down Expand Up @@ -453,6 +456,7 @@ def build_release_decision(
exit_code = exit_code_for_report(
report,
ci_mode,
blockers=blockers,
fail_on=exit_fail_on,
new_findings_only=new_findings_only,
release_decision=decision,
Expand Down
10 changes: 10 additions & 0 deletions src/agents_shipgate/core/semantic_consistency.py
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,16 @@ def validate_semantic_consistency(
raise SemanticConsistencyError(
"strict insufficient_evidence must produce would_fail_ci=true, exit_code=20"
)
# The exit code reads the decision's blockers (#1029): a blocker whose
# severity is in fail_on fails CI in either mode, whatever suppressed the
# finding behind it, and strict mode fails on any blocked decision.
if (
any(item.severity in decision.fail_policy.fail_on for item in decision.blockers)
or (decision.fail_policy.ci_mode == "strict" and decision.decision == "blocked")
) and (decision.fail_policy.would_fail_ci is not True or decision.fail_policy.exit_code != 20):
raise SemanticConsistencyError(
"a release blocker in fail_on must produce would_fail_ci=true, exit_code=20"
)


def _validate_exclusion_ledger(report: ReadinessReport) -> None:
Expand Down
Loading
Loading