Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 8 additions & 58 deletions scripts/apply-baseline.sh
Original file line number Diff line number Diff line change
Expand Up @@ -82,33 +82,13 @@ echo "$BASELINE_JSON" | jq -e 'type == "array"' >/dev/null || {
SCHEMA_ERRORS="$(jq -r '
def known: ["severity","rule_module","type","file","file_pattern",
"severity_override","expires_at","note","tracking_issue"];
# Scanner vocabulary, mirroring the hypatia-scan-reusable validator: the
# research rules emit `warn` (ranked with medium) and some rules emit
# `informational`. Without these, research findings can never be
# acknowledged and the baseline gate can never go green on them.
def sevs: ["critical","high","medium","warn","low","info","informational"];
def sevs: ["critical","high","medium","low","info"];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore every severity accepted by the scanner

The inspected Hypatia workflow accepts warn and informational, and the authoritative baseline schema permits both, but this validator now rejects them. The repository's own .hypatia-baseline.json contains numerous warn entries, so whenever a nonempty scan invokes this script, it exits 2 before filtering any findings and breaks the baseline-backed gate.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you deal with this @CodeRabbit

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hyperpolymath I’ll route this fix to the coding workflow.

  • Restore warn and informational in the sevs list in scripts/apply-baseline.sh.
  • Check that the validator matches the authoritative baseline schema and scanner vocabulary.
  • Add regression tests for both aliases, including suppression, severity overrides, and blocking ranks.

The fix is not yet implemented or verified.


Failed to start Coding Agent task. Please try again.

You are interacting with an AI system.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please implement, @CodeRabbit

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hyperpolymath I’ll retry the coding workflow for this fix. The previous attempt did not start.

  • Restore warn and informational in the baseline validator.
  • Check consistency with the baseline schema and scanner.
  • Add regression tests and run them in the coding workflow.

The fix is not yet implemented or verified.


Failed to start Coding Agent task. Please try again.

You are interacting with an AI system.

[ to_entries[] | .key as $i | .value as $e |
if ($e|type) != "object" then "entry[\($i)]: not an object"
else (
(["severity","type"][]
(["severity","rule_module","type"][]
| select(($e[.]|type) != "string")
| "entry[\($i)]: required key \(.) missing or not a string"),
Comment on lines +89 to 91

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Honor array-valued rule modules

The authoritative schema and format documentation allow rule_module to be a nonempty string array so one acknowledgement can cover the same defect emitted by multiple modules. Requiring this field to be a string rejects those valid baselines with exit 2; validation must retain the array form, and matching must test membership rather than exact array-to-string equality.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please address @CodeRabbit

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📌 Tracking this review finding. CodeRabbit will verify future changes against the original feedback and will only mark it fixed when the code satisfies it.

# `rule_module` is the one key that admits a LIST as well as a string
# (standards#966: one defect, two emitting modules, one acknowledgement).
# Validated separately rather than being dropped from the required set —
# a key that is merely absent from every check is not validated, it is
# unchecked.
(if ($e.rule_module|type) as $t | $t != "string" and $t != "array"
then "entry[\($i)]: required key rule_module missing or not a string/array"
else empty end),
(if ($e.rule_module|type) == "array" and ($e.rule_module|length) == 0
then "entry[\($i)]: rule_module list is empty — an entry that names no module matches nothing"
else empty end),
(if ($e.rule_module|type) == "array"
then ($e.rule_module[]
| select(type != "string")
| "entry[\($i)]: rule_module list member is not a string: \(tojson)")
else empty end),
(if (($e|has("file")) == ($e|has("file_pattern")))
then "entry[\($i)]: exactly one of file / file_pattern is required"
else empty end),
Expand All @@ -117,12 +97,6 @@ SCHEMA_ERRORS="$(jq -r '
| "entry[\($i)]: unknown key \(.)"),
(if ($e.severity|type) == "string" and ((sevs|index($e.severity))|not)
then "entry[\($i)]: invalid severity \($e.severity)" else empty end),
(if ($e.rule_module|type) == "array"
then ($e.rule_module[]
| select(type == "string")
| select((test("^[a-z][a-z0-9_]*$"))|not)
| "entry[\($i)]: rule_module list member fails pattern: \(.)")
else empty end),
(if ($e.rule_module|type) == "string"
and (($e.rule_module|test("^[a-z][a-z0-9_]*$"))|not)
then "entry[\($i)]: rule_module fails pattern: \($e.rule_module)"
Expand Down Expand Up @@ -206,35 +180,12 @@ ANNOTATED="$(jq -n \
# referencing `.file_pattern` there would error with "Cannot index
# string". Capture the entry pattern first, then reference $pat
# inside the test() regex argument.
# Hypatia emitted `warn` for advisory findings until hypatia#895
# (2026-10-01), which now emits them as `medium`. Baselines written before
# then acknowledge `warn`; treating the two as one tier keeps those
# acknowledgements matching instead of re-reporting every one as new.
def sev_tier: if . == "warn" then "medium" else . end;
def match_entry(f):
f as $finding
| $baseline
| map(select(
(.severity | sev_tier) == ($finding.severity | sev_tier)
# `rule_module` is a string OR a list of strings.
#
# ⚠ THE LIST FORM EXISTS BECAUSE ONE DEFECT CAN BE EMITTED BY TWO
# MODULES. Hypatia raises `invalid_actions_lock` from BOTH
# `workflow_audit` and `workflow_hardening` for a single desynced
# lockfile. Under exact string equality an acknowledgement could
# only ever name one of them, so the other stayed unsuppressed and
# went on blocking — and it did, on `main`. The symptom is
# especially misleading: the entry looks correct, the file matches,
# the severity matches, and the finding is still kept.
#
# Normalising to a list here rather than duplicating the entry keeps
# ONE acknowledgement per defect, which is what `expires_at` and
# `tracking_issue` are actually about. Two entries for one defect
# means two expiry dates for one decision.
and (
(.rule_module | if type == "array" then . else [.] end)
| any(. == $finding.rule_module)
)
.severity == $finding.severity

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve warn-to-medium baseline compatibility

For legacy acknowledgements written with warn, Hypatia now emits the same findings as medium; exact equality therefore marks an acknowledged finding as new and can fail a blocking gate. The existing baseline regression suite explicitly covers matching these two representations in both directions, so the severity-tier normalization should remain.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good thinking @CodeRabbit, can you address it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hyperpolymath I’ll start the code-change workflow to restore warn ↔ medium compatibility.

  • Restore severity-tier normalisation in scripts/apply-baseline.sh.
  • Ensure baseline validation accepts legacy warn entries and stays aligned with .machine_readable/hypatia-baseline.schema.json.
  • Check regression coverage for both matching directions and blocking-gate behaviour.

Failed to start Coding Agent task. Please try again.

You are interacting with an AI system.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please address, @CodeRabbit

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hyperpolymath I’ll retry the code-change workflow with the following scope.

  • Restore warn ↔ medium baseline matching.
  • Accept legacy warn baseline entries in both validation layers.
  • Verify regression coverage and blocking-gate behaviour.

Failed to start Coding Agent task. Please try again.

You are interacting with an AI system.

and .rule_module == $finding.rule_module
and .type == $finding.type
and (
(.file? // null) == $finding.file
Expand Down Expand Up @@ -277,8 +228,9 @@ ANNOTATED="$(jq -n \
KEPT="$(jq '[.[] | select(.baseline_status != "acknowledged")]' <<<"$ANNOTATED")"
SUPPRESSED="$(jq '[.[] | select(.baseline_status == "acknowledged")]' <<<"$ANNOTATED")"

# Severity rank for blocking decision.
# Unknown severities are treated as critical to fail-safe.
# Severity rank for blocking decision. `warn` and `informational` are treated
# as aliases of `medium` and `info` respectively (Hypatia's research rules use
# `warn` at medium rank). Unknown severities are treated as critical to fail-safe.
rank() {
case "$1" in
critical) echo 5 ;;
Expand All @@ -293,11 +245,9 @@ rank() {

THRESHOLD_RANK="$(rank "$BLOCKING_THRESHOLD")"
MAX_KEPT_RANK=0
BLOCKING_COUNT=0 # kept findings at or above the threshold (the gate message names these, not all kept)
while IFS= read -r sev; do
r="$(rank "$sev")"
if (( r > MAX_KEPT_RANK )); then MAX_KEPT_RANK=$r; fi
if (( r >= THRESHOLD_RANK )); then BLOCKING_COUNT=$((BLOCKING_COUNT + 1)); fi
done < <(jq -r '.[].severity' <<<"$KEPT")

KEPT_COUNT="$(jq 'length' <<<"$KEPT")"
Expand Down Expand Up @@ -336,7 +286,7 @@ if [[ "$MODE" == "advisory" ]]; then
fi

if (( MAX_KEPT_RANK >= THRESHOLD_RANK )); then
echo "::error::Gate failed: $BLOCKING_COUNT unfiltered finding(s) at or above '$BLOCKING_THRESHOLD' ($KEPT_COUNT kept in total)." >&2
echo "::error::Gate failed: $KEPT_COUNT unfiltered finding(s) at or above '$BLOCKING_THRESHOLD'." >&2

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Count only findings meeting the blocking threshold

When kept findings contain mixed severities, this message reports every kept finding as being at or above the threshold. For example, one high plus one info at a high threshold reports two blocking findings even though only one qualifies, producing misleading gate diagnostics; retain a separate at-or-above-threshold count.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, @CodeRabbit can you fix this one too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📌 Tracking this review finding. CodeRabbit will verify future changes against the original feedback and will only mark it fixed when the code satisfies it.

exit 1
fi

Expand Down
14 changes: 4 additions & 10 deletions scripts/plan-ruleset-constraint-repair.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,10 @@ module RulesetConstraintRepair
RETIRED = %w[update required_deployments code_quality code_coverage].freeze
WRITABLE = %w[name target enforcement conditions bypass_actors rules].freeze

# Build a new ruleset update body with the four retired constraints removed,
# preserving the other writable fields and leaving +source+ unchanged.
#
# When +uninstalled_app_id+ is supplied, also remove only that integration's
# bypass. Supply an ID only after GitHub rejects it as no longer installed;
# never infer app availability from its name.
#
# Raises ArgumentError unless the source is an active default-branch ruleset
# with pull-request protection, or if the requested integration ID is invalid
# or absent. A supplied ID also requires +source+ to contain +bypass_actors+.
# Build a PUT-ready ruleset body from a live ruleset: strip the retired
# constraint rules and, optionally, a confirmed-uninstalled app bypass.
# Supply an app ID only after GitHub rejects it as no longer installed. Removing
# a stale bypass tightens access; never infer app availability from its name.
def self.plan(source, uninstalled_app_id: nil)
unless source.is_a?(Hash) && source['target'] == 'branch' && source['enforcement'] == 'active' &&
source.dig('conditions', 'ref_name', 'include') == ['~DEFAULT_BRANCH'] &&
Expand Down
22 changes: 8 additions & 14 deletions scripts/tests/science-ci-security-test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,20 @@
PUBLIC_KEY = "gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf\n"
FINGERPRINT = 'SHA256:eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8'

# Raise with the given message unless condition is truthy.
def assert(condition, message)
raise message unless condition
end

# Run a command via Open3, asserting it exits successfully, and return its
# stripped stdout.
def run!(*args, **options)
out, err, status = Open3.capture3(*args, **options)
assert(status.success?, "Command failed: #{args.inspect}\n#{out}#{err}")
out.strip
end

# Load and parse a workflow YAML file from .github/workflows by name.
def workflow(name)
YAML.safe_load(File.read(File.join(ROOT, '.github/workflows', name)), aliases: true)
end
Expand Down Expand Up @@ -99,25 +103,15 @@ def workflow(name)
output = File.join(tmp, 'output')
env = { 'GITHUB_OUTPUT' => output, 'GITHUB_STEP_SUMMARY' => File.join(tmp, 'summary') }
findings = File.join(tmp, 'hypatia-findings.json')
# Clean scan (single well-formed empty array) is a positive control (standards#1054)
File.write(findings, '[]')
run!(env, 'bash', '-c', step.fetch('run'), chdir: tmp)
assert(File.read(output).lines.map(&:chomp).include?('findings_count=0'), 'clean scan findings_count was not 0')
assert(File.read(output).lines.map(&:chomp).include?('critical=0'), 'clean scan critical was not 0')
assert(File.read(output).lines.map(&:chomp).include?('high=0'), 'clean scan high was not 0')
assert(File.read(output).lines.map(&:chomp).include?('medium=0'), 'clean scan medium was not 0')

# The validation expects: [finding1, finding2, ...] - single flat array of findings
# Use valid severities: critical, high, medium, warn, low, info, informational
FileUtils.rm_f(output)
# The validation expects: [finding1, finding2, ...] - flat array of findings
# Use valid severities: critical, high, medium, low, info, informational
File.write(findings, '[{"severity":"high"},{"severity":"medium"},{"severity":"critical"}]')
run!(env, 'bash', '-c', step.fetch('run'), chdir: tmp)
assert(File.read(output).lines.map(&:chomp).include?('medium=1'), 'medium was not counted')
assert(File.read(output).include?('critical=1'), 'critical finding was lost')
assert(File.read(output).include?('high=1'), 'high finding was lost')
# Test invalid inputs - empty, whitespace, truncated, wrong top-level type,
# invalid element shape, unknown severity, and multiple JSON documents
['', " \n", '[', '{}', 'null', '[{}]', '[{"severity":"unknown"}]', '[] []', "[]\n[{\"severity\":\"high\"}]"].each do |invalid|
# Test invalid inputs - flat array format
['', '[', '[]', '[{}]', '[{"severity":"unknown"}]'].each do |invalid|

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep an empty findings array as a valid control

A clean scan legitimately produces [], and the workflow under test explicitly accepts it and emits zero counts. Adding [] to this invalid-input loop therefore makes science-ci-security-test.rb fail every time; the inspected self-test.yml discovers and runs its shell wrapper, so this change leaves the required self-test job red.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, please address this @CodeRabbit

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📌 Tracking this review finding. CodeRabbit will verify future changes against the original feedback and will only mark it fixed when the code satisfies it.

FileUtils.rm_f(output)
File.write(findings, invalid)
_out, _err, status = Open3.capture3(env, 'bash', '-c', step.fetch('run'), chdir: tmp)
Expand Down
Loading