From 3673d7126fcf1fddf208801df040b4daeebdf4bc Mon Sep 17 00:00:00 2001 From: Claude Code Bot Date: Wed, 21 Oct 2026 13:08:47 -0700 Subject: [PATCH] feat(digest): survey CodeQL default setup in the daily digest standards.sh already walks every non-archived repo once per owner. It now also reads code-scanning/default-setup for each one and records a codeql object: configured, not-configured (the finding), no-code-security, exempt with a class, or unreadable. render.sh adds a "CodeQL" section in the same style as "Branch protection": exempt classes counted with their reason, not-configured and unreadable repos in a table, and a plain "could not be verified" statement when every read failed. no-code-security is a private repo whose 403 body says Code Security or Advanced Security must be enabled. It is not exempt: those repos are listed by name under their own heading, because buying Code Security is an open cost decision. The state needs that exact body message on a repo the listing says is private. Any other 403, including "Resource not accessible by integration", is unreadable. The message is matched on the response body, because stderr can carry unrelated text before gh's line. Exempt classes: forks, and not-configured with no detected languages. The endpoint needs Administration: read, so all three token mints now request it. GitHub rejects a token request for a permission the installation has not granted (not observed here), so the grant must be in place before this runs. Fixtures: dev-env (configured), nightowlstudiollc/cleanroom and smartwatermelon/claude-config-backup (403s) are real 2026-10-21 captures; the rest are a real body with one field changed. Advances #166 --- .github/workflows/dependabot-digest.yml | 6 ++ .prettierignore | 4 + CLAUDE.md | 2 +- docs/STATUS.md | 9 +- .../runbooks/dependabot-digest-credentials.md | 9 +- scripts/dependabot-digest/render.sh | 92 ++++++++++++++++++ scripts/dependabot-digest/standards.sh | 83 +++++++++++++++- ...nroom_code-scanning_default-setup.403.json | 5 + ...ivate_code-scanning_default-setup.403.json | 5 + ...llc_nostd_code-scanning_default-setup.json | 10 ++ ..._offcheck_code-scanning_default-setup.json | 10 ++ ...jcleaning_code-scanning_default-setup.json | 10 ++ ...s_o_clean_code-scanning_default-setup.json | 10 ++ ...ackup_code-scanning_default-setup.403.json | 5 + ...n_dev-env_code-scanning_default-setup.json | 10 ++ ...flags_code-scanning_default-setup.403.json | 5 + ..._withheld_code-scanning_default-setup.json | 1 + ...paper-MCP_code-scanning_default-setup.json | 10 ++ ...m-private_code-scanning_default-setup.json | 10 ++ ...ublic_code-scanning_default-setup.403.json | 5 + .../dependabot-digest/tests/test-render.sh | 11 ++- .../tests/test-run-digest.sh | 7 ++ .../dependabot-digest/tests/test-standards.sh | 94 +++++++++++++++++++ 23 files changed, 403 insertions(+), 10 deletions(-) create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_cleanroom_code-scanning_default-setup.403.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_nos-private_code-scanning_default-setup.403.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_nostd_code-scanning_default-setup.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_offcheck_code-scanning_default-setup.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_tnjcleaning_code-scanning_default-setup.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_o_clean_code-scanning_default-setup.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_claude-config-backup_code-scanning_default-setup.403.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_dev-env_code-scanning_default-setup.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_noflags_code-scanning_default-setup.403.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_withheld_code-scanning_default-setup.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_Instapaper-MCP_code-scanning_default-setup.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_tm-private_code-scanning_default-setup.json create mode 100644 scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_tm-public_code-scanning_default-setup.403.json diff --git a/.github/workflows/dependabot-digest.yml b/.github/workflows/dependabot-digest.yml index e177b98..17ff969 100644 --- a/.github/workflows/dependabot-digest.yml +++ b/.github/workflows/dependabot-digest.yml @@ -57,6 +57,8 @@ jobs: permission-contents: read permission-pull-requests: read permission-statuses: read + # Reads CodeQL default setup (dev-env#166). + permission-administration: read - name: Mint a token for nightowlstudiollc id: token-nightowlstudiollc @@ -69,6 +71,8 @@ jobs: permission-contents: read permission-pull-requests: read permission-statuses: read + # Reads CodeQL default setup (dev-env#166). + permission-administration: read - name: Mint a token for twistedmelonman id: token-twistedmelonman @@ -81,6 +85,8 @@ jobs: permission-contents: read permission-pull-requests: read permission-statuses: read + # Reads CodeQL default setup (dev-env#166). + permission-administration: read - name: Build and publish the digest env: diff --git a/.prettierignore b/.prettierignore index 75fd344..ead79f3 100644 --- a/.prettierignore +++ b/.prettierignore @@ -6,3 +6,7 @@ docs/data/corpus-440/files/ docs/data/corpus-440/results/ docs/data/corpus-440/files.tar.gz + +# Deliberately truncated JSON: the digest test that a malformed response is +# unreadable (dev-env#166). Prettier cannot parse it and blocks the commit. +scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_withheld_code-scanning_default-setup.json diff --git a/CLAUDE.md b/CLAUDE.md index 0115233..6b7548f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,7 +20,7 @@ This repo is the **dev-env infrastructure repository** — it contains documenta - `docs/token-rotation.md` — Retired 2026-10-12: no repo runs Claude in CI and `CLAUDE_CODE_OAUTH_TOKEN` is deleted; never contains a token - `docs/runbooks/fleet-probe-token-scopes.md` — The two fine-grained-PAT properties a fleet probe needs (`Administration: Read-only` + All-repositories), and why an under-scoped token returns wrong numbers instead of errors - `scripts/org-migration/` — Snapshot/transfer/verify tooling for the 2026-09 org migration; tests in `scripts/org-migration/tests/run-tests.sh` -- `scripts/dependabot-digest/` — Collects open Dependabot PRs across all three owners and upserts one digest issue describing the queue, plus each repo's standards-check warnings and default-branch protection (`standards.sh`); run by `.github/workflows/dependabot-digest.yml`. Credentials (a GitHub App) are installed by hand: see `docs/runbooks/dependabot-digest-credentials.md` +- `scripts/dependabot-digest/` — Collects open Dependabot PRs across all three owners and upserts one digest issue describing the queue, plus each repo's standards-check warnings, default-branch protection and CodeQL default setup (`standards.sh`); run by `.github/workflows/dependabot-digest.yml`. Credentials (a GitHub App) are installed by hand: see `docs/runbooks/dependabot-digest-credentials.md` - `scripts/site-monitor/` — Every 15 minutes, checks each production Netlify site in `sites.json` and its latest `site-check` run; keeps one `site-down` issue and pushes ntfy.sh alerts. Run by `.github/workflows/site-monitor.yml`, which a launchd job on MIMOLETTE starts through `dispatch-monitor.sh` because GitHub's cron never fired. The `NTFY_TOPIC` secret and the dispatch token are installed by hand: see `docs/runbooks/site-monitor-ntfy.md` - `.project-hooks/pre-commit` and `.project-hooks/pre-push` — Project-specific git hook extensions, run by the global hooks at `~/.config/git/hooks/` when executable - `.claude/` is ignored, as in every fleet repo (dev-env#178). The scaffold the post-checkout hook drops there (`config.sh.template`, `README.md`) comes from `smartwatermelon/dotfiles` `git/template/.claude-template/`; edit it there. diff --git a/docs/STATUS.md b/docs/STATUS.md index 8f771fa..7fc1f5a 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -911,8 +911,13 @@ scheduled run passed 2026-10-02, and plan step 7 (delete the three the digest also carries a "Standards warnings" section. Since 2026-10-21 it also carries a "Branch protection" section (closes `dev-env#211`): repos unprotected or not requiring standards-check, with -exempt classes counted. Whether the App token can read `.protection` is -unverified until the first run after merge. +exempt classes counted. The App token reads `.protection`: run +39661428846 (2026-10-21) surveyed 40 repos, 38 required, 2 exempt, 0 +unreadable. A "CodeQL" section (advances `dev-env#166`) reports repos with +default setup off, and lists by name the private repos where scanning is +off because paid Code Security is not bought (cost decision still open on +`#166`). Forks and repos with no scannable language are exempt classes. It +needs the App's Administration: read on all three installations. Earlier, all daily runs from 09-26 failed on `nightowlstudiollc` check-run access, so nothing was published. diff --git a/docs/runbooks/dependabot-digest-credentials.md b/docs/runbooks/dependabot-digest-credentials.md index 6e00bd2..3f43483 100644 --- a/docs/runbooks/dependabot-digest-credentials.md +++ b/docs/runbooks/dependabot-digest-credentials.md @@ -27,7 +27,14 @@ proof that this path works: a published digest means every check was read. - **Name**: `dependabot-digest-swm`, installed on `smartwatermelon`, `nightowlstudiollc` and `twistedmelonman`, **All repositories** each. - **Repository permissions**, all read-only: Checks, Contents, Metadata, - Pull requests, Commit statuses. Nothing else. Webhook inactive. + Pull requests, Commit statuses, Administration. Nothing else. Webhook + inactive. Administration is there only for the CodeQL section: + `GET repos/{owner}/{repo}/code-scanning/default-setup` needs + Administration: read (GitHub's "Permissions required for GitHub Apps" + table; dev-env#166, approved 2026-10-21). The workflow asks for it + when it mints each token. GitHub rejects a token request for a + permission the installation has not granted, so accept the grant on all + three installations before the workflow runs. - The private key lives in 1Password vault `Automation`, item `DIGEST_APP` (fields `client_id`, `private_key`). Use the attached `dependabot-digest-swm.2026-10-01.private-key.pem`, not the `private_key` diff --git a/scripts/dependabot-digest/render.sh b/scripts/dependabot-digest/render.sh index b850232..2fc12f5 100755 --- a/scripts/dependabot-digest/render.sh +++ b/scripts/dependabot-digest/render.sh @@ -348,9 +348,101 @@ render_protection() { fi } +# dev-env#166. As for protection: unreadable is never on or off, and an +# all-unreadable survey says so. +render_codeql() { + local file="$1" recs surveyed configured exempt flagged unreadable by_class owners rows + echo "## CodeQL" + echo + if [[ ! -r "${file}" ]] || ! jq -e -s 'type == "array"' "${file}" >/dev/null 2>&1; then + echo "**Not checked.** The survey produced no readable result, so no repository's CodeQL setup was verified." + echo + return 0 + fi + # A record without a codeql object (an older or crashed survey) is + # unreadable, not clean. + if ! recs="$(jq -cs ' + [.[] | select(type == "object" and (.repo | type == "string") and .state != "archived") + | .c = (if (.codeql | type) == "object" and (.codeql.state | type) == "string" + then .codeql + else {state: "unreadable", detail: "no CodeQL result recorded", exemptClass: null} end)] + ' "${file}" 2>/dev/null)" || [[ -z "${recs}" ]]; then + echo "**Not checked.** The survey output could not be summarized, so no repository's CodeQL setup was verified." + echo + return 0 + fi + owners="$(jq -rs '[.[] | select(type == "object" and .repo == null and .state == "unlisted") | .owner] | unique | join(", ")' "${file}" 2>/dev/null)" + surveyed="$(jq 'length' <<<"${recs}")" + configured="$(jq '[.[] | select(.c.state == "configured")] | length' <<<"${recs}")" + exempt="$(jq '[.[] | select(.c.state == "exempt")] | length' <<<"${recs}")" + flagged="$(jq '[.[] | select(.c.state == "not-configured")] | length' <<<"${recs}")" + nocs="$(jq '[.[] | select(.c.state == "no-code-security")] | length' <<<"${recs}")" + unreadable=$((surveyed - configured - exempt - flagged - nocs)) + + echo "Whether each repository has CodeQL default setup turned on. Exempt repositories are left off by a recorded decision for their class; they are counted, not listed." + echo + if [[ "${surveyed}" -eq 0 ]]; then + echo "**No repository was surveyed**, so no CodeQL setup was verified." + echo + elif [[ "${unreadable}" -eq "${surveyed}" ]]; then + # Zero gaps out of zero reads is not "all configured". + echo "**CodeQL setup could not be verified for any of ${surveyed} repositories.** Every read failed or returned no usable state. The likely cause is a token without \`Administration: Read\`, which the default-setup endpoint needs. Nothing here means CodeQL is on or off anywhere." + echo + else + echo "**${surveyed} repositories surveyed**: ${configured} configured, ${exempt} exempt, **${flagged} not configured, ${nocs} without Code Security, ${unreadable} unreadable.**" + echo + if [[ "${exempt}" -gt 0 ]]; then + by_class="$(jq -r '[.[] | select(.c.state == "exempt")] | group_by(.c.exemptClass) + | map("- \(.[0].c.exemptClass // "unknown class"): \(length) — \(.[0].c.detail // "no reason recorded")") | join("\n")' <<<"${recs}")" + echo "Exempt by class:" + echo + echo "${by_class}" + echo + fi + if [[ "${flagged}" -eq 0 && "${unreadable}" -eq 0 && "${nocs}" -eq 0 ]]; then + echo "No gaps: every surveyed repository has default setup on or is exempt." + echo + fi + fi + if [[ "${flagged}" -gt 0 || "${unreadable}" -gt 0 ]]; then + if ! rows="$(jq -r ' + def cell: tostring | gsub("[\r\n]+"; " ") | gsub("\\|"; "\\|"); + def name: if .c.state == "not-configured" then "NOT CONFIGURED" else "UNREADABLE" end; + sort_by((if .c.state == "not-configured" then 0 else 1 end), .repo)[] + | select(.c.state | IN("configured", "exempt", "no-code-security") | not) + | "| [" + .repo + "](https://github.com/" + .repo + ") | " + name + + (if .c.detail then ": " + (.c.detail | cell) else "" end) + " |" + ' <<<"${recs}")" || [[ -z "${rows}" ]]; then + echo "**Some repositories need attention, but listing them failed.** See the run log." + echo + else + echo "| Repo | CodeQL default setup |" + echo "| --- | --- |" + echo "${rows}" + echo + fi + fi + # Listed by name, not exempt: the cost decision is open (dev-env#166). + if [[ "${nocs}" -gt 0 ]]; then + rows="$(jq -r '[.[] | select(.c.state == "no-code-security") | .repo] | sort[] + | "- [" + . + "](https://github.com/" + . + ")"' <<<"${recs}")" + echo "### Scanning off: paid Code Security not bought" + echo + echo "GitHub refuses code scanning on these private repositories until the paid GitHub Code Security add-on is enabled. Whether to buy it is an open cost decision (dev-env#166)." + echo + echo "${rows:-**${nocs} repositories, but listing them failed.** See the run log.}" + echo + fi + if [[ -n "${owners}" ]]; then + echo "**Not listed, so not surveyed:** ${owners}. CodeQL setup of their repositories is unknown." + echo + fi +} + if [[ -n "${standards_file}" ]]; then render_standards "${standards_file}" render_protection "${standards_file}" + render_codeql "${standards_file}" fi echo "---" diff --git a/scripts/dependabot-digest/standards.sh b/scripts/dependabot-digest/standards.sh index 51bdd8f..2596a16 100755 --- a/scripts/dependabot-digest/standards.sh +++ b/scripts/dependabot-digest/standards.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Usage: standards.sh . Prints one JSON record per repo: standards-check # annotations, or the reason none were read, plus the default branch's -# protection state (dev-env#211). +# protection state (dev-env#211) and CodeQL default setup (dev-env#166). set -uo pipefail unset CDPATH @@ -35,7 +35,7 @@ emit() { --arg detail "$3" --arg source "${4-}" --arg url "${5-}" \ --argjson ann "${6:-[]}" --arg runAt "${7-}" \ --arg rulesAt "${RULES_AT}" --arg rulesErr "${RULES_ERR}" \ - --argjson prot "${PROT_JSON:-null}" \ + --argjson prot "${PROT_JSON:-null}" --argjson cq "${CQ_JSON:-null}" \ '{owner: $owner, repo: (if $repo == "" then null else $repo end), state: $state, detail: $detail, source: (if $source == "" then null else $source end), @@ -43,10 +43,10 @@ emit() { runAt: (if $runAt == "" then null else $runAt end), rulesAt: (if $rulesAt == "" then null else $rulesAt end), rulesErr: (if $rulesErr == "" then null else $rulesErr end), - annotations: $ann, protection: $prot}' 2>/dev/null)" - # If jq fails, the repo must still appear. Its protection is unknown. + annotations: $ann, protection: $prot, codeql: $cq}' 2>/dev/null)" + # If jq fails, the repo must still appear. Protection and CodeQL are unknown. if [[ -z "${line}" ]]; then - printf '{"owner":"%s","repo":"%s","state":"unreadable","detail":"could not encode the result","source":null,"runUrl":null,"runAt":null,"rulesAt":null,"rulesErr":null,"annotations":[],"protection":null}\n' \ + printf '{"owner":"%s","repo":"%s","state":"unreadable","detail":"could not encode the result","source":null,"runUrl":null,"runAt":null,"rulesAt":null,"rulesErr":null,"annotations":[],"protection":null,"codeql":null}\n' \ "${owner}" "$1" return 0 fi @@ -214,6 +214,75 @@ probe_protection() { set_prot "${state}" "${detail}" "${cls}" } +# dev-env#166: CodeQL default setup. Exempt classes are counted, not +# flagged. A private repo without paid Code Security is NOT exempt. +declare -A CQ_EXEMPT_REASON=( + [fork]="a fork is not ours to scan (decided 2026-10-21, dev-env#166)" + [no-supported-language]="GitHub detects no language default setup can scan (2026-10-19, dev-env#166)" +) +# The 403 body when the paid product is off. Any other 403 is a token gap. +# Matched on the body: stderr can carry noise. +CQ_NO_PRODUCT='^(Code|Advanced) Security must be enabled for this repository to use code scanning\.$' + +# Prints "statedetail". An odd field is unreadable, never a result. +read -r -d '' cq_verdict <<'JQ' +if type != "object" then "unreadable\tthe default-setup response is malformed" +elif .state == "configured" then "configured\t" +elif .state != "not-configured" then "unreadable\tunknown default-setup state: " + (.state | tostring) +elif (.languages | type) != "array" then "unreadable\tnot configured, and no language list was returned" +elif (.languages | length) == 0 then "no-language\t" +else "not-configured\tdefault setup is off; languages: " + (.languages | map(tostring) | join(", ")) +end +JQ + +CQ_JSON="" +set_cq() { + # set_cq [exempt class] + CQ_JSON="$(jq -cn --arg s "$1" --arg d "$2" --arg c "${3-}" \ + '{state: $s, detail: (if $d == "" then null else $d end), + exemptClass: (if $c == "" then null else $c end)}' 2>/dev/null)" + if [[ -z "${CQ_JSON}" ]]; then + CQ_JSON='{"state":"unreadable","detail":"could not encode the CodeQL result","exemptClass":null}' + fi +} +set_cq_exempt() { set_cq exempt "${CQ_EXEMPT_REASON[$1]-no reason recorded}" "$1"; } + +# Sets CQ_JSON for one repo. Never fails: a failed read is a state. +probe_codeql() { + local nwo="$1" fork="$2" private="$3" body why msg verdict state detail + if ! body="$(api "repos/${nwo}/code-scanning/default-setup")"; then + why="$(last_err "no detail")" + msg="$(jq -r 'if type == "object" then .message // "" else "" end' <<<"${body}" 2>/dev/null)" + # Only the exact product message, on a repo the listing says is + # private: a public repo has no paywall to explain it. + if [[ "${why}" == *"(HTTP 403)"* && "${msg}" =~ ${CQ_NO_PRODUCT} \ + && "${private}" == "true" ]]; then + set_cq no-code-security "${msg}" + else + set_cq unreadable "cannot read default setup: ${why}" + fi + return 0 + fi + verdict="$(jq -r "${cq_verdict}" <<<"${body}" 2>/dev/null)" + state="${verdict%%$'\t'*}" + detail="${verdict#*$'\t'}" + case "${state}" in + configured | unreadable) set_cq "${state}" "${detail}" ;; + no-language) set_cq_exempt no-supported-language ;; + not-configured) + # A fork is exempt; without a fork flag, exemption cannot be decided. + if [[ "${fork}" == "true" ]]; then + set_cq_exempt fork + elif [[ "${fork}" != "false" ]]; then + set_cq unreadable "not configured, but the repository listing gave no fork flag, so exemption cannot be decided" + else + set_cq not-configured "${detail}" + fi + ;; + *) set_cq unreadable "the default-setup response is malformed" ;; + esac +} + survey_repo() { local nwo="$1" branch="$2" head pulls pr pr_head run source ann count url if ! head="$(api "repos/${nwo}/commits/${branch}" --jq '.sha')" \ @@ -313,13 +382,17 @@ fi while IFS=$'\t' read -r nwo archived fork private branch; do [[ -z "${nwo}" ]] && continue PROT_JSON="" + CQ_JSON="" if [[ "${archived}" == "true" ]]; then emit "${nwo}" archived "archived; not surveyed" elif [[ -z "${branch}" ]]; then set_prot unreadable "no default branch reported" + # Default setup is per repo, not per branch, so it is still read. + probe_codeql "${nwo}" "${fork}" "${private}" emit "${nwo}" unreadable "no default branch reported" else probe_protection "${nwo}" "${branch}" "${fork}" "${private}" + probe_codeql "${nwo}" "${fork}" "${private}" survey_repo "${nwo}" "${branch}" fi done <<<"${repo_list}" diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_cleanroom_code-scanning_default-setup.403.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_cleanroom_code-scanning_default-setup.403.json new file mode 100644 index 0000000..69a37d6 --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_cleanroom_code-scanning_default-setup.403.json @@ -0,0 +1,5 @@ +{ + "message": "Code Security must be enabled for this repository to use code scanning.", + "documentation_url": "https://docs.github.com/rest/code-scanning/code-scanning#get-a-code-scanning-default-setup-configuration", + "status": "403" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_nos-private_code-scanning_default-setup.403.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_nos-private_code-scanning_default-setup.403.json new file mode 100644 index 0000000..7272842 --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_nos-private_code-scanning_default-setup.403.json @@ -0,0 +1,5 @@ +{ + "message": "Resource not accessible by integration", + "documentation_url": "https://docs.github.com/rest/code-scanning/code-scanning#get-a-code-scanning-default-setup-configuration", + "status": "403" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_nostd_code-scanning_default-setup.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_nostd_code-scanning_default-setup.json new file mode 100644 index 0000000..775a026 --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_nostd_code-scanning_default-setup.json @@ -0,0 +1,10 @@ +{ + "state": "not-configured", + "languages": [], + "query_suite": "default", + "threat_model": "remote", + "updated_at": "2026-10-19T23:22:09Z", + "schedule": "weekly", + "runner_type": "standard", + "runner_label": "" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_offcheck_code-scanning_default-setup.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_offcheck_code-scanning_default-setup.json new file mode 100644 index 0000000..6a52df5 --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_offcheck_code-scanning_default-setup.json @@ -0,0 +1,10 @@ +{ + "state": "enabled", + "languages": ["actions"], + "query_suite": "default", + "threat_model": "remote", + "updated_at": "2026-10-19T23:22:09Z", + "schedule": "weekly", + "runner_type": "standard", + "runner_label": "" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_tnjcleaning_code-scanning_default-setup.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_tnjcleaning_code-scanning_default-setup.json new file mode 100644 index 0000000..581c0db --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_nightowlstudiollc_tnjcleaning_code-scanning_default-setup.json @@ -0,0 +1,10 @@ +{ + "state": "not-configured", + "languages": ["actions"], + "query_suite": "default", + "threat_model": "remote", + "updated_at": "2026-10-19T23:22:09Z", + "schedule": "weekly", + "runner_type": "standard", + "runner_label": "" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_o_clean_code-scanning_default-setup.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_o_clean_code-scanning_default-setup.json new file mode 100644 index 0000000..581c0db --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_o_clean_code-scanning_default-setup.json @@ -0,0 +1,10 @@ +{ + "state": "not-configured", + "languages": ["actions"], + "query_suite": "default", + "threat_model": "remote", + "updated_at": "2026-10-19T23:22:09Z", + "schedule": "weekly", + "runner_type": "standard", + "runner_label": "" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_claude-config-backup_code-scanning_default-setup.403.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_claude-config-backup_code-scanning_default-setup.403.json new file mode 100644 index 0000000..d26e02c --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_claude-config-backup_code-scanning_default-setup.403.json @@ -0,0 +1,5 @@ +{ + "message": "Advanced Security must be enabled for this repository to use code scanning.", + "documentation_url": "https://docs.github.com/rest/code-scanning/code-scanning#get-a-code-scanning-default-setup-configuration", + "status": "403" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_dev-env_code-scanning_default-setup.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_dev-env_code-scanning_default-setup.json new file mode 100644 index 0000000..11fb085 --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_dev-env_code-scanning_default-setup.json @@ -0,0 +1,10 @@ +{ + "state": "configured", + "languages": ["actions"], + "query_suite": "default", + "threat_model": "remote", + "updated_at": "2026-10-19T23:22:09Z", + "schedule": "weekly", + "runner_type": "standard", + "runner_label": "" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_noflags_code-scanning_default-setup.403.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_noflags_code-scanning_default-setup.403.json new file mode 100644 index 0000000..69a37d6 --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_noflags_code-scanning_default-setup.403.json @@ -0,0 +1,5 @@ +{ + "message": "Code Security must be enabled for this repository to use code scanning.", + "documentation_url": "https://docs.github.com/rest/code-scanning/code-scanning#get-a-code-scanning-default-setup-configuration", + "status": "403" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_withheld_code-scanning_default-setup.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_withheld_code-scanning_default-setup.json new file mode 100644 index 0000000..87f4975 --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_smartwatermelon_withheld_code-scanning_default-setup.json @@ -0,0 +1 @@ +{"state":"configured","languages":["acti diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_Instapaper-MCP_code-scanning_default-setup.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_Instapaper-MCP_code-scanning_default-setup.json new file mode 100644 index 0000000..581c0db --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_Instapaper-MCP_code-scanning_default-setup.json @@ -0,0 +1,10 @@ +{ + "state": "not-configured", + "languages": ["actions"], + "query_suite": "default", + "threat_model": "remote", + "updated_at": "2026-10-19T23:22:09Z", + "schedule": "weekly", + "runner_type": "standard", + "runner_label": "" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_tm-private_code-scanning_default-setup.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_tm-private_code-scanning_default-setup.json new file mode 100644 index 0000000..033b06b --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_tm-private_code-scanning_default-setup.json @@ -0,0 +1,10 @@ +{ + "state": "configured", + "languages": [], + "query_suite": "default", + "threat_model": "remote", + "updated_at": "2026-10-19T23:22:09Z", + "schedule": "weekly", + "runner_type": "standard", + "runner_label": "" +} diff --git a/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_tm-public_code-scanning_default-setup.403.json b/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_tm-public_code-scanning_default-setup.403.json new file mode 100644 index 0000000..69a37d6 --- /dev/null +++ b/scripts/dependabot-digest/tests/fixtures/standards/repos_twistedmelonman_tm-public_code-scanning_default-setup.403.json @@ -0,0 +1,5 @@ +{ + "message": "Code Security must be enabled for this repository to use code scanning.", + "documentation_url": "https://docs.github.com/rest/code-scanning/code-scanning#get-a-code-scanning-default-setup-configuration", + "status": "403" +} diff --git a/scripts/dependabot-digest/tests/test-render.sh b/scripts/dependabot-digest/tests/test-render.sh index bb4745a..28d86e8 100755 --- a/scripts/dependabot-digest/tests/test-render.sh +++ b/scripts/dependabot-digest/tests/test-render.sh @@ -137,12 +137,21 @@ fi _hasnt '## Branch protection' "no survey, no protection section" printf '%s\n' '{not json' >"${WORK}/bad.ndjson" : | bash "${DIR}/render.sh" --standards "${WORK}/bad.ndjson" >"${WORK}/bad.md" 2>/dev/null -prot_section="$(sed -n '/^## Branch protection/,/^---/p' "${WORK}/bad.md")" +# Each range stops at the next section, so one section's text cannot pass for another's. +prot_section="$(sed -n '/^## Branch protection/,/^## CodeQL/p' "${WORK}/bad.md")" if [[ "${prot_section}" == *"**Not checked.**"* && "${prot_section}" != *"No gaps"* ]]; then _pass "a malformed survey renders protection as not checked" else _fail "a malformed survey did not render protection as not checked: ${prot_section}" fi +# dev-env#166: the same for the CodeQL section. +_hasnt '## CodeQL' "no survey, no CodeQL section" +cq_section="$(sed -n '/^## CodeQL/,/^---/p' "${WORK}/bad.md")" +if [[ "${cq_section}" == *"**Not checked.**"* && "${cq_section}" != *"No gaps"* ]]; then + _pass "a malformed survey renders CodeQL as not checked" +else + _fail "a malformed survey did not render CodeQL as not checked: ${cq_section}" +fi if [[ "${fail}" -eq 0 ]]; then echo "test-render: all assertions passed" diff --git a/scripts/dependabot-digest/tests/test-run-digest.sh b/scripts/dependabot-digest/tests/test-run-digest.sh index a9a64fb..34bf7db 100755 --- a/scripts/dependabot-digest/tests/test-run-digest.sh +++ b/scripts/dependabot-digest/tests/test-run-digest.sh @@ -173,6 +173,13 @@ if grep -q '| all of one | OWNER NOT LISTED:' <<<"${populated}" \ else _fail "an unlistable owner did not render as not checked" fi +# dev-env#166: the CodeQL section appears too, and names the unlisted owner. +if grep -q '^## CodeQL' <<<"${populated}" \ + && grep -q 'one. CodeQL setup of their repositories is unknown' <<<"${populated}"; then + _pass "the digest carries a CodeQL section that names an unlisted owner" +else + _fail "the digest has no CodeQL section, or it hides the unlisted owner" +fi if [[ "${fail}" -eq 0 ]]; then echo "test-run-digest: all assertions passed" diff --git a/scripts/dependabot-digest/tests/test-standards.sh b/scripts/dependabot-digest/tests/test-standards.sh index 6aaeb04..aa3a8e4 100755 --- a/scripts/dependabot-digest/tests/test-standards.sh +++ b/scripts/dependabot-digest/tests/test-standards.sh @@ -35,6 +35,12 @@ done path="${path%%\?*}" file="${STUB_FIX}/${path//\//_}.json" [[ "${path}" == "installation/repositories" ]] && file="${file}.${STUB_OWNER}" +# .403.json: a refusal with a body, written to both streams as real gh does. +if [[ -f "${file%.json}.403.json" ]]; then + cat "${file%.json}.403.json" + echo "gh: $(jq -r '.message' "${file%.json}.403.json") (HTTP 403)" >&2 + exit 1 +fi if [[ ! -f "${file}" ]]; then echo "gh: Resource not accessible by integration (HTTP 403)" >&2 exit 1 @@ -281,6 +287,94 @@ _has '| [o/legacy](https://github.com/o/legacy) | UNREADABLE: no protection resu "a record without a protection field is unreadable, not clean" "${WORK}/legacy.md" _hasnt 'No gaps' "a record without a protection field never reads as no gaps" "${WORK}/legacy.md" +# dev-env#166. Real 2026-10-21 captures: dev-env, cleanroom, +# claude-config-backup. The rest change one field of a real body. +_cq() { + local got + got="$(jq -r --arg r "$1" 'select(.repo == $r) | (.codeql.state // "none") + " " + (.codeql.exemptClass // "-")' "${prot}")" + if [[ "${got}" == "$2" ]]; then _pass "$3"; else _fail "$3 (codeql: '${got}', want '$2')"; fi +} +_cq smartwatermelon/dev-env "configured -" "a real configured repo is configured" +_cq twistedmelonman/tm-private "configured -" "configured with no languages yet is still configured" +_cq nightowlstudiollc/tnjcleaning "not-configured -" "a public non-fork repo with default setup off is flagged" +_cq nightowlstudiollc/cleanroom "no-code-security -" \ + "a real private 'Code Security must be enabled' 403 is no-code-security, not exempt" +_cq smartwatermelon/claude-config-backup "no-code-security -" \ + "a real private 'Advanced Security must be enabled' 403 is no-code-security, not exempt" +_cq nightowlstudiollc/nos-private "unreadable -" \ + "a private 'Resource not accessible by integration' 403 is unreadable, never no-code-security" +_cq smartwatermelon/refused "unreadable -" "a refused read with no body is unreadable" +_cq twistedmelonman/tm-public "unreadable -" \ + "a paid-product 403 on a public repo is unreadable, not no-code-security" +_cq smartwatermelon/noflags "unreadable -" \ + "a paid-product 403 with no private flag is unreadable, not no-code-security" +_cq nightowlstudiollc/nostd "exempt no-supported-language" "not configured with no languages is exempt" +_cq twistedmelonman/Instapaper-MCP "exempt fork" "a fork with default setup off is exempt" +_cq nightowlstudiollc/offcheck "unreadable -" "an unknown state is unreadable" +_cq smartwatermelon/withheld "unreadable -" "malformed JSON is unreadable" +_cq twistedmelonman/tm-old "none -" "an archived repo has no CodeQL result" +if jq -e 'select(.repo == "nightowlstudiollc/nos-private") | .codeql.detail | test("Resource not accessible by integration")' "${prot}" >/dev/null 2>&1; then + _pass "the reason a default-setup read failed is kept" +else + _fail "the refused default-setup read lost its reason" +fi +o_clean_cq="$(jq -r 'select(.repo == "o/clean") | .codeql.state + ": " + .codeql.detail' "${out}")" +if [[ "${o_clean_cq}" == "unreadable: not configured, but the repository listing gave no fork flag"* ]]; then + _pass "not configured with no fork flag is unreadable, not flagged or exempt" +else + _fail "not configured with no fork flag was decided anyway (got '${o_clean_cq}')" +fi + +_has '## CodeQL' "the CodeQL section is rendered" "${pbody}" +_has '**14 repositories surveyed**: 2 configured, 2 exempt, **1 not configured, 2 without Code Security, 7 unreadable.**' \ + "the CodeQL summary counts each state" "${pbody}" +_has '- fork: 1 — a fork is not ours to scan (decided 2026-10-21, dev-env#166)' \ + "CodeQL exemptions are counted by class with the reason" "${pbody}" +_hasnt '- no-code-security:' "no-code-security is not an exempt class" "${pbody}" +_hasnt '| [nightowlstudiollc/cleanroom](' "a repo without Code Security is not a table row" "${pbody}" +nocs_list="$(sed -n '/^### Scanning off: paid Code Security not bought/,/^\(##\|---\|\*\*Not listed\)/p' "${pbody}")" +if [[ "${nocs_list}" == *"open cost decision (dev-env#166)"* \ + && "${nocs_list}" == *"- [nightowlstudiollc/cleanroom](https://github.com/nightowlstudiollc/cleanroom)"* \ + && "${nocs_list}" == *"- [smartwatermelon/claude-config-backup](https://github.com/smartwatermelon/claude-config-backup)"* \ + && "${nocs_list}" != *"nos-private"* && "${nocs_list}" != *"tm-public"* ]]; then + _pass "repos without Code Security are listed by name under their own heading, and only they are" +else + _fail "the no-Code-Security list is wrong: ${nocs_list}" +fi +_has '| [nightowlstudiollc/tnjcleaning](https://github.com/nightowlstudiollc/tnjcleaning) | NOT CONFIGURED: default setup is off; languages: actions |' \ + "a not-configured repo is listed with its languages" "${pbody}" +_has '| [nightowlstudiollc/nos-private](https://github.com/nightowlstudiollc/nos-private) | UNREADABLE: cannot read default setup: ' \ + "a permission 403 is listed as unreadable with its reason" "${pbody}" +cq_body="$(sed -n '/^## CodeQL/,/^---/p' "${pbody}")" +for r in smartwatermelon/dev-env twistedmelonman/Instapaper-MCP nightowlstudiollc/nostd; do + if [[ "${cq_body}" == *"[${r}]"* ]]; then _fail "configured or exempt repo ${r} is listed"; else _pass "configured or exempt repo ${r} is counted, not listed"; fi +done + +# Every o and p read failed: a token without Administration: Read must not +# render as a clean fleet. +cq_all="$(sed -n '/^## CodeQL/,/^---/p' "${body}")" +if [[ "${cq_all}" == *"**CodeQL setup could not be verified for any of 8 repositories.**"* \ + && "${cq_all}" == *'Administration: Read'* && "${cq_all}" != *"No gaps"* \ + && "${cq_all}" != *"repositories surveyed"* ]]; then + _pass "an all-unreadable CodeQL survey says it verified nothing and names the likely permission" +else + _fail "an all-unreadable CodeQL survey read as a result: ${cq_all}" +fi + +jq -c 'select(.repo == "smartwatermelon/dev-env" or .repo == "twistedmelonman/Instapaper-MCP")' "${prot}" >"${WORK}/cqclean.ndjson" +: | bash "${DIR}/render.sh" --standards "${WORK}/cqclean.ndjson" >"${WORK}/cqclean.md" 2>/dev/null +_has 'No gaps: every surveyed repository has default setup on' "a fully configured CodeQL survey says so" "${WORK}/cqclean.md" +_hasnt '| Repo | CodeQL default setup |' "a fully configured CodeQL survey has no table" "${WORK}/cqclean.md" +# A repo without Code Security alone is a gap, not "No gaps", and not a table row. +jq -c 'select(.repo == "smartwatermelon/dev-env" or .repo == "nightowlstudiollc/cleanroom")' "${prot}" >"${WORK}/cqnocs.ndjson" +: | bash "${DIR}/render.sh" --standards "${WORK}/cqnocs.ndjson" >"${WORK}/cqnocs.md" 2>/dev/null +sed -n '/^## CodeQL/,/^---/p' "${WORK}/cqnocs.md" >"${WORK}/cqnocs-section.md" +_hasnt 'No gaps' "a repo without Code Security is never reported as no gaps" "${WORK}/cqnocs-section.md" +_has '- [nightowlstudiollc/cleanroom]' "a lone repo without Code Security is listed by name" "${WORK}/cqnocs.md" +_hasnt '| Repo | CodeQL default setup |' "a repo without Code Security is not a table row" "${WORK}/cqnocs.md" +_has '| [o/legacy](https://github.com/o/legacy) | UNREADABLE: no CodeQL result recorded |' \ + "a record without a codeql field is unreadable, not clean" "${WORK}/legacy.md" + if [[ "${fail}" -eq 0 ]]; then echo "test-standards: all assertions passed" else