diff --git a/.github/wiki/_Sidebar.md b/.github/wiki/_Sidebar.md index df92e1ce6..4e1413044 100644 --- a/.github/wiki/_Sidebar.md +++ b/.github/wiki/_Sidebar.md @@ -177,6 +177,10 @@ - [[CI scheduling and cost|guides/ci/Scheduling-and-Cost]] - [[Hosted providers|guides/ci/Hosted-Providers]] - [[Cache and artifacts|guides/ci/Cache-and-Artifacts]] +- [[Walkthrough: a home lab|guides/ci/Walkthrough-Home-Lab]] +- [[Walkthrough: several projects on one fleet|guides/ci/Walkthrough-Multiple-Projects]] +- [[Walkthrough: a team|guides/ci/Walkthrough-Team]] +- [[Walkthrough: agents|guides/ci/Walkthrough-Agents]] - [[CI reference|guides/ci/Reference]] - [[CI agent reference|guides/ci/Agent-Reference]] - [[Auto placement and pools|guides/ci/Auto-Placement-and-Pools]] diff --git a/.github/wiki/guides/ci/Walkthrough-Agents.md b/.github/wiki/guides/ci/Walkthrough-Agents.md new file mode 100644 index 000000000..256dfeb69 --- /dev/null +++ b/.github/wiki/guides/ci/Walkthrough-Agents.md @@ -0,0 +1,110 @@ +# Walkthrough: agents + +This page walks a coding agent through CI. The agent starts a run, waits for it, and reads the evidence. It can do this through the CLI, through MCP tools or through HTTP routes. All three return the same JSON. + +Commands that need a real run are shown as text, each with the test that runs it. The full list of MCP tools and HTTP routes, with each command's side effect and confirmation rule, is the generated machine-surface table at the top of the [CI plugin page](../../plugins/CI.md). The command and flag tables are in the generated [Reference](./Reference.md#commands). + +## The CLI loop + +Start the run in the background with `--detach --json`. The reply is one JSON document with the run id and the exact commands to wait for it, follow it, read its evidence, cancel it and read its logs. + + +```text +nself ci --detach --json +``` + +Wait for the result, then read the evidence. `runs wait` returns when the run ends; `--timeout` bounds the wait. + + +```text +nself ci runs wait --json +``` + + +```text +nself ci evidence --json +``` + +To watch progress instead of waiting, follow the run. Each line is one JSON event, and a reader that reconnects can resume from the last sequence number it saw. + + +```text +nself ci runs follow --json +``` + +The document shapes are in [Evidence and Machine Output](./Evidence-and-Machine-Output.md). + +## The MCP loop + +Start the MCP server with `nself mcp` inside the project directory. The agent needs only three tools for the loop: + +1. `nself_ci` starts a run. Pass `{"args": ["--detach"]}`. +2. `nself_ci_runs_wait` waits for it. Pass `{"args": [""]}`. +3. `nself_ci_evidence` returns the evidence. Pass `{"args": [""]}`. + +Every other tool is listed in the machine-surface table. Commands that only make sense in a terminal, that stream, or that ask questions have no tool. + +## Confirmations + +A bare run executes code, so it needs a confirmation. The first `nself_ci` call does not run anything. It returns a request with a `confirm` value. The agent shows the request to its user and, once approved, calls `nself_ci` again with the same arguments plus that `confirm` value. Only the second call starts the run. Destructive commands, such as revoking a node, work the same way. The confirmation column of the machine-surface table says which commands need it. + +```text +first call: {"args": ["--detach"]} +reply: {"kind": "request", "confirm": "<64 hex characters>", ...} +second call: {"args": ["--detach"], "confirm": ""} +``` + +## The HTTP routes + +`nself admin serve` exposes the same commands as HTTP routes under `/v1/commands/`, for example `/v1/commands/ci/runs/wait`. Send a POST with a bearer token and a JSON body such as `{"args": [""]}`. The reply carries the same data as the CLI's `--json` output. Confirmations use the same `confirm` round trip. + +`runs follow` has no MCP tool. Over HTTP, `/v1/commands/ci/runs/follow` answers with an NDJSON stream: one JSON event per line, flushed as the run moves. + +## One server per project + +An MCP or HTTP server works on the project directory it was started in. Start one server per project, each from its own checkout. + +## No commit status from agents by default + +A run started through MCP or HTTP is a machine-invoked run: the server marks it with `NSELF_INVOKED_BY`. A CLI agent should export that variable itself. A machine-invoked run posts no commit status, and prints one line saying so. Add `--post-status` when the agent's result should count as the commit's status. + + +```text +nself ci --detach --json --post-status +``` + +## Capping the trust of agent runs + +The policy key `trust.machine_invoked_cap` caps the source trust class of every machine-invoked run. The default is `owner`, which is no cap. For agents that run unattended, set it to `untrusted`. A machine-invoked caller can lower the cap but never raise it again. + + +```text +nself ci policy set --scope user trust.machine_invoked_cap untrusted +``` + +For a CLI agent the cap is advisory, because the agent controls its own environment. See [machine-invoked runs](./Trust-Security-and-Privacy.md#machine-invoked-runs). + +## Code an agent wrote + +Run code that an agent wrote, and that nobody has read yet, as untrusted. `--trust untrusted` needs the sandboxed container level; it never falls back to a weaker level, never gets service containers and never gets internet access. If the host cannot provide the sandbox, nothing runs. + + +```text +nself ci --trust untrusted --detach --json -- go test ./... +``` + +The isolation levels are in [Trust, Security and Privacy](./Trust-Security-and-Privacy.md#the-sandboxed-container-level-gvisor). + +## Proven by + +| Step | Test | +|---|---| +| Detached run and the handle's commands | `TestDetach` | +| Detached run outlives its parent | `TestDetachSurvivesParent` | +| Wait for a run | `TestRunsWait` | +| Evidence as one JSON document | `TestBareJSONIsOneEvidenceEnvelope` | +| Follow replays from a sequence number | `TestFollowReplayFromSeq` | +| MCP tools, HTTP routes, same output on CLI, MCP and HTTP, confirmation round trip | `scripts/ci/ci-machine-surface.sh` | +| No status from machine-invoked runs unless `--post-status` | `TestMachineDefaults` | +| Trust cap key, lower but never raise | `TestPolicyMachineInvokedCap` | +| `--trust untrusted` needs the sandbox | `TestResolveLevelAndNetwork` | diff --git a/.github/wiki/guides/ci/Walkthrough-Home-Lab.md b/.github/wiki/guides/ci/Walkthrough-Home-Lab.md new file mode 100644 index 000000000..6f649fbcd --- /dev/null +++ b/.github/wiki/guides/ci/Walkthrough-Home-Lab.md @@ -0,0 +1,151 @@ +# Walkthrough: a home lab + +This page walks one real setup from start to finish: a Mac Studio on your desk runs the coordinator, two Mac Minis on the same LAN run jobs, and a small VPS you rent runs Linux jobs. Everything here is personal mode: you are the only person with access. + +The page shows the order of steps and links the guide that explains each one. Commands that need real machines are shown as text, each with the test that runs it. The tables of commands, flags, codes and doctor checks live in the generated [Reference](./Reference.md#commands). + +Names used below: `desk` is the Mac Studio, `mini1` and `mini2` are the Minis, `vps1` is the VPS at `203.0.113.20`. Replace them with your own. + +## The desk Mac runs the coordinator + +The coordinator keeps the run store, the node registry and the keys. In personal mode it starts with no extra flag. Its webhook and user API listeners stay on loopback by default; only the agent listener needs to be reachable from the Minis. The listener ports are described in [Nodes and Runners](./Nodes-and-Runners.md#team). + + +```text +nself ci serve +``` + +A coordinator started without `--team` stays personal. Do not add `--team` here: the switch is permanent for that data directory, and a home lab used by one person does not need it. + +## Adding the Minis + +There are two ways to add a Mac. Use `nodes add --system` when you can reach the Mini over SSH. Use pairing when you cannot, or when you do not want the coordinator to hold an SSH key for it. + +**Add over SSH.** `--system` installs the agent as a system service: a hidden agent user, one job user per slot, and a LaunchDaemon that runs the agent in pull mode. Get the host-key fingerprint from the Mini itself, not from the network. + + +```text +nself ci nodes add ci@mini1 --system --slots 2 --host-key --agent-binary --project --name mini1 --json +``` + +The bootstrap checks every account and folder it creates. Until jobs actually run through the job shim, the node is recorded as not UID-separated, and installing it grants no secret or signing rights. See [provisioning versus bootstrap](./Nodes-and-Runners.md#vps). + +**Pair instead.** Run `nodes pair` on the desk Mac. It opens a short window and prints a `nself-ci-agent join` command, the coordinator fingerprint and a one-use code. Run the printed join command on the Mini, type the code there, and check that both fingerprints match before you approve. + + +```text +nself ci nodes pair --label mini2 --agent-url https://desk.example:3864 --cert-ttl 30m +``` + +A paired Mini runs the same agent in pull mode: it only makes outbound connections to the coordinator. Never put the pairing code in a command line or a shared log. + +## Labels: which jobs may run where + +Give each node the labels your jobs ask for. The Minis have Xcode, so they get the `xcode` label. + + +```text +nself ci nodes label mini1 xcode +nself ci nodes label mini2 xcode +``` + +A job that needs Xcode says so in `.nself/ci.yaml`. A runner is eligible only when it has every label the job asks for: + +```yaml +jobs: + ios: + kind: test + run: [xcodebuild, test] + requires: {labels: [xcode]} +``` + +Check where each job would go before you push. `plan` shows the placement; `explain` lists every runner that was not chosen and why. + + +```text +nself ci plan --json +``` + + +```text +nself ci explain +``` + +The reasons (`labels.mismatch`, `tools.missing` and others) are listed in [Scheduling and Cost](./Scheduling-and-Cost.md#job-requirements-and-labels) and in the [reason table](./Reference.md#reason-codes). + +## Reservations on the desk machine + +If the desk Mac also takes jobs, keep part of it for yourself. Reservations are scheduler limits: CI work never uses the reserved CPU or memory. `--interactive protect` holds new work back while you are using the machine. `--on-battery` matters only for laptops; a desk machine can leave it at `allow`. + + +```text +nself ci nodes set desk --reserve-cpu 2 --on-battery allow +nself ci nodes set desk --reserve-mem 2048 +``` + +A reservation that would leave nothing for CI is refused. More controls are in [Nodes and Runners](./Nodes-and-Runners.md#lan). + +## The personal VPS + +A VPS is added the same way as a Mini, with its host key pinned. The SSH carrier forces strict host-key checking against the coordinator's own pin file, so a changed key stops the connection before any job runs. Connect as root or a user with passwordless sudo so `--system` can create the agent and job users. + + +```text +nself ci nodes add root@203.0.113.20 --system --slots 3 --host-key --agent-binary --json +``` + +Never add a machine that serves your deployed apps. Deploy hosts are not eligible for CI; see [deploy hosts are ineligible](./Nodes-and-Runners.md#vps). + +## Secrets on a node + +A node gets no project secrets until you grant them. Grant the `project` class to the VPS if its jobs need project secrets. The `release` class needs a node dedicated to releases. + + +```text +nself ci nodes secrets grant vps1 --class project +``` + +Every grant and revoke is written to the audit log. The secret rules are in [Trust, Security and Privacy](./Trust-Security-and-Privacy.md#secrets). + +## When a Mini sleeps + +A sleeping or frozen Mini stops renewing its lease. It fences itself before the coordinator's lease expires, so it cannot finish the old job late. The coordinator marks the attempt lost and runs it again on another eligible node. If the Mini wakes and reports the old result, that result is refused. A job that failed on its own merits is not retried. + +Mark machines that sleep on purpose as intermittent so the scheduler prefers always-on nodes: + + +```text +nself ci nodes set mini2 --reserve-mem 1024 --persistence intermittent --json +``` + +Lost leases and what to check are covered in [Troubleshooting](./Nodes-and-Runners.md#troubleshooting). + +## Backing up the coordinator + +Back up the desk Mac's data directory to a file. The archive does not contain the key-encryption key (KEK). Keep the KEK somewhere else; without it the archive cannot be restored, and a wrong KEK is refused before anything is written. + + +```text +nself ci serve backup --data-dir --out --json +nself ci serve restore --data-dir --in --kek --json +``` + +Stop the coordinator before a restore. What restore keeps and resets is in [backup/restore and the KEK](./Nodes-and-Runners.md#team). + +## Proven by + +| Step | Test | +|---|---| +| Personal coordinator start, no `--team` | `TestServeModeFlagAndStart` | +| Add a Mac with `--system` (LaunchDaemon, users, layout) | `TestMacOSNode` | +| Pair a node and run the printed join | `TestNodesPairEndToEnd` | +| Labels and the placement they cause | `TestSeveralProjects` | +| Placement plan in JSON | `TestPlanPlacementJSON` | +| Every alternative explained | `TestScenario6ExplainListsEveryAlternative` | +| Reservations and battery policy | `TestNodesSetFlagsMayFollowOrPrecedeTheNodeAndKeepWhatIsNotGiven` | +| Add a Linux VPS with `--system` | `TestSystemInstallBootstrapsAndKeepsTheNodeUnseparated` | +| Grant project secrets to a node | `TestGrantProjectCallsCliAndAudits` | +| Release secrets need a dedicated node | `TestGrantReleaseNeedsDedicatedNode` | +| A frozen node fences and the job runs elsewhere | `TestFaultScenario9Freeze` | +| Intermittent node setting | `TestNodesSetJSONEnvelopeCarriesTheStoredSettings` | +| Backup and restore with the KEK | `TestBackupRestoreCommandsRoundTrip` | diff --git a/.github/wiki/guides/ci/Walkthrough-Multiple-Projects.md b/.github/wiki/guides/ci/Walkthrough-Multiple-Projects.md new file mode 100644 index 000000000..bb8e4aa9c --- /dev/null +++ b/.github/wiki/guides/ci/Walkthrough-Multiple-Projects.md @@ -0,0 +1,95 @@ +# Walkthrough: several projects on one fleet + +This page walks one fleet that serves two projects. Project A and project B share a Mac and a VPS. A second VPS belongs to project A only. Every project keeps its own queue share, cache and statistics. + +Names used below: `mac1` has Xcode, `vps-shared` serves both projects, `vps-a` serves project A only. `~/src/project-a` and `~/src/project-b` are the two checkouts. Commands that need real machines are shown as text, each with the test that runs it. The command and flag tables are in the generated [Reference](./Reference.md#commands). + +## One pool for several projects + +A node runs jobs only for the projects it is authorized for. Authorize by checkout directory or by repository id; a directory is turned into the same repository id that evidence uses. + + +```text +nself ci nodes projects add mac1 ~/src/project-a ~/src/project-b +nself ci nodes projects add vps-shared ~/src/project-a ~/src/project-b +nself ci nodes projects list vps-shared +``` + +To take a project off a node, remove it. `--all` removes every project from that node. + + +```text +nself ci nodes projects remove vps-shared +``` + +## A VPS for one project + +Authorize the dedicated VPS for project A only. Project B's jobs never land there, even when it is idle. + + +```text +nself ci nodes projects add vps-a ~/src/project-a +nself ci nodes projects list vps-a +``` + +A personal node is never chosen for a private repository it is not authorized for. Adding the project makes it eligible. + +## Pinning a project with labels + +Labels describe a node; a project can require extra labels for all its jobs. Give `vps-a` a label, then require that label at project scope for project A. Project A's jobs then need it on top of their own `requires.labels`. + + +```text +nself ci nodes label mac1 xcode +nself ci nodes label vps-a project=a +nself ci policy set --scope project sched.overrides.labels '["project=a"]' +``` + +Run the `policy set` line inside project A's checkout. After it, all of A's jobs go to `vps-a`. A job of A that also needs `xcode` waits: `mac1` has Xcode but not `project=a`, and `vps-a` has the reverse. `explain` names the override as the reason. + + +```text +nself ci explain +``` + +How overrides combine with job requirements is in [Scheduling and Cost](./Scheduling-and-Cost.md#overrides). + +## Fairness and the queue + +When the fleet is busy, the scheduler shares capacity between projects and never admits more than the fleet can hold. Interactive runs start before queued background work. `runs queue` shows each waiting run with its position, its band and the expected wait. + + +```text +nself ci runs queue +``` + +The fairness rules and presets are in [Scheduling and Cost](./Scheduling-and-Cost.md#how-a-runner-is-chosen). + +## Cache and artifacts stay separate + +Each cache entry lives in a namespace made of the project, the trust class of the source and whether the ref is protected. The same key in another project's namespace neither blocks nor reveals the entry. Artifacts are stored per run, and runs belong to one project. Nothing needs to be set up for this. See [namespaces](./Cache-and-Artifacts.md#namespaces-per-trust-and-per-pr). + +## Statistics per project + +`runs stats` summarises runs, queue time and failures. `--project` limits it to one project; `--since` sets the window. A caller of a team coordinator sees only the projects it has a role on. + + +```text +nself ci runs stats --since 168h --project --json +``` + +The fields are described in [CI health](./Scheduling-and-Cost.md#ci-health). + +## Proven by + +| Step | Test | +|---|---| +| Authorize, list and remove projects on a node; per-project VPS; label override | `TestSeveralProjects` | +| A personal node needs authorization for a private repo | `TestPersonalNodePrivateRepoOptIn` | +| Every alternative explained | `TestScenario6ExplainListsEveryAlternative` | +| Fair share under saturation | `TestScenario10SaturationStaysWithinCapacityAndFair` | +| Interactive before background | `TestScenario10InteractiveStartsBeforeQueuedBackground` | +| Queue position, band and wait | `TestRunsQueueListsPositionBandAndWait` | +| Cache namespaces do not leak | `TestSameKeyStringInAnotherNamespaceNeitherBlocksNorRevealsTheTrustedEntry` | +| Stats window and project filter | `TestStatsReadWindowAndProject` | +| Stats scoped to the caller's projects | `TestStatsScopedCallerSeesOnlyAllowedProjects` | diff --git a/.github/wiki/guides/ci/Walkthrough-Team.md b/.github/wiki/guides/ci/Walkthrough-Team.md new file mode 100644 index 000000000..8f8f86937 --- /dev/null +++ b/.github/wiki/guides/ci/Walkthrough-Team.md @@ -0,0 +1,127 @@ +# Walkthrough: a team + +This page walks a small team that shares CI. One box runs the team coordinator. Two other boxes run jobs. Members submit runs and read results from their own machines. Each member may also keep personal nodes at home. + +Names used below: `ci.example` is the coordinator box, `runner1` and `runner2` are the job boxes, `example-org/example-repo` is a team repository. Commands that need real machines are shown as text, each with the test that runs it. The command and flag tables are in the generated [Reference](./Reference.md#commands). + +## A team coordinator on its own box + +Start the coordinator with `--team` as soon as anyone besides you will hold a role, a token or a certificate. The switch is permanent and audited; starting the same data directory without the flag later does not make it personal again. + + +```text +nself ci serve --team +``` + +Pick the box with care: + +- Never a production box. A machine that serves deployed apps is refused as a CI node; keep the coordinator and its keys off it as well. +- Keep jobs off the coordinator. In team mode any job on the coordinator machine needs VM isolation or stronger. Two boxes are simpler: one keeps state and keys, the others run jobs. + +Add `runner1` and `runner2` from the coordinator box as described in [Nodes and Runners](./Nodes-and-Runners.md#lan). The coordinator-host rule is in [Coordinator host](./Trust-Security-and-Privacy.md#coordinator-host). + +## Roles and tokens + +Roles are `member`, `release` and `admin`, each for one or more projects. Grant a role to a person, or issue a scoped token for a machine. A token is shown once. + + +```text +nself ci serve admin roles grant bob --role member --project example-org/example-repo --json +nself ci serve admin roles list +``` + + +```text +nself ci serve admin token issue build-bot --role release --project example-org/example-repo --ttl 48h --json +``` + +Revoking a role also ends that person's tokens. Every change is in the audit log. See [roles, tokens and the audit log](./Trust-Security-and-Privacy.md#roles-tokens-and-the-audit-log). + +## Members work through `--coordinator` + +A member's commands go to the team coordinator when they name it with `--coordinator`. Members can list and show nodes; draining, enabling, revoking or removing a node needs `admin`. + + +```text +nself ci nodes list --coordinator https://ci.example:3865 --json +``` + + +```text +nself ci --detach --json --coordinator https://ci.example:3865 +``` + +Without the flag, commands stay on the member's own machine, even when a coordinator is configured. Adding and pairing nodes happen on the coordinator box only. + +## Personal nodes stay personal + +A member's home nodes belong to that member's personal coordinator. They are not part of the team pool, and a team repository's commit of record never runs on them. A project can list which pools may run it; a personal pool left off that list is excluded. `nodes pools explain` shows the effective set and why each runner is left out. + + +```text +nself ci nodes pools explain --project example-org/example-repo +``` + +The pool rules are in [Auto Placement and Pools](./Auto-Placement-and-Pools.md#pools). + +## The organization's GitHub credentials + +The coordinator holds the organization's GitHub credentials, not each member. Write the dispatch workflow into each repository with `providers github init`. + + +```text +nself ci providers github init --stdout --agent-source source +``` + +Scopes and which token does what are in [credentials and scopes](./Hosted-Providers.md#credentials-and-scopes). + +## Open source repositories use GitHub-hosted runners + +For a public repository, a pull request from a fork is untrusted. It gets no secrets, no LAN access and no trusted cache. Its jobs need the sandboxed container level on a runner with no route to your LAN, or a disposable GitHub-hosted runner. Hosted runners are free for public repositories and nSelf never plans paid hosted minutes. A failure on the hosted runner stays a failure. + +Read [Public FOSS PRs](./Trust-Security-and-Privacy.md#public-foss-prs) and [hosted capacity](./Hosted-Providers.md#hosted-capacity) before you open a public repository to your own nodes. + +## The required check + +Make `nself-ci` a required status check on the protected branch. Run it with `--dry-run` first: it prints the change and writes nothing. + + +```text +nself ci policy protect --repo example-org/example-repo --branch main --dry-run +``` + +Without `--dry-run` it saves a snapshot of the old protection before it writes the new one. + +## Own nodes satisfy the check by polling + +Your nodes do not need to accept inbound webhooks. `serve watch` polls GitHub over outbound connections only, claims a commit, runs it on your nodes and posts one final `nself-ci` status. With several watchers, exactly one final result is posted per commit; if the preferred watcher dies, another takes over once its claim is dead. + + +```text +nself ci serve watch --once --json +``` + +If the branch pins the required check to a GitHub App, the watcher posts the advisory `nself-ci/watch` context instead, never `nself-ci`. See [Watch and Services](./Watch-and-Services.md#what-watch-does). + +## Proven by + +| Step | Test | +|---|---| +| Team mode flag, permanent switch | `TestServeModeFlagAndStart` | +| Coordinator mode rules | `TestCoordinatorMode` | +| Grant, list and revoke roles | `TestServeAdminRolesGrantListRevoke` | +| Issue a token once | `TestServeAdminTokenIssueOnce` | +| Revoking a role ends its tokens | `TestServeAdminRevokeEndsTokensToo` | +| Member reads, cannot change nodes | `TestNodesRemoteMemberReadsButCannotRevoke` | +| Add and pair stay on the coordinator box | `TestNodesRemoteAddAndPairAreLocalOnly` | +| Detached run sent to a coordinator | `TestDetach` | +| Personal pool excluded from team work | `TestScopePersonalVsTeamPool` | +| Commit of record stays on the record pool | `TestRecordPoolTransportExclusion` | +| Dispatch workflow rendering | `TestGithubInitStdout` | +| Hosted result decisions | `TestHostedDecisionMatrix` | +| Hosted capacity is never metered | `TestEconomicsGitHubHostedIsNeverMetered` | +| Required check dry run | `TestPolicyProtectDryRunChangesNothing` | +| Snapshot before applying protection | `TestPolicyProtectApplyWritesSnapshotFirst` | +| One final status from several watchers | `TestWatchThreeHostsExactlyOneStatus` | +| Takeover after a dead claim | `TestWatchKillPreferredTakeover` | +| Pinned check gets the advisory context | `TestStatusContextPinned` | diff --git a/scripts/docs/check-walkthroughs.py b/scripts/docs/check-walkthroughs.py new file mode 100755 index 000000000..bad5b9126 --- /dev/null +++ b/scripts/docs/check-walkthroughs.py @@ -0,0 +1,230 @@ +#!/usr/bin/env python3 +"""Check the CI walkthrough pages against the generated reference and the test suite. + +Purpose: every `nself ci ...` line in a code block of a walkthrough page names a command that the generated + Reference.md lists, every `--flag` on a bare `nself ci` line is a flag that table lists, every + ```text block that contains `nself ci` carries a `` line directly above + it, every named proof exists in the repository, and the page has a "Proven by" table that lists + each proof its text blocks name. +Inputs: page paths, or --self-test; --reference overrides the Reference.md path (default: the generated page + beside the walkthroughs). Proofs are looked up with `git grep` in the repository that holds the + script: a Go test name must match `func Name(`, a path must be a tracked file. +Outputs: one line per finding on stdout; exit 0 clean, 1 findings, 2 usage or unreadable input. +Constraints: stdlib only; read-only; no network. +""" + +import argparse +import os +import re +import subprocess +import sys + +REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) +DEFAULT_REFERENCE = os.path.join(REPO, ".github", "wiki", "guides", "ci", "Reference.md") +TESTDATA = os.path.join(os.path.dirname(os.path.abspath(__file__)), "testdata", "walkthroughs") + +FENCE = re.compile(r"^\s*(`{3,})\s*([A-Za-z0-9_+-]*)") +PROVEN_BY = re.compile(r"^\s*\s*$") +COMMAND_ROW = re.compile(r"^\|\s*`(nself ci(?: [a-z][a-z0-9-]*)*)`\s*\|") +FLAG_ROW = re.compile(r"^\|\s*`(--[a-z0-9][a-z0-9-]*)`\s*\|") +WORD = re.compile(r"^[a-z][a-z0-9-]*$") + + +def load_reference(path): + """Return (commands, flags) from the generated command and flag tables of Reference.md.""" + commands, flags = set(), set() + with open(path, encoding="utf-8") as f: + for line in f: + m = COMMAND_ROW.match(line) + if m: + commands.add(m.group(1)) + continue + m = FLAG_ROW.match(line) + if m: + flags.add(m.group(1)) + if not commands: + raise ValueError("%s: no command rows found" % path) + return commands, flags + + +def code_blocks(lines): + """Yield (language, fence line index, body lines as (index, text)) for each fenced block.""" + i = 0 + while i < len(lines): + m = FENCE.match(lines[i]) + if not m: + i += 1 + continue + ticks, lang, start = m.group(1), m.group(2), i + body = [] + i += 1 + while i < len(lines) and lines[i].strip() != ticks and not ( + lines[i].strip().startswith(ticks) and lines[i].strip().strip("`") == "" + ): + body.append((i, lines[i])) + i += 1 + yield lang, start, body + i += 1 + + +def ci_tokens(text): + """Return the tokens after `nself ci` when the line is an nself ci command, else None.""" + s = text.strip() + if s.startswith("$ "): + s = s[2:].lstrip() + parts = s.split() + if len(parts) < 2 or parts[0] != "nself" or parts[1] != "ci": + return None + return parts[2:] + + +def check_command(tokens, commands, flags): + """Return a finding for one `nself ci` line, or None.""" + words = [] + for tok in tokens: + if not WORD.match(tok): + break + words.append(tok) + path = "nself ci" + used = 0 + for n in range(len(words), 0, -1): + cand = "nself ci " + " ".join(words[:n]) + if cand in commands: + path, used = cand, n + break + if used < len(words): + nxt = path + " " + words[used] + if any(c == nxt or c.startswith(nxt + " ") for c in commands): + k = used + 1 + while k < len(words): + head = "nself ci " + " ".join(words[: k + 1]) + if not any(c == head or c.startswith(head + " ") for c in commands): + break + k += 1 + return "`%s` is not a command in Reference.md" % " ".join(["nself ci"] + words[: k + 1]) + if path == "nself ci": + # A bare run takes flags and paths, never a plain word: a leading word is a misspelt verb. + return "`nself ci %s` is not a command in Reference.md" % words[0] + if path == "nself ci": + for tok in tokens: + if tok == "--": + break + if tok.startswith("--"): + name = tok.split("=", 1)[0] + if name not in flags: + return "`%s` is not a flag listed in Reference.md" % name + return None + + +def proof_exists(name, cache): + """Report whether a proof (a Go test name or a repo path) exists in the repository.""" + if name in cache: + return cache[name] + if "/" in name: + args = ["git", "-C", REPO, "ls-files", "--error-unmatch", "--", name] + else: + args = ["git", "-C", REPO, "grep", "-q", "-F", "func %s(" % name, "--", "*.go"] + ok = subprocess.run(args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL).returncode == 0 + cache[name] = ok + return ok + + +def proven_by_table(lines): + """Return the proof names in the first column of the "## Proven by" table, or None without the section.""" + names, inside = [], False + for line in lines: + if line.startswith("## "): + inside = line.strip() == "## Proven by" + continue + if inside and line.startswith("|"): + cells = [c.strip() for c in line.strip().strip("|").split("|")] + for m in re.finditer(r"`([A-Za-z0-9_./-]+)`", cells[-1] if len(cells) > 1 else ""): + names.append(m.group(1)) + return names if any(l.strip() == "## Proven by" for l in lines) else None + + +def check_page(path, commands, flags, cache): + """Return the findings for one page as strings.""" + try: + with open(path, encoding="utf-8") as f: + lines = f.read().split("\n") + except OSError as e: + return ["%s: %s" % (path, e)] + out = [] + used_proofs = [] + for lang, start, body in code_blocks(lines): + has_ci = False + for idx, text in body: + tokens = ci_tokens(text) + if tokens is None: + continue + has_ci = True + msg = check_command(tokens, commands, flags) + if msg: + out.append("%s:%d: %s" % (path, idx + 1, msg)) + if lang == "text" and has_ci: + m = PROVEN_BY.match(lines[start - 1]) if start > 0 else None + if not m: + out.append("%s:%d: text block with `nself ci` has no proven-by line directly above it" % (path, start + 1)) + continue + used_proofs.append((start, m.group(1))) + table = proven_by_table(lines) + if table is None: + out.append("%s: no \"## Proven by\" section" % path) + table = [] + for name in sorted(set(table) | {n for _, n in used_proofs}): + if not proof_exists(name, cache): + out.append("%s: proof %s does not exist in the repository" % (path, name)) + for start, name in used_proofs: + if name not in table: + out.append("%s:%d: proof %s is missing from the Proven by table" % (path, start, name)) + return out + + +def self_test(): + """Run the fixtures: good.md must pass and every bad-*.md must fail. Return the exit code.""" + ref = os.path.join(TESTDATA, "Reference.md") + commands, flags = load_reference(ref) + cache, status = {}, 0 + for name in sorted(os.listdir(TESTDATA)): + if not name.endswith(".md") or name == "Reference.md": + continue + findings = check_page(os.path.join(TESTDATA, name), commands, flags, cache) + want_fail = name.startswith("bad-") + if want_fail and not findings: + print("self-test: %s passed but must fail" % name) + status = 1 + elif not want_fail and findings: + print("self-test: %s must pass:\n %s" % (name, "\n ".join(findings))) + status = 1 + else: + print("self-test: %s ok (%s)" % (name, findings[0] if findings else "clean")) + return status + + +def main(argv): + ap = argparse.ArgumentParser(description="Check CI walkthrough pages.") + ap.add_argument("pages", nargs="*") + ap.add_argument("--reference", default=DEFAULT_REFERENCE) + ap.add_argument("--self-test", action="store_true") + a = ap.parse_args(argv) + if a.self_test: + return self_test() + if not a.pages: + ap.print_usage(sys.stderr) + return 2 + try: + commands, flags = load_reference(a.reference) + except (OSError, ValueError) as e: + print(e, file=sys.stderr) + return 2 + cache, findings = {}, [] + for page in a.pages: + findings += check_page(page, commands, flags, cache) + for f in findings: + print(f) + return 1 if findings else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/docs/testdata/walkthroughs/Reference.md b/scripts/docs/testdata/walkthroughs/Reference.md new file mode 100644 index 000000000..8fe28e550 --- /dev/null +++ b/scripts/docs/testdata/walkthroughs/Reference.md @@ -0,0 +1,17 @@ +# CI reference (fixture) + +## Commands + +| Command | Surface | Side effect | Summary | +|---|---|---|---| +| `nself ci` | v1.5 | remote | Run the gate | +| `nself ci nodes projects add` | v1.5 | write | - | +| `nself ci nodes projects list` | v1.5 | read | - | +| `nself ci runs wait` | v1.5 | read | - | + +### Flags + +| Flag | Commands | +|---|---| +| `--detach` | - | +| `--json` | `runs *` | diff --git a/scripts/docs/testdata/walkthroughs/bad-missing-proven-by.md b/scripts/docs/testdata/walkthroughs/bad-missing-proven-by.md new file mode 100644 index 000000000..fc2d3f3f2 --- /dev/null +++ b/scripts/docs/testdata/walkthroughs/bad-missing-proven-by.md @@ -0,0 +1,11 @@ +# Missing proven-by + +```text +nself ci runs wait run_123 --json +``` + +## Proven by + +| Step | Test | +|---|---| +| Wait | `TestMachineDefaults` | diff --git a/scripts/docs/testdata/walkthroughs/bad-root-flag.md b/scripts/docs/testdata/walkthroughs/bad-root-flag.md new file mode 100644 index 000000000..ad6872c95 --- /dev/null +++ b/scripts/docs/testdata/walkthroughs/bad-root-flag.md @@ -0,0 +1,12 @@ +# Unknown root flag + + +```text +nself ci --detatch --json +``` + +## Proven by + +| Step | Test | +|---|---| +| Detached run | `TestMachineDefaults` | diff --git a/scripts/docs/testdata/walkthroughs/bad-subcommand.md b/scripts/docs/testdata/walkthroughs/bad-subcommand.md new file mode 100644 index 000000000..2815020f0 --- /dev/null +++ b/scripts/docs/testdata/walkthroughs/bad-subcommand.md @@ -0,0 +1,12 @@ +# Wrong subcommand + + +```text +nself ci nodes projects ad vps-a ~/src/project-a +``` + +## Proven by + +| Step | Test | +|---|---| +| Authorize | `TestMachineDefaults` | diff --git a/scripts/docs/testdata/walkthroughs/bad-unknown-test.md b/scripts/docs/testdata/walkthroughs/bad-unknown-test.md new file mode 100644 index 000000000..0c10aba04 --- /dev/null +++ b/scripts/docs/testdata/walkthroughs/bad-unknown-test.md @@ -0,0 +1,12 @@ +# Unknown test name + + +```text +nself ci runs wait run_123 --json +``` + +## Proven by + +| Step | Test | +|---|---| +| Wait | `TestNoSuchWalkthroughProof` | diff --git a/scripts/docs/testdata/walkthroughs/good.md b/scripts/docs/testdata/walkthroughs/good.md new file mode 100644 index 000000000..d7f84881d --- /dev/null +++ b/scripts/docs/testdata/walkthroughs/good.md @@ -0,0 +1,14 @@ +# Good fixture + + +```text +nself ci --detach --json +nself ci nodes projects add vps-a ~/src/project-a +nself ci runs wait run_123 --json +``` + +## Proven by + +| Step | Test | +|---|---| +| Detached run | `TestMachineDefaults` |