From 7d4d315022b5a2aef04e53a5b4cb619253c1cab1 Mon Sep 17 00:00:00 2001 From: Aric Camarata Date: Thu, 29 Oct 2026 14:40:09 -0400 Subject: [PATCH] docs(ci): add the core CI wiki pages and doc-commands workflow, fold the adoption guide (P7-CI-48) --- .github/wiki/Home.md | 3 +- .github/wiki/_Sidebar.md | 5 + .github/wiki/guides/Nself-CI-Adoption.md | 65 ----- .github/wiki/guides/ci/Architecture.md | 75 ++++++ .../guides/ci/Configuration-and-Policy.md | 251 ++++++++++++++++++ .../guides/ci/Evidence-and-Machine-Output.md | 110 ++++++++ .github/wiki/guides/ci/Getting-Started.md | 127 +++++++++ .github/wiki/guides/ci/Migration.md | 88 ++++++ .github/wiki/guides/ci/Troubleshooting.md | 136 ++++++++++ .github/workflows/doc-commands.yml | 80 ++++++ 10 files changed, 874 insertions(+), 66 deletions(-) delete mode 100644 .github/wiki/guides/Nself-CI-Adoption.md create mode 100644 .github/wiki/guides/ci/Architecture.md create mode 100644 .github/wiki/guides/ci/Configuration-and-Policy.md create mode 100644 .github/wiki/guides/ci/Getting-Started.md create mode 100644 .github/wiki/guides/ci/Migration.md create mode 100644 .github/wiki/guides/ci/Troubleshooting.md create mode 100644 .github/workflows/doc-commands.yml diff --git a/.github/wiki/Home.md b/.github/wiki/Home.md index 3576a32e3..fc0fc75b4 100644 --- a/.github/wiki/Home.md +++ b/.github/wiki/Home.md @@ -63,7 +63,8 @@ See [[All Plugins|#plugin-directory-sample]] below for a sample; the full catalo - [[Deployment|guides/Deployment]] - [[Migration|guides/Migration]] - [[Best Practices|guides/Best-Practices]] -- [[Adopting the nself-ci Gate|guides/Nself-CI-Adoption]] +- [[Getting started with nself ci|guides/ci/Getting-Started]] +- [[Migrating to the new CI engine|guides/ci/Migration]] - [[CI trust, security and privacy|guides/ci/Trust-Security-and-Privacy]] - [[Troubleshooting FAQ|troubleshooting/FAQ]] diff --git a/.github/wiki/_Sidebar.md b/.github/wiki/_Sidebar.md index df92e1ce6..0644beebf 100644 --- a/.github/wiki/_Sidebar.md +++ b/.github/wiki/_Sidebar.md @@ -171,12 +171,17 @@ ### CI +- [[Getting started with nself ci|guides/ci/Getting-Started]] +- [[CI architecture|guides/ci/Architecture]] +- [[CI configuration and policy|guides/ci/Configuration-and-Policy]] - [[Evidence and machine output|guides/ci/Evidence-and-Machine-Output]] - [[Nodes and Runners|guides/ci/Nodes-and-Runners]] - [[CI trust, security and privacy|guides/ci/Trust-Security-and-Privacy]] - [[CI scheduling and cost|guides/ci/Scheduling-and-Cost]] - [[Hosted providers|guides/ci/Hosted-Providers]] - [[Cache and artifacts|guides/ci/Cache-and-Artifacts]] +- [[Migrating to the new CI engine|guides/ci/Migration]] +- [[CI troubleshooting|guides/ci/Troubleshooting]] - [[CI reference|guides/ci/Reference]] - [[CI agent reference|guides/ci/Agent-Reference]] - [[Auto placement and pools|guides/ci/Auto-Placement-and-Pools]] diff --git a/.github/wiki/guides/Nself-CI-Adoption.md b/.github/wiki/guides/Nself-CI-Adoption.md deleted file mode 100644 index 86e257ed2..000000000 --- a/.github/wiki/guides/Nself-CI-Adoption.md +++ /dev/null @@ -1,65 +0,0 @@ -# Adopting the `nself-ci` Gate (Dogfood Pattern) - -**Status:** wired into `plugins` (this repo) 2026-09-11 — the first of 13 nself-org repos to adopt it. Zero repos had it wired before this, including `plugins` itself, which owns the source at `free/ci`. - -This guide is the copy-paste pattern for the other 12 repos. It also records, honestly, what the gate does and does not check today — read "Known gaps" before requiring it as a sole merge gate. - ---- - -## Why `plugins` first - -- It owns the `nself-ci` source (`free/ci/`) and the wrapper (`scripts/nself-ci.sh`) — self-hosting the gate here is the purest dogfood: a regression in the gate breaks its own repo's checks first. -- It is public, so the workflow runs on free GitHub-hosted Actions — no runner-fleet dependency to prove the pattern. -- Its own `.claude/docs/CI-LOCAL.md` (project-level) already documents the branch-protection recipe; this guide is the repo-level companion showing the wiring actually in place. - -## The pattern - -1. **Workflow** — `.github/workflows/nself-ci.yml`: checkout → `setup-go` (builds the binary) → `setup-node`/`pnpm` (for the Node gate path) → install `gitleaks` → run `scripts/nself-ci.sh .` with `permissions: statuses: write` and `GH_TOKEN: ${{ github.token }}` so `gh api` can post the status without a PAT. -2. **Wrapper** — `scripts/nself-ci.sh` builds `free/ci/nself-ci` if missing/stale, then `exec`s it. Same script works locally and in CI. -3. **Status** — the binary posts a `nself-ci` commit status via `gh api repos/{owner}/{repo}/statuses/{sha}` (OAuth, no token in a URL). -4. **Branch protection (NOT applied by this PR — owner's call)** — add `nself-ci` to `required_status_checks.contexts` per the exact `gh api` command in `~/Sites/nself/.claude/docs/CI-LOCAL.md` § "Requiring nself-ci in branch protection". Do this as an **additional** required check alongside the repo's existing checks, not a replacement — see gaps below. - -Copy `.github/workflows/nself-ci.yml` from this repo into each of the other 12 (`cli`, `admin`, `web`, `nchat`, `nclaw`, `nsentry`, `ntask`, `nfamily`, `clawde`, `homebrew-nself`, `packages`, `plugins-pro`), adjusting the toolchain setup steps to match that repo's stack (e.g. Flutter repos need `subosito/flutter-action` instead of `setup-node`). - ---- - -## Verified locally (2026-09-11, this repo, HEAD `5c17916`) - -``` -$ scripts/nself-ci.sh --check -v . -[nself-ci] building gate binary... -[nself-ci] built .../free/ci/nself-ci -[nself-ci] running: gitleaks detect --source . --exit-code 1 --config .github/gitleaks.toml - -nself-ci gate results — . -Stacks: node -──────────────────────────────────────────────────────────── - secrets:gitleaks PASS (3.617s) -──────────────────────────────────────────────────────────── - Overall: PASSED (4s) -``` - -That is the real, complete output — one gate ran (`secrets:gitleaks`), it passed, in under 4 seconds. Read that literally: **no lint, typecheck, test, or build gate ran**, for the reason below. - ---- - -## Known gaps — read before requiring this as the only gate - -**1. This repo's Node gate is a no-op.** `nself-ci` detects Node stacks by checking the root `package.json` for `lint` / `typecheck` / `test` / `build` scripts (or workspace members' scripts, if `pnpm-workspace.yaml` exists). This repo's root `package.json` has neither: it has one custom script, `ci:local` (a stub — `tsc --noEmit --allowJs || echo skip; echo passed`), and no `pnpm-workspace.yaml`, even though `shared/` has its own `package.json`. Result: the workspace-recursion path in `free/ci/internal/legacy/gate_runners.go` (`runNodeGates`) never triggers, the non-workspace path finds no matching scripts, and the gate silently runs nothing but the secret scan — while still printing `Overall: PASSED`. This is exactly the shape of failure flagged in `.claude/memory/lesson_hollow_ci_gates.md`: a gate that passes while checking nothing. - - **Fix options (not applied by this PR):** (a) add real `lint`/`typecheck`/`test`/`build` scripts to root `package.json` (or add `pnpm-workspace.yaml` so member scripts are picked up), or (b) extend `gate_runners.go` to also recognize this repo's `ci:local` convention. Until one of these lands, `nself-ci` on this repo is a secrets-only gate, not a lint/test/build gate. -2. **No per-plugin checks.** The real quality bar for this repo is per-plugin (schema validation, the conformance gate (ports included), registry consistency — see `validate.yml`, `registry-check.yml`, `hygiene.yml`, `license-gate.yml`). `nself-ci` only inspects the repo root; it does not walk `free/*/plugin.json` or run plugin-level TypeScript builds. It does not replace any of those workflows. -3. **No lint/typecheck/test/build for Go-only subpackages either.** `free/ci` itself (and other Go-based plugins) has a `go.mod`, but stack detection only looks at the *repo root*, which has no root `go.mod`. A regression inside `free/ci`'s own Go code would not be caught by `nself-ci` running against repo root — only by its own `go test ./...` if invoked directly in that subdirectory, or by `build-test.yml`. -4. **Per the plugin's own README** ("What it explicitly does NOT do"): `nself ci` is not a GitHub Actions emulator. E2E, Lighthouse, axe-a11y, CodeQL, Trivy, SBOM, license audit, coverage ratchet, bundle-size, commitlint, i18n-check, and deploy jobs all still need to run somewhere else — free-runner Actions for public repos (this repo), or a dedicated additional gate for private ones. -5. **Fork PRs:** `GH_TOKEN: ${{ github.token }}` on a fork PR is read-only for status writes; the binary treats a failed status post as a warning (non-fatal), so the job's pass/fail still reflects the real gate result, but the `nself-ci` check may not appear on fork PRs. - -**Bottom line:** this PR adds `nself-ci` as evidence the wiring works end-to-end (workflow → build → gates → status), not as proof the gate is a complete substitute for this repo's existing checks. Recommend requiring it in branch protection **in addition to**, not instead of, `validate.yml` / `build-test.yml` / `gitleaks.yml` / `registry-check.yml` until gap 1 is closed. - ---- - -## Rollout checklist for the other 12 repos - -- [ ] Copy `.github/workflows/nself-ci.yml`, adjust toolchain setup for the repo's stack. -- [ ] Run `scripts/nself-ci.sh --check -v .` (or `plugins/free/ci/nself-ci` directly) locally first — confirm which stacks it detects and which gates actually execute. Do not assume; read the printed gate table. -- [ ] If the repo's real lint/typecheck/test/build live under a non-standard script name (as `ci:local` does here) or in workspace members without `pnpm-workspace.yaml`, fix that *before* treating a green `nself-ci` as meaningful. -- [ ] Open a PR, let Actions run it once, confirm the `nself-ci` commit status appears on the PR. -- [ ] Propose (do not silently apply) the branch-protection addition per `CI-LOCAL.md` — owner approves it. diff --git a/.github/wiki/guides/ci/Architecture.md b/.github/wiki/guides/ci/Architecture.md new file mode 100644 index 000000000..6350cd0b6 --- /dev/null +++ b/.github/wiki/guides/ci/Architecture.md @@ -0,0 +1,75 @@ +# CI architecture + + + +This page explains how `nself ci` is built: the path a run takes, the layers of the code, the local store that holds state, and the rules that keep each part replaceable. Read it to reason about behaviour. To use the tool, start with [[Getting started with nself ci|guides/ci/Getting-Started]]. + +## The path of a run + +A run is a pure plan followed by an audited execution. Each step has one owner and one record. + +1. **Detect.** Manifests and lockfiles become facts: which stacks exist, which tools they need, which services a compose file declares. Detection reads files and runs nothing. +2. **Plan.** The configuration, the facts, the check catalogue and the revision context produce a pipeline: a validated graph of jobs. Every job in it, and every job left out, carries a reason code. The planner is a pure function, which is why `nself ci plan` can explain a run without starting one. +3. **Place.** Each job is matched to a runner: this machine, a node you enrolled, or a hosted provider. Placement is covered in [[CI scheduling and cost|guides/ci/Scheduling-and-Cost]] and [[Nodes and Runners|guides/ci/Nodes-and-Runners]]. +4. **Admit and lease.** The coordinator commits the job's demand against the host in one store transaction. If the host is full the job waits; it is never half-started. +5. **Execute.** The executor starts the job, enforces its timeout, passes its output through the redactor and kills the whole process tree on cancel. +6. **Record.** The result of each check, the inputs, the binding to the commit and the runner become one evidence document. [[Evidence and machine output|guides/ci/Evidence-and-Machine-Output]] describes it. + +A job moves through a closed set of states: queued, leased, running, finalizing, then passed, failed, cancelled or lost, with retryable and needs-operator for the cases that need a second attempt or a person. The transitions are a table; nothing else moves a job. + +## Layers + +The code under `free/ci` is split into layers, and a test fails the build when a package imports a layer it may not. The layer file is [layers.yaml](../../../../free/ci/architecture/layers.yaml). + +| Layer | Holds | Rule the test enforces | +|---|---|---| +| kernel | the data model, the schemas and the error and reason registries | imports no other package of the engine, and no `os` or `net` | +| pure | planning, detection, scoring, diagnosis, trust rules | imports no `os` and no `net` | +| execution | the process and container executor, the store | the only code that starts a process, and the only code that touches SQL | +| domain | everything that joins them: config, runs, reports, evidence, history, doctor, nodes, scheduling | may use the layers above; no package imports `cmd` | + +Two consequences matter in practice: + +- A pure package cannot touch the file system or the network, so planning and scoring are testable without a machine. +- There is exactly one place that starts a process or a container, the executor, and exactly one that speaks SQL, the store. Provisioning, ad hoc runs, drills and services all go through the executor, so redaction, timeouts and isolation are applied once and cannot be skipped by a new feature. + +## State and the store + +The engine keeps its state in one SQLite database named `state.db` in the state directory (`$NSELF_CI_HOME`). SQLite is a deliberate choice here. CI state is a local file owned by one user on one machine, not an application backend, so it does not use a database server. The driver is pure Go, built without cgo, so the same binary runs on every supported platform. + +**Local disk only.** Put the state directory on a local disk. SQLite in WAL mode depends on file locking and shared memory that network file systems do not provide reliably. Do not point `NSELF_CI_HOME` at an NFS or SMB share, and do not share one directory between machines. Several machines share work through a coordinator, not through a file. + +How the store is used: + +- **WAL, one writer.** The database runs in write-ahead-log mode with one writer connection and a busy timeout of five seconds. Readers never block the writer. Foreign keys are enforced. +- **Aggregate interface.** Callers do not write SQL. They call operations such as register a coordinator, admit and lease a job, record evidence, transition a job, request a cancel, back up and restore. Each operation is one transaction. +- **Compare-and-set transitions.** A state change names the state it expects. If another writer got there first, the change is refused with an error rather than overwritten. This is why two processes can share a store safely. +- **Expand-only migrations.** Schema changes only add. Each package registers its own migration set, and a version table records what ran. A stored minimum reader version tells an older binary that it must not open a newer database: it stops with a clear error and asks you to upgrade, instead of misreading rows. A downgrade therefore fails closed. +- **At-most-once for releases.** A release job, and any job marked `idempotent: false`, is never retried on its own. After a failure it waits in `needs-operator`, and a retry is allowed only after an operator action has been written to the audit table. A reused idempotency key is refused. +- **Audit sealing.** Security-relevant actions append rows to an audit table. Sealing chains the rows with hashes, so a deleted or edited row breaks the chain. The store holds no secret values in audit rows. Verification is `nself ci serve admin audit --verify`. +- **Host admission.** Admission is a list of steps run inside the lease transaction: the host's reservations, the demand of running jobs and any budget. Each step can refuse. Because the check and the lease commit together, two coordinators cannot both take the last slot. +- **Backup and restore.** A backup is a consistent snapshot of the database written while it is in use. A restore replaces the database and fences any coordinator that was running against the old one, so a stale process cannot write into the restored state. + +A coordinator is one process holding an identity in the store. A coordinator that stopped without finishing leaves a stale identity. On the next start the engine ends its in-flight attempts as lost or cancelled, closes runs that have no live coordinator, and sweeps containers it left behind. + +## What lives outside the store + +| Data | Where | Why | +|---|---|---| +| Logs | files under the state directory, one folder per attempt | large and append-only; redacted before they are written | +| Detached run streams | files under the state directory | the event stream of a run started with `--detach` | +| Branch-protection snapshots | files under the state directory, mode 0600 | the rollback for a policy change | +| Cache and artifacts | a content-addressed store | see [[Cache and artifacts|guides/ci/Cache-and-Artifacts]] | +| Configuration | `.nself/ci.yaml` in the repository | reviewed with the code it governs | + +## Extension seams + +Features plug in through hooks the run engine calls at fixed points, rather than by editing the engine. Examples are the policy resolver, the evidence builder, the report and status hooks, the services hook and the isolation level. A hook that panics is caught and the attempt ends with an error instead of taking the process down. This is how trust, scheduling and caching were added without changing how a job is planned or recorded. + +## Design rules worth knowing + +- **A pass is a claim with a proof.** A result is recorded as evidence that says exactly which inputs it covers, and it is bound to a commit only when the tree matched that commit. A dirty tree, or a tree that changed during the run, is recorded as unbound, and an unbound run posts no commit status. +- **Nothing hidden skips.** A job that does not run has a reason code. A required job that cannot run fails the run; it does not vanish. +- **Failures have a class.** Code, infrastructure, timeout, cancelled and lost are different, and only infrastructure failures are retried automatically. See [[CI troubleshooting|guides/ci/Troubleshooting]]. +- **One config, one schema.** `.nself/ci.yaml` is validated against a generated schema, and the same document describes the plan, the evidence and the events. +- **Generated reference.** Every table of commands, codes and versions is generated from the code, so the [[CI reference|guides/ci/Reference]] cannot drift from the binary. diff --git a/.github/wiki/guides/ci/Configuration-and-Policy.md b/.github/wiki/guides/ci/Configuration-and-Policy.md new file mode 100644 index 000000000..ba07c3f0c --- /dev/null +++ b/.github/wiki/guides/ci/Configuration-and-Policy.md @@ -0,0 +1,251 @@ +# CI configuration and policy + + + +Two things decide what `nself ci` does: the **configuration** in your repository, which says what to run, and the **policy** on your machine and your branches, which says what is allowed. This page covers both. The commands and codes it names are in the generated [[CI reference|guides/ci/Reference]]. + +## The configuration file + +The file is `.nself/ci.yaml` at the repository root. It is optional. With no file, `nself ci` detects your stacks and plans the standard checks. Declare any job and detection turns off for the jobs you declare; to tune a detected check instead, use `checks:` (below). + +```bash +mkdir proj && cd proj && git init -q +git config user.email docs@example.com && git config user.name docs +mkdir .nself && cat > .nself/ci.yaml <<'EOF' +version: 1 +jobs: + lint: + kind: static + run: [sh, -c, "echo lint"] + timeout: 5m + unit: + kind: test + run: [sh, -c, "echo test"] + deps: [lint] + retry: + infra_max: 2 +EOF +git add -A && git commit -q -m init +nself ci config show +nself ci plan +``` + +`config show` prints every effective value with its source: a flag, an environment variable, the file or a built-in default. Add `--json` for the same data as a document, where each value carries `value`, `source` and `origin`. + +**Precedence is flag, then environment, then file, then default.** That holds for every key, the timeout included: `--timeout` or `NSELF_CI_TIMEOUT` overrides a job's timeout in the file. + +**The file is validated strictly.** Unknown keys, wrong types, unsafe paths and unpinned images are refused with `E620`, naming the file, the line and a JSON pointer to the key. The message is built from the schema and never repeats the offending value, so a secret pasted into the wrong place is not echoed. Other rules: + +- Job, service and pipeline names, `workdir`, `when.paths`, `drills_dir` and cache and artifact paths must be safe paths inside the repository. +- `container.image` must be pinned by digest. +- Environment names must be variable names. Files are read only when regular and at most one mebibyte. +- A leftover legacy `.ci.yaml` is `E621`. A git-ignored `.nself/ci.yaml` is `E623`, and the message prints the lines to add to `.gitignore`. To move from the legacy file, see [[Migrating to the new CI engine|guides/ci/Migration]]. + +## Jobs + +A job is one command with a kind, and everything the planner needs to place and judge it. + +| Key | Meaning | +|---|---| +| `kind` | `static`, `build`, `test`, `quality`, `advanced`, `release` or `drill` | +| `run` | the command as a list; it is exec'd as given, with no shell unless you write one | +| `workdir` | directory inside the repository the job runs in | +| `deps` | jobs that must finish first; a failed dependency skips the job with `dep.failed` | +| `needs_services` | disposable services the job uses; see [Services](#services) | +| `matrix.platform` | a list of platforms such as `darwin/arm64`; one planned job per platform | +| `required` | `true` by default; an optional job cannot fail the run | +| `allow_failure` | the job runs and is reported, and its failure does not fail the run | +| `idempotent` | `false` marks a job that must never be retried automatically | +| `timeout` | a duration such as `5m` | +| `retry.infra_max` | how many times an infrastructure failure may be retried | +| `priority`, `path` | `low`, `normal`, `high`; `fast` or `deep` | +| `when` | `branches`, `paths`, `triggers` and `env_present` conditions | +| `env`, `secrets` | environment values, and names of secrets the job may read | +| `isolation`, `container` | the minimum isolation level; an image pinned by digest | +| `artifacts`, `cache` | outputs to keep, and cache keys and paths | +| `requires` | runner requirements: `labels`, `tools`, `accelerators`, `cpu`, `mem_mb` | + +**Every job has a reason.** `nself ci plan` shows, for each job, whether it is included or skipped and the reason code: `pipeline.excluded`, `selection.affected`, `when.paths`, `when.env_missing` and so on. `nself ci explain ` shows where a job would be placed and why. The full list of reason codes is generated in the [[CI reference|guides/ci/Reference]]. + +**Affected selection.** When a base commit is known, a job whose `when.paths` do not match the change is left out, except that a default-branch or release plan always keeps every required deep job, even one outside the named pipeline. With no base the whole plan runs, with the reason `selection.no_base`. A job that does not run always has a reason code, so a skip is never silent. + +**Matrices come from your support policy.** `support:` lists the platforms you support, and a matrix is expanded from it, never invented. + +### Requirements and Xcode + +`requires` narrows which runners may take a job. A job that needs Xcode names the `xcode` tool and the platform, and the planner only places it on a Mac where `xcodebuild -version` actually runs. The Command Line Tools alone do not count, because they ship a stub: + +```bash +mkdir mac && cd mac && git init -q +git config user.email docs@example.com && git config user.name docs +mkdir .nself && cat > .nself/ci.yaml <<'EOF' +version: 1 +jobs: + ui-test: + kind: test + run: [xcodebuild, test, -scheme, App, -destination, "platform=macOS"] + matrix: + platform: [darwin/arm64] + requires: + tools: [xcode] + timeout: 20m +EOF +git add -A && git commit -q -m init +nself ci config show +``` + +Then, on the machine you care about, see where the job would go: + +```text +nself ci plan +nself ci explain ui-test +``` + +On a machine that cannot satisfy the platform or the tool, the planner reports the job with an `unplaceable.*` reason such as `unplaceable.platform` or `unplaceable.tool` instead of starting it, which is the point: it waits for a Mac node. Detected Swift projects get this requirement automatically. See [[Nodes and Runners|guides/ci/Nodes-and-Runners]] for adding a Mac. + +## Checks + +When you declare no jobs, the catalogue plans checks per detected stack. Each check is either required, so a failure fails the run, or advisory, so it is reported but does not. + +| Stack | Required by default | Advisory by default | +|---|---|---| +| Go | `go.fmt`, `go.vet`, `go.test` | `go.govulncheck`, and a coverage floor when you configure one | +| Node | `node.lint`, `node.typecheck`, `node.test`, `node.build` | | +| Rust | `rust.clippy`, `rust.test` | | +| Flutter | `flutter.analyze`, `flutter.test` | | +| Swift | `swift-build`, `swift-test`, `xcode-test` | | +| Python | `python-test` | `python-lint` (ruff), `python-types` (mypy, planned only when mypy is configured) | +| Containers | | `container.hadolint`, `container.trivy.config` | +| Any | `secrets.gitleaks` | `security.trivy.fs` | + +The `container`, `release` and `strict` modes require the advisory checks as well, whatever `required:` says. The Python lint and type checks become required in the same modes and under `--container`. + +A missing tool depends on the check. A missing required tool, or an unsupported Node package manager, ends the check with `E614` and a remediation line. A missing advisory tool skips the check with `advisory.tool_missing`, and unavailable Trivy data skips it with `advisory.data_offline`. + +Tune a check under `checks:`: `required`, `env`, `secrets`, `coverage_floor` and, for the truth check, `rules`. Turn off an ecosystem with `detect.disable`, and point detection at subdirectories with `detect.roots`. + +### Node preparation + +Before a Node check runs, the job finds the lockfile in the working directory or above, and checks the declared package manager's major version against the one on `PATH`. A mismatch is `E811` and the job does not retry. If there is no `node_modules`, it runs a frozen install: `npm ci`, `pnpm install --frozen-lockfile`, `yarn --immutable` or its older form, or `bun --frozen-lockfile`, by lockfile. Caches and the pnpm store live inside the job workspace. Astro projects run `astro sync` before typecheck and build. A failed install is classified as a code failure when the lockfile is out of sync or a lifecycle script failed, and as an infrastructure failure when the registry or disk was the cause, which is retried. See [[CI troubleshooting|guides/ci/Troubleshooting]]. + +### The secret scan + +`secrets.gitleaks` honours `.gitleaksignore`, with one rule: every entry needs a review comment on the same line or the line before. + +```text +# Reviewed 2026-10-29 by alice: a documented test value, not a credential + +``` + +An entry without one fails the check with `E812`, which names line numbers and never prints content. Add entries with `nself ci config ignore add --reason [--by ]`. A run never adds lines itself. A missing `gitleaks` is `E614`, not a skip. + +### The truth check + +The truth check keeps retired names, dead hosts and banned strings out of your tree. It makes no network calls. Rule files are listed under `checks.truth.rules` in `.nself/ci.yaml`, as repository-relative paths; a rule file may `include` others to a limited depth, and cycles are refused. + +A rule file has `version: 1` and these parts: + +- `rules[]`, each with an `id`, a `kind`, a `message` and optional `paths` globs. The kinds are `banned-literal` (a `literal`), `retired-term` (a `term`, matched case-insensitively as a whole term) and `dead-host` (a list of SHA-256 hashes, so the file does not publish the host). +- `skip[]`, globs that are never scanned. +- `allow[]`, each naming a `rule` (an id, or a glob such as `retired-*`), the `paths` it covers and a `reason` of at least eight characters. + +Unknown keys, duplicate ids, bad hashes, inert globs, extra YAML documents and symlinked rule files are `E644`, naming the file and the key. A dead-host rule stores the hash of the lower-cased host. The engine hashes every address and host name token on a line and compares the whole token, so a sub-domain is a different host. + +The scan covers regular files under the root except `.git`, `node_modules`, the rule files themselves and your `skip` globs. A file over four mebibytes, or with a NUL byte, is skipped and counted in the summary; a tree with nothing to scan is an error. The evidence holds one check per rule kind. Findings print `path:line`, the rule id and the message, and a dead-host finding prints only a hash prefix. + +## Services + +A job that needs a database or cache names it, and gets a throwaway nSelf project for that attempt. + +```text +services: + db: + component: postgres + version: "16" +jobs: + integration: + kind: test + run: [go, test, ./...] + needs_services: [db] +``` + +A name is either declared under `services:` or is the name of a component. `--service ` adds one to a single run. Each attempt gets its own project on random loopback ports with random credentials, and the job sees these variables, with the service name upper-cased: `NSELF_CI_SERVICE__HOST`, `_PORT`, `_USER`, `_PASSWORD`, `_DB` and `_URL`, the same set under `_NET_` for a container job, and `NSELF_CI_SERVICE_NETWORK`. + +Services need Docker and an nSelf with the `config services` command: `E637` means no Docker, `E638` means the CLI is too old (run `nself update`), `E639` means an unknown service, a duplicate component or a bad version, and `E636` means provisioning or teardown failed. Services are torn down on pass, fail, cancel and timeout, and after a hard kill the next run sweeps them. Services need the v1.5 engine. + +## Pipelines, schedules and drills + +A named pipeline picks a set of jobs for a set of triggers: + +```bash +mkdir sched && cd sched && git init -q +git config user.email docs@example.com && git config user.name docs +mkdir -p .nself scripts/drill/restore +printf '#!/bin/sh\necho restored\n' > scripts/drill/restore/run +chmod +x scripts/drill/restore/run +cat > .nself/ci.yaml <<'EOF' +version: 1 +jobs: + report: + kind: quality + run: [sh, -c, "echo report"] + required: false +pipelines: + nightly: + triggers: [schedule, manual] + schedule: "0 3 * * *" + max_age: 36h + drills_dir: scripts/drill + jobs: [report] +EOF +git add -A && git commit -q -m init +nself ci plan --pipeline nightly +``` + +`nself ci plan --trigger schedule` plans the pipelines that list the trigger. A **drill** is a directory under `drills_dir` with an executable `run`; each becomes a job named `drill-`, in a stable order. The exit code is the result. A drill may print one `ci.evidence` document as a single line on stdout, which is validated and nested as the job's checks. A config job with the same name replaces the discovered one, which is how a drill gets its own timeout, since discovered drills use the default. + +`max_age` makes a silent job visible. `nself ci evidence --pipeline ` reports each job's last passing run and marks a job `overdue`, with exit code 10, when it is older than `max_age`. A pipeline without `max_age` is never overdue. A scheduled pipeline on a throwaway hosted runner has no store, so there staleness shows as a missing workflow run. The org-neutral workflow for it is `free/ci/templates/scheduled-pipeline.yml`. + +## Policy + +Policy is not in the repository. It lives in your user configuration, in a team coordinator's configuration, and on the host. Read it with `nself ci policy show` and change it with `policy set`, `policy unset` and `policy usage`. Trust, privacy and spend keys are covered in [[CI trust, security and privacy|guides/ci/Trust-Security-and-Privacy]], scheduling keys in [[CI scheduling and cost|guides/ci/Scheduling-and-Cost]]. + +### Per-run values + +Three flags change policy for a single run: `--prefer`, `--max-queue` and `--runner`, with `--where` for the run target. They apply to bare runs only; `nself ci plan` ignores them. + +- A per-run value can only **tighten**. It is merged last and refused if it would loosen a trust or privacy key. A refusal is `E819`, shown as a run note, and the key is ignored. +- Spend and secret keys cannot be set per run. +- A key that is unset may be filled only in the scheduling area, for the runner, operating system, architecture and allowed providers, because an unset trust list means "deny all". +- `--prefer` is the exception to tighten-only: any preset in the closed set is accepted, so you can override a protected branch's preset for one run. +- `--runner` pins only runners that your dispatcher knows; before nodes are enrolled that is the local machine. +- An interactive run, one started from a real terminal with no trigger and no `NSELF_INVOKED_BY`, waits at most ten minutes for a runner by default. A scripted run does not get that cap, and a `max_queue` in your policy longer than ten minutes stands. Git Bash terminals and runs wrapped by an agent count as scripted. + +The base policy digest is what evidence records, so `evidence verify` still matches a run that used per-run values. + +### Branch protection, hook and audit + +`nself ci policy protect` makes `nself-ci` a required check on a branch, safely, and gives you the pieces to undo it. + +```text +nself ci policy protect --repo owner/repo --branch main --dry-run +nself ci policy protect --repo owner/repo --branch main +nself ci policy protect --repo owner/repo --branch main --restore +nself ci policy protect --hook +nself ci policy protect --audit 30 +``` + +- **Dry run** prints the exact request body. Existing required checks keep their pinned app, the `nself-ci` check is appended (override the app with `--app-id`), every other field is replayed from the live answer, and status checks are always sent as `checks`, never as `contexts`. A protection block with an unknown field, or with required signatures on, refuses the plan. An unprotected branch gets protection that holds only the check. +- **Apply** writes a snapshot of the current protection first, to `$NSELF_CI_HOME/protect/` with mode 0600 and never overwriting a file. It sends the body from a file, never on the command line, then reads the result back and compares it, ignoring the order of checks. +- **Restore** saves what it replaces, then puts the snapshot back. A snapshot of an unprotected branch cannot be restored; removing protection is a manual step. +- If the host's plan has no branch protection, the command prints the hook and audit equivalents and exits 3 (`E643`). Other 403 responses, such as a rate limit or a token without rights, are ordinary errors. +- **Hook** installs a pre-push hook for the default branch at git's own hooks path. It runs `NSELF_V15=1 nself ci evidence verify --json` and refuses the push on any non-zero exit: missing, stale, invalid, unbound or failing evidence. It also refuses when neither `nself` nor `nself-ci` is on the path. An existing hook is never overwritten (`E642` prints the line to chain it). `git push --no-verify` is your override. +- **Audit** lists commits on the default branch, newest first, that have no green `nself-ci` status and no green check run from the pinned app, and exits 1 when there are any. + +Exit codes: 0 success, 1 failure or missing commits or `E642`, 2 infrastructure, 3 `E643`. + +### The workflow template + +Hosted runs can produce the same check from a workflow. Copy `free/ci/templates/nself-ci.yml` to `.github/workflows/nself-ci.yml`, set the three release values it asks for (the plugin version, the release URL and its SHA-256), add setup for your own toolchain where it says so, and then run `policy protect`. Nothing is downloaded until all three values are set. + +The template keeps rules that make the gate honest: the job is named exactly `nself-ci`, because branch protection matches that name; it has no `paths:` filter, no matrix and no cancel-in-progress, because a skipped or cancelled required check blocks merges or hides a red run; its actions are pinned by commit; it holds read-only repository permissions and posts no commit status; and the binary is the pinned release, never one built from the checkout under test. A check run, unlike a commit status, is also produced on fork and Dependabot pull requests, whose tokens are read-only. A pull request that edits the workflow controls its own check run, which is inherent to hosted runners, so protect `.github/workflows` with code owners as well. diff --git a/.github/wiki/guides/ci/Evidence-and-Machine-Output.md b/.github/wiki/guides/ci/Evidence-and-Machine-Output.md index 52b4422ec..177e64a4f 100644 --- a/.github/wiki/guides/ci/Evidence-and-Machine-Output.md +++ b/.github/wiki/guides/ci/Evidence-and-Machine-Output.md @@ -1,5 +1,115 @@ # Evidence and machine output + + +Every run of `nself ci` leaves a record you can read, verify and hand to another program. This page covers that record, the machine-readable output of the commands, how to start a run and collect it later, and ad hoc commands. For the first run, see [[Getting started with nself ci|guides/ci/Getting-Started]]. + +## Evidence + +A run's evidence is one JSON document of schema `ci.evidence/v1`. It states which revision was checked, which inputs the result covers, which runner produced it, what each check concluded and how the document itself was produced. The tables at the end of this page are generated from the schema and list every field. + +Read it with: + +- `nself ci evidence --json`, the evidence of one run. +- `nself ci evidence --pipeline `, the last passing run of each job of a scheduled pipeline, with `overdue` marked when it is older than the pipeline's `max_age`. +- `nself ci evidence verify [--json]`, a yes or no for a commit under the policy in force. + +### Binding and freshness + +**Binding** says whether the result is about a commit. It is `bound` when the working tree matched the commit throughout the run. It is `unbound`, with a reason, when the tree was dirty or changed during the run. Hidden edits count: the tree is read again when the run completes. Tracked files marked `skip-worktree` or `assume-unchanged` unbind the run, except a `skip-worktree` path that is absent because of a sparse checkout. Ignored files such as `.env` or `go.work` stay outside the binding, and jobs still see them, which is a known limitation. + +**Freshness** says whether the evidence still describes the present: `current`, `stale` or `unknown`. It is computed when you read the evidence, never stored, and a document's own claim is never trusted. Evidence goes stale when the inputs it covers have changed, when it was made under another policy, or when the working tree has uncommitted changes. It is `unknown` when the inputs cannot be read, and unknown is never treated as current. + +### Verify + +`evidence verify ` returns a verdict you can act on in a script or a hook. It serves stored evidence only when that evidence is a finished run, with a full selection, of the default pipeline, for this repository, under the policy in force. Anything else is rerun in a temporary worktree with hooks, file-system monitors and LFS smudging turned off, and always with a full selection. An interrupted or errored rerun exits 2. The verdict is the exit code: 0 for current, bound, passing evidence; 1 when the evidence fails; 10 when it is stale, which includes another policy; 11 when it is invalid, which includes another repository; and 12 when it is unbound. Evidence from ad hoc commands and drills never gates a commit. The whole exit-code table is generated in the [[CI reference|guides/ci/Reference]]. + +`--file ` verifies a document you were given. A file alone proves nothing about where it came from, so a file with no local record of its run is refused with exit 11 when it would otherwise pass. A document signed by a remote runner goes through the registered verifier, and its repository and policy digest are checked. + +## Machine output + +Add `--json` to any command and it prints one envelope on stdout: + +```text +{ "schema_version": "1", "command": "ci runs wait", "data": { ... } } +{ "schema_version": "1", "command": "ci plan", "error": { "code": "E620", "message": "...", "exit_code": 1, "class": "user" } } +``` + +`data` is the result, and `error` replaces it on failure with a stable code, the exit code and a class. Programs should read the code and the exit code, not the message. + +A run also emits a stream of events with schema `ci.events/v1`, one JSON object per line, in order: `queued`, `planned`, `leased`, `started`, `log`, `check_result`, `retrying`, `cancelled`, `needs_operator`, `finished` and `evidence`. Each carries a sequence number, a time, the run and, when it applies, the job. `--stream` prints it live. One writer produces the stream; log records are best effort and are dropped after a bounded wait rather than slowing a job, with no gap in the sequence. A failed stream flushes its error line first. + +Usage errors exit 1 across the tool. The full table of exit codes and error codes is generated in the [[CI reference|guides/ci/Reference]]. + +## Start a run and come back later + +A foreground run holds your terminal. An agent, a script or an editor often cannot wait, so a run can be started in the background and collected later. The handle is a document of schema `ci.run-handle/v1`, which is experimental until the next major version. + +```text +NSELF_V15=1 nself ci --detach --json [flags] [repo] [-- ] +``` + +`--detach` starts the run in a new session, detached from your terminal, and prints the handle once the run is queued. The handle gives the run id and argument lists for the next steps: `wait`, `follow`, `evidence`, `cancel` and `logs`. If the run cannot be queued within thirty seconds, the command fails with `E648` and a reason (`spawn`, `no_queued_event` or `child_exit`); it ends the child and shows a redacted tail of what the child printed. + +Then choose how to collect it: + +- **Poll.** `nself ci runs wait [--timeout d] [--json]` waits up to the timeout, and exits 0 when the run is over or 10 while it is still running. When it exits 0, read `data.state` for the outcome: `passed`, `failed`, `cancelled` or `lost`. The timeout defaults to four minutes and is shortened to fit `NSELF_INVOKE_DEADLINE_MS` when a caller sets one, so a tool with its own time limit never hangs. Exit 10 means call it again. +- **Stream.** `nself ci runs follow [--from-seq n]` prints the events of the run as NDJSON. It replays what happened and tails the run until the `result` line. A run with no stored stream is rebuilt from the store in the same order, so the sequence never depends on when you looked. A run that died without a result ends with an `E648` error line and exit 2. +- **Read the result.** `nself ci evidence --json`. +- **Stop it.** `nself ci runs cancel `. + +A complete example, which starts a run, waits for it with a bounded loop, then reads the events and the evidence: + +```bash +mkdir async && cd async && git init -q +git config user.email docs@example.com && git config user.name docs +mkdir .nself && cat > .nself/ci.yaml <<'CFG' +version: 1 +jobs: + show: + kind: test + run: [sh, -c, "echo hello"] +CFG +git add -A && git commit -q -m init +nself ci --detach --json --no-status > handle.json +run=$(sed -n 's/.*"run_id": "\(.*\)",/\1/p' handle.json) +rc=10 +while [ "$rc" -eq 10 ]; do + rc=0 + nself ci runs wait "$run" --timeout 30s --json > wait.json || rc=$? +done +[ "$rc" -eq 0 ] +grep -q '"state": "passed"' wait.json +nself ci runs follow "$run" > events.ndjson +grep -q '"type":"finished"' events.ndjson +nself ci evidence "$run" --json > evidence.json +grep -q '"result": "pass"' evidence.json +``` + +A few limits to know. Streams and pid files of runs that no longer exist are removed by `runs prune` and by every `--detach`. A system that kills every process in a control group when the caller exits can still kill the detached child; start the run from a context that outlives the caller if you rely on it. `--coordinator` is read from the arguments or `NSELF_CI_COORDINATOR`. + +## Ad hoc commands + +`nself ci [flags] -- [args]` runs one command through the same machinery: a real run with a store entry, a log and an evidence document, but a pipeline of kind `adhoc` that never gates anything and never posts a status. It is the safe way to let a person or an agent run a command with a record. + +| Flag | Meaning | +|---|---| +| `--isolation container\|process\|sandboxed-container` | the isolation level; `container` is the default and needs Docker | +| `--image ` | the container image, pinned by digest | +| `--network none\|restricted\|internet\|lan` | the network scope; none by default, or the service network when `--service` is given | +| `--service ` | a disposable service the command can reach | +| `--trust owner\|collaborator\|internal\|untrusted` | the trust class of the command's source | + +The command is exec'd as given. The workspace is mounted read-only, scratch space is a temporary file system, and the environment is only a language setting, a home and a temporary directory. + +**What isolation does and does not do.** A container reduces what a command can touch, and it is not a boundary against hostile code. `--isolation process` runs on the host with the host network and a writable copy of the workspace, prints a warning and records the network scope as `lan`. An `untrusted` source needs the sandboxed level, and a host without it refuses the command with `E647` and runs nothing. `--network internet` is refused until the network planner for the level allows it, and is never widened silently. Your model of risk is in [[CI trust, security and privacy|guides/ci/Trust-Security-and-Privacy]]. + +**What it records.** The evidence names the isolation level actually used, the requested trust class, the redacted command line and the network scope. Redaction of the command line goes by the shape and name of an argument, so a secret passed as a bare positional argument is not recognised. Do not pass secrets that way. + +## Evidence fields + +The tables below are generated from the schema. Change the model, then regenerate. + #### Evidence document (root) diff --git a/.github/wiki/guides/ci/Getting-Started.md b/.github/wiki/guides/ci/Getting-Started.md new file mode 100644 index 000000000..2ea4480cd --- /dev/null +++ b/.github/wiki/guides/ci/Getting-Started.md @@ -0,0 +1,127 @@ +# Getting started with nself ci + + + +`nself ci` runs your project's checks on your own machine and records what happened as signed-off evidence. It needs no cloud account, no hosted runner and no network beyond what your own tools use. This page takes you from nothing to a first recorded run, then covers what is detected, what stays local, the history you get for free, and container mode. + +Everything on this page is the v1.5 engine. Turn it on with `NSELF_V15=1` in your environment. Without it, `nself ci` runs the older gate; see [[Migrating to the new CI engine|guides/ci/Migration]]. The commands, flags and codes are listed in the generated [[CI reference|guides/ci/Reference]]. + +## Install + +Install the plugin once, then check the machine: + +```text +nself plugin install ci +NSELF_V15=1 nself ci doctor +``` + +`doctor` checks the tools your repository needs, the container runtime, status posting, the local store and your configuration. Each finding has an id, a one-line reason and, when it can, a fix. A warning never blocks a run. Exit codes and the full list of ids are in the [[CI reference|guides/ci/Reference]]; what to do about one is in [[CI troubleshooting|guides/ci/Troubleshooting]]. + +## Your first run + +Make a project, commit it, and ask what `nself ci` would do before it does anything: + +```bash +mkdir hello && cd hello && git init -q +git config user.email docs@example.com && git config user.name docs +printf 'module example.com/hello\n\ngo 1.22\n' > go.mod +printf 'package main\n\nfunc main() {}\n' > main.go +git add -A && git commit -q -m init +nself ci plan +nself ci config show +``` + +`plan` prints the job graph: each job, whether it is required, why it was chosen (`selection.no_base` here, because nothing says what changed, so everything runs) and where it would be placed. `config show` prints every effective setting with the layer it came from. With no `.nself/ci.yaml` the detected defaults apply. + +A bare `nself ci` then runs the plan: + +```text +NSELF_V15=1 nself ci +``` + +It runs the required jobs, prints one line per job, records the run in a local store, and ends with a result line. When the repository has a GitHub remote it also posts a `nself-ci` commit status; with no remote it says in one line why it did not. Add `--no-status` to never post. + +To run any single command through the same machinery, put it after `--`. The command runs in a container by default, with no network. On a machine without Docker, `--isolation process` runs it on the host instead and says so: + +```text +nself ci --isolation process --no-status -- cat file.txt +``` + +An ad hoc run is recorded like any other, as a pipeline of kind `adhoc`, and never posts a status. The flags and their limits are in [[Evidence and machine output|guides/ci/Evidence-and-Machine-Output]]. + +## What is detected + +`nself ci` reads manifests and lockfiles, never your source, to decide which checks apply. A check is planned only when its stack is present. + +| Stack | Detected from | Checks | +|---|---|---| +| Go | `go.mod` | format, vet, test, plus a vulnerability scan that is advisory | +| Node | `package.json`, with npm, pnpm, yarn or bun chosen by lockfile; workspaces are walked | lint, typecheck, test, build | +| Rust | `Cargo.toml` | clippy, test | +| Flutter and Dart | `pubspec.yaml` | analyze, test | +| Swift | `Package.swift`, or Xcode projects and workspaces with a shared scheme | `swift build` and `swift test`; `xcodebuild test` on a Mac with Xcode | +| Python | `pyproject.toml` or requirements files; uv, poetry or pip chosen by lockfile | pytest when configured or when `tests/` exists; ruff and mypy when configured | +| Dockerfiles | `Dockerfile*` | hadolint and a configuration scan, both advisory | +| Secrets | any repository | `gitleaks` | + +Lockfile digests are recorded as evidence inputs. Services declared in a compose file are noticed, but a client library or an env sample never implies a service: a job gets a database only when you name one. See [[CI configuration and policy|guides/ci/Configuration-and-Policy]]. + +Notes on the Apple and Python stacks: + +- Xcode checks need a shared scheme. Share the scheme in Xcode, or the project yields no check. The destination is the Mac itself, so iOS-only schemes need a custom job. +- With `uv.lock` the Python checks use uv; otherwise with `poetry.lock` they use poetry; otherwise pip into a `.venv` in the job root. +- Which checks are required and which are advisory is the table in [[CI configuration and policy|guides/ci/Configuration-and-Policy]]. + +## Local-only, and what that does and does not give you + +The default run is local. Nothing leaves the machine unless a tool you run reaches out or you post a status. A drill in this repository proves the first run with the network cut off. + +- **Reservations.** A run started from your terminal uses the machine freely. Background work leaves headroom for you: by default two CPUs and a quarter of memory stay free, and deep jobs wait while a laptop is on battery below a threshold. A held job shows the reason code `admission.reservation` or `admission.battery`. `--force` ignores the reservations. A run under `CI=true` or without a terminal does not reserve anything. +- **Isolation.** The `process` level runs a job as a child process in a copy of your workspace with a controlled environment, a time limit and a kill of the whole process tree on cancel. It is not a sandbox: the job shares your network, your user and your disk. The container level is stronger, and is still not a boundary against hostile code. Treat both as guard rails for your own code. The model is in [[CI trust, security and privacy|guides/ci/Trust-Security-and-Privacy]]. +- **Secrets.** Job output passes a redactor before it is stored, so a stored log never holds a value the redactor knew. Evidence holds redacted excerpts only. + +## History, logs and retention + +Every run is stored locally, with its jobs, attempts, evidence and logs. + +- `nself ci runs list [--limit N] [--json]` lists recent runs with result, duration, revision, binding and freshness. +- `nself ci runs ` shows one run. `nself ci logs [-f] [--json] [--run ] [--attempt N]` prints its log, and `-f` follows a live one within about a second. +- Logs hold redacted output only, as JSON lines with a sequence number. The sequence is the order; two lines in one burst can share a timestamp. + +Retention keeps the newest runs, drops old ones and caps the size of stored logs. It runs when a run starts and on `nself ci runs prune [--dry-run]`, and a run in progress is never removed. The limits are defaults you override in `retention.json` in the state directory. `runs prune` states what it removed. + +```bash +mkdir keep && cd keep && git init -q +git config user.email docs@example.com && git config user.name docs +mkdir .nself && cat > .nself/ci.yaml <<'CFG' +version: 1 +jobs: + check: + kind: test + run: [sh, -c, "echo checked"] +CFG +git add -A && git commit -q -m init +nself ci --no-status +nself ci runs list +nself ci logs check +nself ci runs prune --dry-run +``` + +The state directory is `$NSELF_CI_HOME`, which defaults to a folder in your home directory. + +## Container mode + +`nself ci --container` runs each job in one pinned toolchain image instead of the tools on your machine, so a result does not depend on which Go or Node you have installed. The image carries its tools by checksum, and the pinned versions are in the generated table of the [[CI reference|guides/ci/Reference]]. The first run on a host pulls the whole image; after that a tool bump re-pulls one layer. + +- The image reference must be pinned by digest. A tag alone is refused. `--image ` or `NSELF_CI_IMAGE` selects another image. +- There is no silent fallback to host tools. If the image is not available the run stops with an error that names the problem. +- Container mode needs Docker or a compatible runtime. `doctor` tells you whether it is reachable. + +Flutter has no tool in the image, so Flutter checks always run with the host tools. + +## Next steps + +- Describe your own jobs, services and pipelines: [[CI configuration and policy|guides/ci/Configuration-and-Policy]]. +- Read results as data, or start a run and come back later: [[Evidence and machine output|guides/ci/Evidence-and-Machine-Output]]. +- Add a second machine: [[Nodes and Runners|guides/ci/Nodes-and-Runners]]. +- Something failed: [[CI troubleshooting|guides/ci/Troubleshooting]]. diff --git a/.github/wiki/guides/ci/Migration.md b/.github/wiki/guides/ci/Migration.md new file mode 100644 index 000000000..be7e6cf09 --- /dev/null +++ b/.github/wiki/guides/ci/Migration.md @@ -0,0 +1,88 @@ +# Migrating to the new CI engine + + + +This page is for you if you already use the earlier `nself ci` gate, the `.ci.yaml` manifests that `nself ci run` discovers, or the wrapper-script workflow that older versions of this wiki described. It lists what changes, how to move your configuration, and how to adopt the gate in another repository. It replaces the old adoption guide. + +The new engine is on when `NSELF_V15=1` is set. The plugin binary does not follow the CLI's own switch yet, so the workflow template sets the variable itself. Until the older path is removed, both run side by side, and you can try the new one in a shell without touching a team workflow. + +## What changes + +| Before | Now | +|---|---| +| Configuration lived in `.ci.yaml` files per plugin, discovered by `nself ci run` | One `.nself/ci.yaml` at the repository root, validated against a schema. `nself ci run` becomes `nself ci --trigger manual`. | +| A fixed gate per stack, read from the repository root | Checks per detected stack and directory, planned as a job graph with a reason for every job. See [[CI configuration and policy|guides/ci/Configuration-and-Policy]]. | +| A pass or a fail, printed | A pass, fail, error or cancelled result with evidence, history, logs and a stored record. See [[Evidence and machine output|guides/ci/Evidence-and-Machine-Output]]. | +| The commit status was posted by every run | A status is posted only for a full, bound, default-pipeline run from an owner or internal source that was not cancelled. `--job`, `--pipeline`, a dirty tree, a tree that changed during the run and a `--config` outside the repository never post. | +| A missing `gitleaks` was treated as a pass | A missing required tool is `E614`, so the secret scan can no longer pass by being absent. Install `gitleaks` or the run stops. | +| `--sha` chose the commit | Not supported. Use `nself ci evidence verify `. | +| `nself ci eval` was part of the gate | `eval` is a separate command of the `nself-eval-gate` plugin, `nself eval`. A project that wants it in CI runs it as a job. | +| `nself ci build --artifact android` built in place with Gradle | The arguments are unchanged. Under the new engine it runs a release pipeline and publishes a release directory you can verify. The earlier path is removed when the earlier engine is. | +| Output was text | `--json` gives a stable envelope for every command, and `--stream` gives events. | + +Two rules for upgrades and downgrades: the evidence documents gained fields, and an older build rejects a newer document, so downgrading means regenerating evidence. The local database refuses to open under a binary older than it needs, with a clear error. + +## Move your configuration + +`nself ci config migrate` reads the legacy `.ci.yaml` files and prints the `.nself/ci.yaml` they would become. Nothing is written until you ask: + +```bash +mkdir legacy && cd legacy && git init -q +git config user.email docs@example.com && git config user.name docs +cat > .ci.yaml <<'EOF' +plugin: example +stages: + - name: unit + cmd: ["go", "test", "./..."] + timeout: 120 + - name: integration + cmd: ["go", "test", "-tags", "integration", "./..."] + env_required: + - DATABASE_URL + skip_if_env_missing: true +EOF +nself ci config migrate +``` + +Run it, read the diff and the notes it prints, then write the file with `nself ci config migrate --write`. It never writes through a symlink and never overwrites a different file. When you are satisfied, delete the legacy files; a leftover `.ci.yaml` is reported as `E621`. + +**Migration follows what your file declares.** The earlier runner did not parse YAML. It read `timeout` and `skip_if_env_missing` only at one indentation, read list items only at another, filed all-caps list items under `env_required` and skipped a manifest with no top-level `plugin:` key. A manifest written as valid YAML, with `cmd: [..]` on one line, could therefore declare commands that never ran. `migrate` prints a note for each stage and field where the old reading and the declared meaning differ, so the first run under the new engine may run commands the old one silently skipped. That is the intended change. + +What carries over: + +| Legacy | Becomes | +|---|---| +| `cmd` | `run` | +| `timeout` | `timeout`, in seconds converted to a duration; a stage without one gets the old default of 300 seconds, written explicitly | +| `env_required` with `skip_if_env_missing` | `required: false` and `when.env_present` | +| stage order | nothing: jobs are independent | + +The last row matters. The old runner ran stages in order in one directory and gave no information about which stage consumed another's output, so `migrate` invents no dependencies and prints a note. Add `deps:` wherever one stage reads files another writes, or merge the commands into one job. Null or nested list items, anchors, aliases, merge keys, a second YAML document and unknown fields are refused with `E622`. + +## Adopt the gate in another repository + +This is the pattern first proven on this repository, rewritten for the current engine. It replaces the earlier wrapper-script recipe, whose script no longer exists. + +1. **Describe the gate.** If your repository is a single stack with standard scripts, you need no file. Otherwise write `.nself/ci.yaml` with the jobs you want. Declaring jobs turns detection off for the repository, so a gate is exactly the list you declare. A repository with many modules should say so: detection would otherwise plan a job per module. +2. **Run it locally first.** `nself ci plan` shows which checks will run, and `NSELF_V15=1 nself ci` runs them. Read the plan. Do not assume. +3. **Add the workflow.** Copy `free/ci/templates/nself-ci.yml` from the plugins repository to `.github/workflows/nself-ci.yml`, fill in the pinned release, and add your toolchain setup. The template is described under [Configuration and policy](./Configuration-and-Policy.md#the-workflow-template). +4. **Require it, as an addition.** Run `nself ci policy protect --repo / --dry-run`, review the body, then apply it. It saves the previous protection first and can restore it. Add `nself-ci` next to your existing required checks, not instead of them, until you have seen it catch what they catch. +5. **Guard the main branch.** `nself ci policy protect --hook` installs a pre-push hook for the default branch, and `--audit ` lists commits that never had a green `nself-ci`. + +### Read this before making it your only gate + +A gate can pass while checking nothing. That is the failure to look for, and it is why step 2 matters. + +- **A stack the detector does not see.** The earlier gate decided whether a Node repository had work by looking for scripts in the root `package.json`. A repository whose real scripts sat under another name, or in workspace members without a workspace file, ran only the secret scan and printed a pass. The new planner walks directories and workspaces and gives every skipped job a reason, but a custom script name still needs a job of your own. `nself ci plan` is the place to see it. +- **Per-package checks.** Repository-wide checks do not replace checks that belong to a package or a plugin, such as schema validation, conformance or registry consistency. Keep those workflows. +- **Work the engine does not do.** `nself ci` is not an emulator for hosted workflow files. End-to-end tests, accessibility audits, code scanning, licence audits, coverage ratchets, bundle size checks and deploys need their own jobs. Public repositories can keep them on free hosted runners. +- **Forks.** A token on a fork pull request is read-only, so a commit status cannot be posted from it. The workflow template produces a check run instead, which fork and bot pull requests still get. + +### Rollout checklist + +- [ ] `.nself/ci.yaml` written, or the detected plan reviewed and accepted. +- [ ] `NSELF_V15=1 nself ci plan` read, and each stack you care about appears. +- [ ] `nself ci` passes locally with `gitleaks` installed. +- [ ] The workflow runs once on a pull request and the `nself-ci` check appears. +- [ ] `policy protect --dry-run` reviewed by the person who owns branch protection, then applied as an additional required check. +- [ ] The earlier workflow or wrapper script removed, and no page of your documentation links to it. diff --git a/.github/wiki/guides/ci/Troubleshooting.md b/.github/wiki/guides/ci/Troubleshooting.md new file mode 100644 index 000000000..32a5c930a --- /dev/null +++ b/.github/wiki/guides/ci/Troubleshooting.md @@ -0,0 +1,136 @@ +# CI troubleshooting + + + +Start here when a run is red, slow, stuck or surprising. The order that finds most problems is: ask `doctor` about the machine, ask `explain` about the plan, read the log, then reproduce the job. Codes such as `E614` are listed with their summaries and fixes in the generated [[CI reference|guides/ci/Reference]]; this page tells you which one to look for and what to do next. + +## Ask the machine: doctor + +```text +NSELF_V15=1 nself ci doctor [--json] [--repo dir] +``` + +`doctor` runs a set of read-only checks and prints one line per finding with an id, a verdict (`PASS`, `WARN` or `FAIL`), a reason and, when it knows one, a fix. It covers the toolchain for each detected stack, whether the container runtime is reachable, whether status posting is possible, whether the local store opens and is the right version, whether `.nself/ci.yaml` validates, a leftover legacy `.ci.yaml`, a git-ignored config, and drift between the tools you have and the pinned toolchain image. Nodes, providers and isolation add their own sections when they are in use. + +- Exit 0 means clean, 10 means at least one failure, 12 means warnings only, 1 is a usage error and 2 is a bad `--repo`. +- A section that fails to run, panics or takes too long is reported as its own finding. One broken section never hides the rest. +- `doctor` never creates the local store. A missing store passes with "created on first run". +- The table of ids is generated: see the doctor section of the [[CI reference|guides/ci/Reference]]. A warning about drift in a tool version is information: run with `--container` for the pinned version, or install it. +- Known limits: the toolchain requirement is matched on the first word of a job's command, so a job run through `make` or `sh -c` does not mark a toolchain as required; a job that needs a service does not by itself mark Docker as needed. + +## Ask the plan: explain + +If a job did not run, or ran where you did not expect, do not guess: + +```text +nself ci plan [--pipeline name] [--trigger t] [--base sha] +nself ci explain +``` + +`plan` lists every job with its state and a reason code. `explain` shows the requirements of a job, the runner it chose, the score components that led to that choice and the alternatives. Both only read; neither starts anything. + +| You see | It usually means | +|---|---| +| `selection.no_base` | nothing said what changed, so everything runs; pass `--base` or run from a branch with a known base | +| `pipeline.excluded` | the job is not in the pipeline you named | +| `when.paths`, `when.branch`, `when.trigger` | a condition in `when` was not met | +| `when.env_missing` | the variable named in `when.env_present` is not set | +| `dep.failed` | a job it depends on failed, so it was skipped | +| `unplaceable.*` | no runner can take it: wrong platform, a missing tool, no Docker or a missing label | +| `admission.reservation`, `admission.battery` | the machine is protecting itself; see [[Getting started with nself ci|guides/ci/Getting-Started]] | +| `advisory.tool_missing`, `advisory.data_offline` | an advisory check skipped because its tool or data was unavailable | + +## Read the log + +```text +nself ci logs [--run ] [--attempt N] [-f] [--json] +``` + +Without `--attempt` it shows the highest attempt of the job. After `--confirm-flaky` that is the passing rerun, not the failing attempt the evidence points to, so pass `--attempt` when you want the original. With `--json` you get NDJSON log records and a final result record. No stored log for that job or attempt is `E633`, and a stored log that cannot be read is `E632`; check that the state directory is writable and not full. A log can be missing when the disk was full or the folder unwritable; the job is unaffected, and `ci logs` is where you see it. + +Logs hold redacted output, so a value that matches a known secret shape shows as a mask. Lines carry a sequence number; use it for order, since lines written in one burst can share a timestamp. + +## Failure or infrastructure + +A failed check carries a **failure class**, and the class decides what happens next. + +| Class | Meaning | Retried automatically | +|---|---|---| +| `code` | your code or tests failed | no | +| `infra` | the machine, the network or the tool installation failed | yes, up to the budget | +| `timeout` | the job ran out of time | no | +| `cancelled` | someone stopped it | no | +| `lost` | the runner went away | by recovery, not by retry | + +The classifier looks for infrastructure evidence in a fixed order: the executor died without an exit code, a container hit a resource limit, the executor reported its own error, or the last lines of output match a known infrastructure signature, and only when no test-failure marker such as `FAIL`, `not ok`, `assertion` or `panic` is in the tail. Anything else is `code`. **A plain exit 137 or 127 is a code failure**: a program chose it. An unknown job failure is `code`. An error on the executor side that cannot be proven is `infra`, but it is never retried. + +The evidence for a failed check names the class, the exit code, a `diagnosis` that matches a signature, a `remediation`, a redacted excerpt and where the log is stored. + +**Retries.** An infrastructure failure is retried up to `retry.infra_max`, bounded by the plan's own limit. Release jobs and jobs marked `idempotent: false` run once and wait for an operator instead. Each retry records why the previous attempt failed. + +**Install failures.** For Node, a failed frozen install prints a bounded report and is classified from the manager's own error lines: an out-of-sync lockfile, a failing lifecycle script or a test failure is `code`; a registry error, a full disk or a killed child is `infra`. A package manager that does not match the declared one is `E811` and is not retried. A tool that is not installed at all is `E614`. + +### Flaky checks + +A check that fails once and passes on identical inputs is flaky, and you can measure it without changing a result: + +```text +nself ci --confirm-flaky 3 +nself ci runs --flaky [--limit N] [--json] +``` + +`--confirm-flaky N` reruns a job that failed with a code failure up to N more times, stopping at the first pass. N is at most ten. Each rerun is authorised by an audit row naming the failed attempt and is recorded with the diagnosis `confirm-flaky`. The original result stays **failed**: confirmation tells you the check is flaky, it never turns red into green. `runs --flaky` lists checks that both passed and failed on identical inputs (same repository, revision, tracked digest, pipeline, policy, source trust and input digests) among recent runs. Dirty, unbound and ad hoc runs are ignored. The finished run's evidence carries the same marker in the `flaky` field. + +## Reproduce a job + +```text +nself ci runs reproduce [--run ] [--check] [--shell] [--json] [--repo dir] +``` + +Reproduce takes the newest stored run that holds the job, rebuilds its inputs from the evidence (revision, trigger, pipeline, isolation), and prints a **drift report** before doing anything. The drift kinds are a tool whose version changed, a tool that is missing, and a platform that differs. + +- A platform that differs is refused (`E634`) before anything runs, and the message names both platforms. +- If the original evidence was unbound, only the recorded revision is reproduced and you get a warning (`E635`): uncommitted changes are never reproduced. +- `--check` stops after the report. +- `--shell` prepares the worktree and runs nothing, and keeps it for you. Remove it with `git worktree remove` when you finish. +- Otherwise it makes a temporary worktree at the recorded revision, runs the one job through the normal engine, and appends a link line to the state directory under `reproductions/`. The line records both outcomes and `reproduced: true` when the result and the failure class match. The temporary worktree is removed on error, panic and cancel, and a cancelled reproduction writes no link. + +Services appear in the report only after the run, not before. + +```bash +mkdir broken && cd broken && git init -q +git config user.email docs@example.com && git config user.name docs +mkdir .nself && cat > .nself/ci.yaml <<'CFG' +version: 1 +jobs: + build: + kind: test + run: [sh, -c, "echo building; echo boom >&2; exit 3"] +CFG +git add -A && git commit -q -m init +rc=0 +nself ci --no-status || rc=$? +[ "$rc" -ne 0 ] +nself ci logs build +nself ci runs list +nself ci runs reproduce build --check +``` + +## Common problems + +| Symptom | Look at | +|---|---| +| `E614` before any job runs | a required tool is not on `PATH`; the message names it. Install it or run with `--container`. | +| `E620` on a config you did not touch | the file is validated strictly; the message names the line and key. See [[CI configuration and policy|guides/ci/Configuration-and-Policy]]. | +| `E812` from the secret scan | a `.gitleaksignore` entry has no review comment; add one with `nself ci config ignore add`. | +| A required job is `unplaceable.platform` | it needs a platform you have no node for; see [[Nodes and Runners|guides/ci/Nodes-and-Runners]] | +| `E637` from a service job | no container runtime is reachable; start Docker, Colima or the service | +| `E646` from an ad hoc command | the default is a container and none is available; use `--isolation process` knowingly | +| The run is red but every check passed | a required job ended in error, or a required job was skipped; read `nself ci plan` and the evidence `result` | +| `E606` opening the store | the store was written by a newer binary; upgrade the plugin | +| A run stays `running` after a crash | start any run; recovery closes runs with no live coordinator, or run `nself ci runs wait ` | +| `nself ci runs wait` exits 10 | the run is still going; call it again | +| A detached run produced `E648` | the run could not be queued; the message ends with a redacted tail of what it printed | +| Disk use grows | `nself ci runs prune --dry-run`, then without `--dry-run`; see retention in [[Getting started with nself ci|guides/ci/Getting-Started]] | + +If you are stuck on a node rather than on a job, the lost-lease, drain and enrollment problems are in [[Nodes and Runners|guides/ci/Nodes-and-Runners]]. For trust and secrets questions, see [[CI trust, security and privacy|guides/ci/Trust-Security-and-Privacy]]. diff --git a/.github/workflows/doc-commands.yml b/.github/workflows/doc-commands.yml new file mode 100644 index 000000000..7528a4906 --- /dev/null +++ b/.github/workflows/doc-commands.yml @@ -0,0 +1,80 @@ +# CI wiki doc checks (P7-CI-48): every command block of the pages in guides/ci runs, and the generated parts agree with the code. +# +# Purpose: keep the CI wiki pages honest. Step one runs every fenced bash block of every page of +# .github/wiki/guides/ci/ in a clean fixture project (.github/scripts/check-doc-commands.sh), so a page cannot +# document a command, flag or output that the binary built from this very checkout does not accept. That covers the +# pages of the other CI tickets as well as the core pages. Step two runs the reference generator in check mode, so +# Reference.md, the evidence tables and the sidebar block cannot drift from the code. +# Inputs: this checkout. `nself ci ...` in a block is answered by the plugin built from this checkout, through a +# one-line shim named `nself` (the real CLI hands `ci ...` to the plugin binary unchanged). No secrets, and no +# network beyond the runner's own checkout and toolchain download. +# Outputs: pass or fail; a failure names the page, the block number and the exit code. +# Constraints: actions pinned by SHA; `permissions: contents: read`; the workflow interpolates no event payload field, +# no head ref name and no secret, and never runs on the privileged fork-PR trigger. Blocks run with a scrubbed +# environment (see free/ci/tools/doccmds), so nothing from the runner's environment reaches them. + +name: doc-commands + +on: + pull_request: + paths: + - ".github/wiki/guides/ci/**" + - ".github/wiki/_Sidebar.md" + - ".github/scripts/check-doc-commands.sh" + - ".github/workflows/doc-commands.yml" + - "free/ci/**" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: doc-commands-${{ github.ref }} + cancel-in-progress: true + +jobs: + doc-commands: + name: doc-commands + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - name: Check out the repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false + + - name: Set up Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7 + with: + go-version-file: free/ci/go.mod + cache: false + + - name: Build the plugin from this checkout and put `nself` on the path + run: | + set -euo pipefail + bindir="$RUNNER_TEMP/doc-commands-bin" + mkdir -p "$bindir" + CGO_ENABLED=0 go -C free/ci build -mod=vendor -o "$bindir/nself-ci" ./cmd + cat > "$bindir/nself" <<'SHIM' + #!/bin/sh + # nself ci ARGS... -> nself-ci ARGS... (the plugin binary built from this checkout) + if [ "${1:-}" != ci ]; then + echo "nself shim: only 'nself ci' is available in the doc check" >&2 + exit 127 + fi + shift + exec "$(dirname "$0")/nself-ci" "$@" + SHIM + chmod +x "$bindir/nself" + echo "$bindir" >> "$GITHUB_PATH" + + - name: Run every command block of every guides/ci page + run: | + set -euo pipefail + bash .github/scripts/check-doc-commands.sh .github/wiki/guides/ci/*.md + + - name: The generated parts agree with the code + working-directory: free/ci + run: | + set -euo pipefail + CGO_ENABLED=0 go run -mod=vendor ./tools/refgen -root ../.. -check