diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index bd771b8..d8661b9 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -296,6 +296,14 @@ the language side is shipped and documented in the ### Tooling +- [Offline and air-gapped use](offline/index.md): how Comline works with no + internet access, on an offline machine, in an isolated build network, in an + air-gapped deployment, and when bringing the tooling itself in. Two parts: + [fetching](offline/fetching.md) (on demand with a hard offline switch, + package settings that only tighten, when a hash is mandatory) and + [supplying dependencies and tooling](offline/supply.md) (several sources per + dependency, a shared cache, vendoring, mirrors, bundles, release artifacts). + *draft.* - [Project dashboard](project-dashboard.md): a separate static site where the benchmark, fuzzing, conformance, size, coverage and CI results of every repository can be watched over time, and any published versions compared; diff --git a/docs/docs/proposals/offline/fetching.md b/docs/docs/proposals/offline/fetching.md new file mode 100644 index 0000000..0450304 --- /dev/null +++ b/docs/docs/proposals/offline/fetching.md @@ -0,0 +1,114 @@ +# Fetching + +Status: **draft** · part of [Offline and air-gapped use](index.md) + +**The question.** When may a Comline tool use the network, who decides, and +what must a fetched copy prove before it's used? + +## Choosing and getting + +Two different acts, which today are mixed: + +- **Choosing** what a package depends on: `comline add`, and a future + `comline update`. It writes pins (a commit, a version, a hash) into + `config.idp`, and is the only act that may look for something new. It uses + the network only because the user asked. +- **Getting** the bytes a pin already names. The pin decides what they are; + the network only decides whether they're here yet. + +Everything below is about getting. + +## The default: on demand, with a hard switch + +A build (`check`, `build`, `generate`, ...) fetches any pinned content it +doesn't have yet, as today. + +- **`comline fetch`** fetches ahead of time exactly what a build would fetch, + and nothing else: the step to run before a machine goes offline, or in a CI + stage that's allowed out. +- **The offline switch** forbids all network access: `--offline` on any + command, `COMLINE_OFFLINE=1`, or `offline = true` in machine or + organisation configuration. A build that + is missing something then fails with one error listing *everything* + missing, not the first thing it tripped on, and saying how to supply it + (`comline fetch` elsewhere, a vendored copy, a bundle). +- **The language server never fetches**, offline or not. It shows a missing + dependency the way `check` would describe it. + +## Package settings + +A package can tighten this, for when its situation needs it: a package built +for a sealed customer, say, should behave strictly on every developer's +machine and in every CI, not only on the machines set up for it. The settings +go in the project's `comline.toml`: it's committed, so they travel with the +package; it's never frozen, so changing them is no new version; and only the +root package's is ever read, which is the precedence rule below. + +```toml +[fetch] +mode = "explicit" # "on-demand" (the default) or "explicit" +require-hashes = true +``` + +- **Explicit fetching:** a build never fetches; only `comline fetch` does. + Builds stay offline by construction, at the cost of one step after a clone + or a pin change. +- **Required hashes:** every non-path dependency must declare its hash (see + below). + +## Precedence: the strictest wins + +The environment (the switch, machine and organisation configuration) and the +**root** package's settings combine, and the strictest applies: + +- **Offline is absolute.** No package setting can allow the network back. +- **A package only tightens.** Its settings can make a build stricter than the + environment, never looser. +- **A dependency's settings apply to its own builds only.** Otherwise a + dependency could make every consumer's build stricter, or looser, without + them knowing. + +| Environment | Root package | A build may fetch | Hash required (git) | +|---|---|---|---| +| online | default | yes, pinned content only | no (warning) | +| online | explicit fetching | no; `comline fetch` may | no (warning) | +| online | required hashes | yes | yes | +| offline | anything | no | yes | + +## Hashes + +A dependency's `hash` is the blake3 of its compiled content, and core already +checks a declared one. When it's **mandatory**: + +- **A registry dependency: always.** A version string can be republished with + other content, so without a hash the pin names nothing. +- **A git dependency: when offline, or when the root package requires hashes.** + Otherwise the pinned commit already identifies the content, and today's + warning stays. +- **Any dependency with more than one source** ([supply](supply.md#several-sources-per-dependency)): + the sources don't share any other identity. + +Offline is when the hash matters most: the bytes then come from a vendored +copy, a cache or a carried bundle, which have passed through other hands, and +the hash is what proves a copy is the pinned content. + +A copy whose hash doesn't match is **an error, never a fallback**, naming +where it came from: it may be compromised. + +## Options considered + +| | On demand + switch (proposed) | Explicit fetch only | +|---|---|---| +| Day to day | as today | one more step after a clone or a pin change | +| A sealed environment | safe once the switch is set | safe by default | +| Strictness for one package | the package's "explicit fetching" setting | always | + +On-demand with a switch keeps everyday use as it is; a mandatory hash where +it matters means getting can't change what is built; and a package that +needs the stricter behaviour asks for it, so it holds wherever the package is +built. + +## Open questions + +- Whether `comline fetch` takes `--for `, to fetch only what one + target's generation needs. diff --git a/docs/docs/proposals/offline/index.md b/docs/docs/proposals/offline/index.md new file mode 100644 index 0000000..cf62e83 --- /dev/null +++ b/docs/docs/proposals/offline/index.md @@ -0,0 +1,82 @@ +# Offline and air-gapped use + +Status: **draft** · two parts, read in order: +[fetching](fetching.md), then [supplying dependencies and tooling](supply.md) + +**The question.** Nothing yet says how Comline works without internet +access. **What must hold, in every repository, so Comline can be used on a +machine or in a network that can't reach the internet, and so a build there +is exactly the build anywhere else?** + +## The situations + +All four are in scope: + +| Situation | What it needs | +|---|---| +| **An offline developer machine** | Work from what was fetched earlier, or from a bundle, and be told clearly what's missing. | +| **An isolated build or CI network** | Build and generate from vendored copies or internal mirrors, never reaching outside. | +| **An air-gapped deployment** | Services built with Comline run in a sealed network: the runtime and generated code need nothing beyond the peer they talk to. | +| **The tooling into the gap** | The CLI, the language server, generators and docs brought into a sealed network, verifiably. | + +## What it looks like today + +| Part | Network today | Already offline-friendly | +|---|---|---| +| **core** (dependencies) | A `Git` dependency is fetched by the system `git` into a per-package `.comline/deps-cache/`. `Registry` sources exist in the config (with `hash` and `signature`) but nothing resolves them. | Pins are exact commits, and a checkout is reused. A mode that never fetches exists, which the language server uses. A declared `hash` (blake3 of the dependency's compiled content) is checked; without one, core warns and trusts the fetch. | +| **cli** | `check`, `add` and `build` fetch through core. It installs only with `cargo install --path .`: no release binaries, checksums or signatures. | Generators are built in, or local executables and `.wasm` files: nothing is downloaded. | +| **language server** | None. | It never fetches, and says "isn't fetched yet — run `comline check`". | +| **generated code** | Rust depends on `comline-runtime` by a git rev; TypeScript on `@comline/runtime` from npm. | Otherwise self-contained. | +| **runtime** | None beyond the peer it's given. | Fits air-gapped deployment as it is. | +| **specification** | The vendoring scripts clone from GitHub, only when a developer runs them. | Copies are committed, pinned and checked. | +| **docs** | The theme loads Google Fonts by default. | — | + +Nothing reports usage or checks for updates. + +## Rules that hold whatever the details + +- **Choosing is separate from getting.** Choosing a dependency (`add`, a + future `update`) writes pins, and uses the network only when the user asks. + Getting bytes that are already pinned follows [the fetching + rules](fetching.md). +- **Identity is the hash, location is replaceable.** Where bytes come from + (a registry, git, a mirror, a vendored copy, a cache, a bundle) never + changes what is built. A copy is accepted only when it matches the pin. +- **Offline is absolute.** When the environment says offline, nothing touches + the network, whatever any package says. +- **Nothing calls home.** The runtime and generated code never reach anything + but the peer they're given. No tool sends telemetry or checks for updates. + The language server never fetches. + +## Rollout + +In stages, each useful on its own: + +1. **Fetching rules in core and the CLI:** the offline switch, `comline + fetch`, the package settings and their precedence, and the hash rules. +2. **The shared cache and vendoring:** one hash-keyed cache per machine, in + place of the per-package deps cache, and `comline vendor`. +3. **Several sources per dependency, and mirrors:** ordered sources, machine + and organisation mirrors, and the root package's additions. +4. **Bundles:** export and import of a signed archive. +5. **Tooling into the gap:** release artifacts with checksums, signatures and + an SBOM, reproducible builds, and an offline docs bundle. +6. **Generated code's dependencies:** the runtimes published to crates.io and + npm, a hard requirement for 1.0. Until then, the docs say how to vendor + them. + +## Where settings live + +| Where | What | Frozen | +|---|---|---| +| `config.idp` | each dependency's identity (version or commit, and hash) and its `sources` | identity only; `sources` is never frozen | +| the project's `comline.toml` (committed, read only for the root package) | the fetch policy, and sources the root adds for transitive dependencies | never | +| machine and organisation config: a system file (`/etc/comline/config.toml`, `%PROGRAMDATA%`), a user file (`~/.config/comline/config.toml` and the platform's equivalents), or one named by `COMLINE_CONFIG` | `offline`, mirrors, the cache's location, trusted keys | — | +| the environment and flags | `COMLINE_OFFLINE=1`, `--offline` | — | + +They combine, and the strictest applies. + +## Remaining open questions + +- Whether `comline fetch` takes `--for `, to fetch only what one + target's generation needs. diff --git a/docs/docs/proposals/offline/supply.md b/docs/docs/proposals/offline/supply.md new file mode 100644 index 0000000..ffc053d --- /dev/null +++ b/docs/docs/proposals/offline/supply.md @@ -0,0 +1,163 @@ +# Supplying dependencies and tooling + +Status: **draft** · part of [Offline and air-gapped use](index.md) + +**The question.** How do dependencies, and Comline itself, get onto a machine +or into a network that can't fetch them, without that ever changing what is +built? + +## Identity is the hash, location is replaceable + +A dependency is identified by its pin: a version or a commit, and its content +hash. *Where* the bytes come from is a separate, replaceable thing. Every +route below is just another place to find the same hash-checked content, so +none can change a build: at worst one can fail to supply it. + +## Several sources per dependency + +A dependency may list several sources, in order, so it stays available +whatever happens to one of them: + +```text +billing = { + version = "1.4.0" + hash = "blake3:…" + sources = [ + { registry = "comline" } + { registry = "https://registry.acme.internal" } + { git = "https://github.com/acme/billing", commit = "…" } + ] +} +``` + +- **More than one source requires a hash.** A registry download and a git + checkout of the same package give the same content hash, and nothing else + they carry (a version string, a commit) says anything about the other. +- **An unreachable source, or one without the content, falls through** to the + next. +- **A source serving the wrong content is a hard error** naming it. Moving on + would still build the right bytes, but would hide what may be a + compromised source. + +### Who can add sources + +- **The package**, in its `config.idp`. The list is **never frozen**: only the + identity (version or commit, and hash) is, so moving a repository or adding + a source is no new version. +- **Machine or organisation configuration:** mirrors, as URL prefix rewrites + (`https://github.com/` → `https://git.acme.internal/`) or sources put ahead + of a package's own, without editing any package. Safe because the hash + decides. +- **The root package, for a transitive dependency,** in its `comline.toml`, + when that dependency's pin carries a hash: if C's only source is unreachable from your network and + the pin is in B's config, the root adds a source for C. It **adds, never + replaces**: content other than what B pinned would change what is built, + and possibly C's given-up guarantees. Replacing (a patch) is out of scope. + +### Order + +Machine and organisation mirrors first, then the root's additions, then the +package's own sources. Mirrors first keeps lookups inside the organisation: +each source tried learns the dependency's name. Offline, only local sources +count: the vendored copy, the cache, bundles. + +The build reports which source served each dependency (verbose output and +`--json`). That's never frozen: location isn't identity. + +## The shared cache + +One content-addressed cache per machine, keyed by hash, in place of today's +per-package `.comline/deps-cache/`. Every route below feeds it, and a build +reads from it. Projects on the same machine share what's been fetched, and a +copy is checked against its hash on the way in. + +- **Where:** the platform's cache directory: `$XDG_CACHE_HOME/comline` + (`~/.cache/comline`) on Linux, `~/Library/Caches/comline` on macOS, + `%LOCALAPPDATA%\comline\cache` on Windows. `COMLINE_CACHE`, or `cache-dir` + in machine configuration, moves it. +- **A read-only layer underneath:** an organisation can provision a shared + cache (`/opt/comline/cache`, say) that every user reads and none writes, + which suits a sealed network. +- **Cleaning is never automatic:** evicting silently could break a machine + that has since gone offline. `comline cache list` shows what's there, and + `comline cache prune` cleans by age or size. + +## Vendoring + +`comline vendor` writes every dependency a package needs, transitive ones +included, into a directory in the project, each with its pin and hash. It +travels with the repository through any channel that carries it, and a +change to a dependency shows up as a diff in review. A build uses a vendored +copy only when it matches its pin. + +## Mirrors + +An internal git server or registry mirror, set once per machine or +organisation, as above. Suited to organisations with many projects; a future +registry needs the same mechanism. + +```toml +[[mirror]] +from = "https://github.com/" +to = "https://git.acme.internal/github/" + +[[mirror]] +registry = "comline" +to = "https://registry.acme.internal/comline" +``` + +A mirror is **the only route** for what it covers: when it lacks something, +the original isn't tried, since every source tried learns what you depend on. +`fallback = true` on a mirror tries the original after it. + +## Bundles + +A single signed archive of dependencies, carried across the gap and imported +into the cache on the other side: `comline bundle export` on a connected +machine, `comline bundle import` on the sealed one. The same format can carry +the tooling. + +- **Format:** a `.tar.zst` holding a `manifest.json` (each entry's kind, + dependency or tool, its pin, hash and size), the content stored by hash, + and a detached signature. Entries are in a fixed order with fixed + timestamps, so exporting the same set twice gives the same bytes. +- **On import,** each entry is checked against its hash before it enters the + cache, and the signature against the keys trusted in machine or + organisation configuration (`trusted-keys`). +- **An unsigned bundle is refused** unless `--allow-unsigned` is given. The + hash already proves a pinned dependency; the signature proves who + assembled the bundle, which matters for tooling nothing pins yet and for + an organisation's rule on what may cross. +- **Keys** are minisign's (Ed25519), the same as releases', so one tool checks + both. + +## Generated code's dependencies + +Generated Rust depends on `comline-runtime` by a git rev, and generated +TypeScript on `@comline/runtime` from npm. Each ecosystem has its own offline +tools (`cargo vendor`, registry mirrors, npm caches), which work best with +published packages. + +- **Publishing the runtimes to crates.io and npm is a hard requirement for + 1.0.** Generated code then names a version, not a git rev. +- **Until then**, the docs say how to vendor them for a sealed build: + `cargo vendor` handles git dependencies, and `npm pack` produces a tarball + to install from. + +## Tooling into the gap + +- **Release artifacts:** the CLI and the language server prebuilt for each + platform, with a checksums file, rather than `cargo install` from source + only. +- **Signatures:** minisign over the checksums file is the primary signature, + since it checks with one public key and no network. GitHub artifact + attestations (sigstore) are added on top, for provenance when checking from + a connected machine. +- **Reproducible builds:** anyone can rebuild an artifact from source and + compare bytes, as comline-typescript already does for its wasm core. +- **An SBOM per release**, for sealed environments' audits, as **CycloneDX + JSON**: what vulnerability scanners read, with mature Rust + (`cargo-cyclonedx`) and npm tooling. SPDX can follow if someone needs it. +- **An offline docs bundle:** the documentation as a self-contained archive, + with no outside assets (no Google Fonts), and generated schema docs + (`doc-html`) likewise. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index e878588..0df5e41 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -241,6 +241,10 @@ nav: - 'Proposal B': 'proposals/guarantee-marker/assumes.md' - 'Proposal C': 'proposals/guarantee-marker/unverified.md' - 'Proposal D': 'proposals/guarantee-marker/unchecked.md' + - 'Offline and air-gapped use': + - 'proposals/offline/index.md' + - 'Fetching': 'proposals/offline/fetching.md' + - 'Supplying dependencies and tooling': 'proposals/offline/supply.md' - 'A shared vocabulary': - 'proposals/shared-vocabulary/index.md' - 'The catalog': 'proposals/shared-vocabulary/catalog.md'