From d2ea1efeecd68601e5b5cf44325d47caa754c720 Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 9 Nov 2026 20:46:14 +0800 Subject: [PATCH 1/2] proposals: offline and air-gapped use How Comline works with no internet access: on an offline machine, in an isolated build network, in an air-gapped deployment, and when bringing the tooling itself in. Fetching (on demand with a hard offline switch, package settings that only tighten, when a hash is mandatory) and supply (several sources per dependency, a shared cache, vendoring, mirrors, bundles, release artifacts, the runtimes published for 1.0). --- docs/docs/proposals/index.md | 8 ++ docs/docs/proposals/offline/fetching.md | 105 ++++++++++++++++++++ docs/docs/proposals/offline/index.md | 77 +++++++++++++++ docs/docs/proposals/offline/supply.md | 125 ++++++++++++++++++++++++ docs/mkdocs.yml | 4 + 5 files changed, 319 insertions(+) create mode 100644 docs/docs/proposals/offline/fetching.md create mode 100644 docs/docs/proposals/offline/index.md create mode 100644 docs/docs/proposals/offline/supply.md diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index 20f228c..4b00b09 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -218,6 +218,14 @@ the language side is shipped and documented in the ### Tooling +- [Offline and air-gapped use](offline/index.md): how Comline works with no + internet access, on an offline machine, in an isolated build network, in an + air-gapped deployment, and when bringing the tooling itself in. Two parts: + [fetching](offline/fetching.md) (on demand with a hard offline switch, + package settings that only tighten, when a hash is mandatory) and + [supplying dependencies and tooling](offline/supply.md) (several sources per + dependency, a shared cache, vendoring, mirrors, bundles, release artifacts). + *draft.* - [Project dashboard](project-dashboard.md): a separate static site where the benchmark, fuzzing, conformance, size, coverage and CI results of every repository can be watched over time, and any published versions compared; diff --git a/docs/docs/proposals/offline/fetching.md b/docs/docs/proposals/offline/fetching.md new file mode 100644 index 0000000..e5f616c --- /dev/null +++ b/docs/docs/proposals/offline/fetching.md @@ -0,0 +1,105 @@ +# Fetching + +Status: **draft** · part of [Offline and air-gapped use](index.md) + +**The question.** When may a Comline tool use the network, who decides, and +what must a fetched copy prove before it's used? + +## Choosing and getting + +Two different acts, which today are mixed: + +- **Choosing** what a package depends on: `comline add`, and a future + `comline update`. It writes pins (a commit, a version, a hash) into + `config.idp`, and is the only act that may look for something new. It uses + the network only because the user asked. +- **Getting** the bytes a pin already names. The pin decides what they are; + the network only decides whether they're here yet. + +Everything below is about getting. + +## The default: on demand, with a hard switch + +A build (`check`, `build`, `generate`, ...) fetches any pinned content it +doesn't have yet, as today. + +- **`comline fetch`** fetches ahead of time exactly what a build would fetch, + and nothing else: the step to run before a machine goes offline, or in a CI + stage that's allowed out. +- **The offline switch** forbids all network access: `--offline` on any + command, an environment variable, or machine configuration. A build that + is missing something then fails with one error listing *everything* + missing, not the first thing it tripped on, and saying how to supply it + (`comline fetch` elsewhere, a vendored copy, a bundle). +- **The language server never fetches**, offline or not. It shows a missing + dependency the way `check` would describe it. + +## Package settings + +A package can tighten this in `config.idp`, for when its situation needs it: +a package built for a sealed customer, say, should behave strictly on every +developer's machine and in every CI, not only on the machines set up for it. + +- **Explicit fetching:** a build never fetches; only `comline fetch` does. + Builds stay offline by construction, at the cost of one step after a clone + or a pin change. +- **Required hashes:** every non-path dependency must declare its hash (see + below). + +## Precedence: the strictest wins + +The environment (the switch, machine and organisation configuration) and the +**root** package's settings combine, and the strictest applies: + +- **Offline is absolute.** No package setting can allow the network back. +- **A package only tightens.** Its settings can make a build stricter than the + environment, never looser. +- **A dependency's settings apply to its own builds only.** Otherwise a + dependency could make every consumer's build stricter, or looser, without + them knowing. + +| Environment | Root package | A build may fetch | Hash required (git) | +|---|---|---|---| +| online | default | yes, pinned content only | no (warning) | +| online | explicit fetching | no; `comline fetch` may | no (warning) | +| online | required hashes | yes | yes | +| offline | anything | no | yes | + +## Hashes + +A dependency's `hash` is the blake3 of its compiled content, and core already +checks a declared one. When it's **mandatory**: + +- **A registry dependency: always.** A version string can be republished with + other content, so without a hash the pin names nothing. +- **A git dependency: when offline, or when the root package requires hashes.** + Otherwise the pinned commit already identifies the content, and today's + warning stays. +- **Any dependency with more than one source** ([supply](supply.md#several-sources-per-dependency)): + the sources don't share any other identity. + +Offline is when the hash matters most: the bytes then come from a vendored +copy, a cache or a carried bundle, which have passed through other hands, and +the hash is what proves a copy is the pinned content. + +A copy whose hash doesn't match is **an error, never a fallback**, naming +where it came from: it may be compromised. + +## Options considered + +| | On demand + switch (proposed) | Explicit fetch only | +|---|---|---| +| Day to day | as today | one more step after a clone or a pin change | +| A sealed environment | safe once the switch is set | safe by default | +| Strictness for one package | the package's "explicit fetching" setting | always | + +On-demand with a switch keeps everyday use as it is; a mandatory hash where +it matters means getting can't change what is built; and a package that +needs the stricter behaviour asks for it, so it holds wherever the package is +built. + +## Open questions + +- The setting names, and where machine and organisation configuration live. +- Whether `comline fetch` also takes `--for ` to fetch only what one + target's generation needs. diff --git a/docs/docs/proposals/offline/index.md b/docs/docs/proposals/offline/index.md new file mode 100644 index 0000000..2e87ed6 --- /dev/null +++ b/docs/docs/proposals/offline/index.md @@ -0,0 +1,77 @@ +# Offline and air-gapped use + +Status: **draft** · two parts, read in order: +[fetching](fetching.md), then [supplying dependencies and tooling](supply.md) + +**The question.** Nothing yet says how Comline works without internet +access. **What must hold, in every repository, so Comline can be used on a +machine or in a network that can't reach the internet, and so a build there +is exactly the build anywhere else?** + +## The situations + +All four are in scope: + +| Situation | What it needs | +|---|---| +| **An offline developer machine** | Work from what was fetched earlier, or from a bundle, and be told clearly what's missing. | +| **An isolated build or CI network** | Build and generate from vendored copies or internal mirrors, never reaching outside. | +| **An air-gapped deployment** | Services built with Comline run in a sealed network: the runtime and generated code need nothing beyond the peer they talk to. | +| **The tooling into the gap** | The CLI, the language server, generators and docs brought into a sealed network, verifiably. | + +## What it looks like today + +| Part | Network today | Already offline-friendly | +|---|---|---| +| **core** (dependencies) | A `Git` dependency is fetched by the system `git` into a per-package `.comline/deps-cache/`. `Registry` sources exist in the config (with `hash` and `signature`) but nothing resolves them. | Pins are exact commits, and a checkout is reused. A mode that never fetches exists, which the language server uses. A declared `hash` (blake3 of the dependency's compiled content) is checked; without one, core warns and trusts the fetch. | +| **cli** | `check`, `add` and `build` fetch through core. It installs only with `cargo install --path .`: no release binaries, checksums or signatures. | Generators are built in, or local executables and `.wasm` files: nothing is downloaded. | +| **language server** | None. | It never fetches, and says "isn't fetched yet — run `comline check`". | +| **generated code** | Rust depends on `comline-runtime` by a git rev; TypeScript on `@comline/runtime` from npm. | Otherwise self-contained. | +| **runtime** | None beyond the peer it's given. | Fits air-gapped deployment as it is. | +| **specification** | The vendoring scripts clone from GitHub, only when a developer runs them. | Copies are committed, pinned and checked. | +| **docs** | The theme loads Google Fonts by default. | — | + +Nothing reports usage or checks for updates. + +## Rules that hold whatever the details + +- **Choosing is separate from getting.** Choosing a dependency (`add`, a + future `update`) writes pins, and uses the network only when the user asks. + Getting bytes that are already pinned follows [the fetching + rules](fetching.md). +- **Identity is the hash, location is replaceable.** Where bytes come from + (a registry, git, a mirror, a vendored copy, a cache, a bundle) never + changes what is built. A copy is accepted only when it matches the pin. +- **Offline is absolute.** When the environment says offline, nothing touches + the network, whatever any package says. +- **Nothing calls home.** The runtime and generated code never reach anything + but the peer they're given. No tool sends telemetry or checks for updates. + The language server never fetches. + +## Rollout + +In stages, each useful on its own: + +1. **Fetching rules in core and the CLI:** the offline switch, `comline + fetch`, the package settings and their precedence, and the hash rules. +2. **The shared cache and vendoring:** one hash-keyed cache per machine, in + place of the per-package deps cache, and `comline vendor`. +3. **Several sources per dependency, and mirrors:** ordered sources, machine + and organisation mirrors, and the root package's additions. +4. **Bundles:** export and import of a signed archive. +5. **Tooling into the gap:** release artifacts with checksums, signatures and + an SBOM, reproducible builds, and an offline docs bundle. +6. **Generated code's dependencies:** the runtimes published to crates.io and + npm, a hard requirement for 1.0. Until then, the docs say how to vendor + them. + +## Open questions + +Collected from both parts: + +- The names of the settings, and where machine and organisation config live. +- The cache's location on each OS. +- The mirror configuration format. +- The bundle format and its signing keys. +- The release signing scheme (sigstore, minisign, ...). +- The SBOM format (SPDX or CycloneDX). diff --git a/docs/docs/proposals/offline/supply.md b/docs/docs/proposals/offline/supply.md new file mode 100644 index 0000000..e0a95e5 --- /dev/null +++ b/docs/docs/proposals/offline/supply.md @@ -0,0 +1,125 @@ +# Supplying dependencies and tooling + +Status: **draft** · part of [Offline and air-gapped use](index.md) + +**The question.** How do dependencies, and Comline itself, get onto a machine +or into a network that can't fetch them, without that ever changing what is +built? + +## Identity is the hash, location is replaceable + +A dependency is identified by its pin: a version or a commit, and its content +hash. *Where* the bytes come from is a separate, replaceable thing. Every +route below is just another place to find the same hash-checked content, so +none can change a build: at worst one can fail to supply it. + +## Several sources per dependency + +A dependency may list several sources, in order, so it stays available +whatever happens to one of them: + +```text +billing = { + version = "1.4.0" + hash = "blake3:…" + sources = [ + { registry = "comline" } + { registry = "https://registry.acme.internal" } + { git = "https://github.com/acme/billing", commit = "…" } + ] +} +``` + +- **More than one source requires a hash.** A registry download and a git + checkout of the same package give the same content hash, and nothing else + they carry (a version string, a commit) says anything about the other. +- **An unreachable source, or one without the content, falls through** to the + next. +- **A source serving the wrong content is a hard error** naming it. Moving on + would still build the right bytes, but would hide what may be a + compromised source. + +### Who can add sources + +- **The package**, in its `config.idp`. +- **Machine or organisation configuration:** mirrors, as URL prefix rewrites + (`https://github.com/` → `https://git.acme.internal/`) or sources put ahead + of a package's own, without editing any package. Safe because the hash + decides. +- **The root package, for a transitive dependency,** when that dependency's + pin carries a hash: if C's only source is unreachable from your network and + the pin is in B's config, the root adds a source for C. It **adds, never + replaces**: content other than what B pinned would change what is built, + and possibly C's given-up guarantees. Replacing (a patch) is out of scope. + +### Order + +Machine and organisation mirrors first, then the root's additions, then the +package's own sources. Mirrors first keeps lookups inside the organisation: +each source tried learns the dependency's name. Offline, only local sources +count: the vendored copy, the cache, bundles. + +The build reports which source served each dependency (verbose output and +`--json`). That's never frozen: location isn't identity. + +## The shared cache + +One content-addressed cache per machine, keyed by hash, in place of today's +per-package `.comline/deps-cache/`. Every route below feeds it, and a build +reads from it. Projects on the same machine share what's been fetched, and a +copy is checked against its hash on the way in. + +## Vendoring + +`comline vendor` writes every dependency a package needs, transitive ones +included, into a directory in the project, each with its pin and hash. It +travels with the repository through any channel that carries it, and a +change to a dependency shows up as a diff in review. A build uses a vendored +copy only when it matches its pin. + +## Mirrors + +An internal git server or registry mirror, set once per machine or +organisation, as above. Suited to organisations with many projects; a future +registry needs the same mechanism. + +## Bundles + +A single signed archive of dependencies, carried across the gap and imported +into the cache on the other side: `comline bundle export` on a connected +machine, `comline bundle import` on the sealed one. Each entry is checked +against its hash on import, and the archive's signature against a key the +sealed side trusts. The same format can carry the tooling. + +## Generated code's dependencies + +Generated Rust depends on `comline-runtime` by a git rev, and generated +TypeScript on `@comline/runtime` from npm. Each ecosystem has its own offline +tools (`cargo vendor`, registry mirrors, npm caches), which work best with +published packages. + +- **Publishing the runtimes to crates.io and npm is a hard requirement for + 1.0.** Generated code then names a version, not a git rev. +- **Until then**, the docs say how to vendor them for a sealed build: + `cargo vendor` handles git dependencies, and `npm pack` produces a tarball + to install from. + +## Tooling into the gap + +- **Release artifacts:** the CLI and the language server prebuilt for each + platform, with checksums and signatures, rather than `cargo install` from + source only. +- **Reproducible builds:** anyone can rebuild an artifact from source and + compare bytes, as comline-typescript already does for its wasm core. +- **An SBOM per release**, for sealed environments' audits. +- **An offline docs bundle:** the documentation as a self-contained archive, + with no outside assets (no Google Fonts), and generated schema docs + (`doc-html`) likewise. + +## Open questions + +- The cache's location on each OS, and how it's cleaned. +- The mirror configuration format. +- The bundle format, and how its signing keys are distributed and trusted. +- The release signing scheme (sigstore, minisign, ...). +- The SBOM format (SPDX or CycloneDX). diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index f97f52d..bc4b76f 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -225,6 +225,10 @@ nav: - 'Proposal B': 'proposals/guarantee-marker/assumes.md' - 'Proposal C': 'proposals/guarantee-marker/unverified.md' - 'Proposal D': 'proposals/guarantee-marker/unchecked.md' + - 'Offline and air-gapped use': + - 'proposals/offline/index.md' + - 'Fetching': 'proposals/offline/fetching.md' + - 'Supplying dependencies and tooling': 'proposals/offline/supply.md' extra_css: - stylesheets/nav.css?v=23 # bump on every nav.css change — it is not cache-busted otherwise From b84e4f83099bc5f346970d334c3ac3ad13561707 Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 9 Nov 2026 22:01:37 +0800 Subject: [PATCH 2/2] proposals/offline: settle where settings live, the cache, mirrors, bundles, signing and the SBOM - Fetch policy lives in the project's comline.toml (committed, never frozen, only the root's read); offline, mirrors, the cache and trusted keys in layered machine and organisation config. - A dependency's sources are never frozen. - The cache uses the platform cache dir, can sit over a read-only organisation layer, and is never evicted automatically. - A mirror is the only route unless fallback = true. - Bundles are deterministic .tar.zst archives, signed with minisign keys; unsigned ones are refused without --allow-unsigned. - Releases are signed with minisign over checksums, plus GitHub artifact attestations; the SBOM is CycloneDX JSON. --- docs/docs/proposals/offline/fetching.md | 21 +++++--- docs/docs/proposals/offline/index.md | 21 +++++--- docs/docs/proposals/offline/supply.md | 72 +++++++++++++++++++------ 3 files changed, 83 insertions(+), 31 deletions(-) diff --git a/docs/docs/proposals/offline/fetching.md b/docs/docs/proposals/offline/fetching.md index e5f616c..0450304 100644 --- a/docs/docs/proposals/offline/fetching.md +++ b/docs/docs/proposals/offline/fetching.md @@ -27,7 +27,8 @@ doesn't have yet, as today. and nothing else: the step to run before a machine goes offline, or in a CI stage that's allowed out. - **The offline switch** forbids all network access: `--offline` on any - command, an environment variable, or machine configuration. A build that + command, `COMLINE_OFFLINE=1`, or `offline = true` in machine or + organisation configuration. A build that is missing something then fails with one error listing *everything* missing, not the first thing it tripped on, and saying how to supply it (`comline fetch` elsewhere, a vendored copy, a bundle). @@ -36,9 +37,18 @@ doesn't have yet, as today. ## Package settings -A package can tighten this in `config.idp`, for when its situation needs it: -a package built for a sealed customer, say, should behave strictly on every -developer's machine and in every CI, not only on the machines set up for it. +A package can tighten this, for when its situation needs it: a package built +for a sealed customer, say, should behave strictly on every developer's +machine and in every CI, not only on the machines set up for it. The settings +go in the project's `comline.toml`: it's committed, so they travel with the +package; it's never frozen, so changing them is no new version; and only the +root package's is ever read, which is the precedence rule below. + +```toml +[fetch] +mode = "explicit" # "on-demand" (the default) or "explicit" +require-hashes = true +``` - **Explicit fetching:** a build never fetches; only `comline fetch` does. Builds stay offline by construction, at the cost of one step after a clone @@ -100,6 +110,5 @@ built. ## Open questions -- The setting names, and where machine and organisation configuration live. -- Whether `comline fetch` also takes `--for ` to fetch only what one +- Whether `comline fetch` takes `--for `, to fetch only what one target's generation needs. diff --git a/docs/docs/proposals/offline/index.md b/docs/docs/proposals/offline/index.md index 2e87ed6..cf62e83 100644 --- a/docs/docs/proposals/offline/index.md +++ b/docs/docs/proposals/offline/index.md @@ -65,13 +65,18 @@ In stages, each useful on its own: npm, a hard requirement for 1.0. Until then, the docs say how to vendor them. -## Open questions +## Where settings live -Collected from both parts: +| Where | What | Frozen | +|---|---|---| +| `config.idp` | each dependency's identity (version or commit, and hash) and its `sources` | identity only; `sources` is never frozen | +| the project's `comline.toml` (committed, read only for the root package) | the fetch policy, and sources the root adds for transitive dependencies | never | +| machine and organisation config: a system file (`/etc/comline/config.toml`, `%PROGRAMDATA%`), a user file (`~/.config/comline/config.toml` and the platform's equivalents), or one named by `COMLINE_CONFIG` | `offline`, mirrors, the cache's location, trusted keys | — | +| the environment and flags | `COMLINE_OFFLINE=1`, `--offline` | — | + +They combine, and the strictest applies. + +## Remaining open questions -- The names of the settings, and where machine and organisation config live. -- The cache's location on each OS. -- The mirror configuration format. -- The bundle format and its signing keys. -- The release signing scheme (sigstore, minisign, ...). -- The SBOM format (SPDX or CycloneDX). +- Whether `comline fetch` takes `--for `, to fetch only what one + target's generation needs. diff --git a/docs/docs/proposals/offline/supply.md b/docs/docs/proposals/offline/supply.md index e0a95e5..ffc053d 100644 --- a/docs/docs/proposals/offline/supply.md +++ b/docs/docs/proposals/offline/supply.md @@ -41,13 +41,15 @@ billing = { ### Who can add sources -- **The package**, in its `config.idp`. +- **The package**, in its `config.idp`. The list is **never frozen**: only the + identity (version or commit, and hash) is, so moving a repository or adding + a source is no new version. - **Machine or organisation configuration:** mirrors, as URL prefix rewrites (`https://github.com/` → `https://git.acme.internal/`) or sources put ahead of a package's own, without editing any package. Safe because the hash decides. -- **The root package, for a transitive dependency,** when that dependency's - pin carries a hash: if C's only source is unreachable from your network and +- **The root package, for a transitive dependency,** in its `comline.toml`, + when that dependency's pin carries a hash: if C's only source is unreachable from your network and the pin is in B's config, the root adds a source for C. It **adds, never replaces**: content other than what B pinned would change what is built, and possibly C's given-up guarantees. Replacing (a patch) is out of scope. @@ -69,6 +71,17 @@ per-package `.comline/deps-cache/`. Every route below feeds it, and a build reads from it. Projects on the same machine share what's been fetched, and a copy is checked against its hash on the way in. +- **Where:** the platform's cache directory: `$XDG_CACHE_HOME/comline` + (`~/.cache/comline`) on Linux, `~/Library/Caches/comline` on macOS, + `%LOCALAPPDATA%\comline\cache` on Windows. `COMLINE_CACHE`, or `cache-dir` + in machine configuration, moves it. +- **A read-only layer underneath:** an organisation can provision a shared + cache (`/opt/comline/cache`, say) that every user reads and none writes, + which suits a sealed network. +- **Cleaning is never automatic:** evicting silently could break a machine + that has since gone offline. `comline cache list` shows what's there, and + `comline cache prune` cleans by age or size. + ## Vendoring `comline vendor` writes every dependency a package needs, transitive ones @@ -83,13 +96,40 @@ An internal git server or registry mirror, set once per machine or organisation, as above. Suited to organisations with many projects; a future registry needs the same mechanism. +```toml +[[mirror]] +from = "https://github.com/" +to = "https://git.acme.internal/github/" + +[[mirror]] +registry = "comline" +to = "https://registry.acme.internal/comline" +``` + +A mirror is **the only route** for what it covers: when it lacks something, +the original isn't tried, since every source tried learns what you depend on. +`fallback = true` on a mirror tries the original after it. + ## Bundles A single signed archive of dependencies, carried across the gap and imported into the cache on the other side: `comline bundle export` on a connected -machine, `comline bundle import` on the sealed one. Each entry is checked -against its hash on import, and the archive's signature against a key the -sealed side trusts. The same format can carry the tooling. +machine, `comline bundle import` on the sealed one. The same format can carry +the tooling. + +- **Format:** a `.tar.zst` holding a `manifest.json` (each entry's kind, + dependency or tool, its pin, hash and size), the content stored by hash, + and a detached signature. Entries are in a fixed order with fixed + timestamps, so exporting the same set twice gives the same bytes. +- **On import,** each entry is checked against its hash before it enters the + cache, and the signature against the keys trusted in machine or + organisation configuration (`trusted-keys`). +- **An unsigned bundle is refused** unless `--allow-unsigned` is given. The + hash already proves a pinned dependency; the signature proves who + assembled the bundle, which matters for tooling nothing pins yet and for + an organisation's rule on what may cross. +- **Keys** are minisign's (Ed25519), the same as releases', so one tool checks + both. ## Generated code's dependencies @@ -107,19 +147,17 @@ published packages. ## Tooling into the gap - **Release artifacts:** the CLI and the language server prebuilt for each - platform, with checksums and signatures, rather than `cargo install` from - source only. + platform, with a checksums file, rather than `cargo install` from source + only. +- **Signatures:** minisign over the checksums file is the primary signature, + since it checks with one public key and no network. GitHub artifact + attestations (sigstore) are added on top, for provenance when checking from + a connected machine. - **Reproducible builds:** anyone can rebuild an artifact from source and compare bytes, as comline-typescript already does for its wasm core. -- **An SBOM per release**, for sealed environments' audits. +- **An SBOM per release**, for sealed environments' audits, as **CycloneDX + JSON**: what vulnerability scanners read, with mature Rust + (`cargo-cyclonedx`) and npm tooling. SPDX can follow if someone needs it. - **An offline docs bundle:** the documentation as a self-contained archive, with no outside assets (no Google Fonts), and generated schema docs (`doc-html`) likewise. - -## Open questions - -- The cache's location on each OS, and how it's cleaned. -- The mirror configuration format. -- The bundle format, and how its signing keys are distributed and trusted. -- The release signing scheme (sigstore, minisign, ...). -- The SBOM format (SPDX or CycloneDX).