Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/docs/proposals/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,14 @@ the language side is shipped and documented in the

### Tooling

- [Offline and air-gapped use](offline/index.md): how Comline works with no
internet access, on an offline machine, in an isolated build network, in an
air-gapped deployment, and when bringing the tooling itself in. Two parts:
[fetching](offline/fetching.md) (on demand with a hard offline switch,
package settings that only tighten, when a hash is mandatory) and
[supplying dependencies and tooling](offline/supply.md) (several sources per
dependency, a shared cache, vendoring, mirrors, bundles, release artifacts).
*draft.*
- [Project dashboard](project-dashboard.md): a separate static site where the
benchmark, fuzzing, conformance, size, coverage and CI results of every
repository can be watched over time, and any published versions compared;
Expand Down
114 changes: 114 additions & 0 deletions docs/docs/proposals/offline/fetching.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Fetching

Status: **draft** · part of [Offline and air-gapped use](index.md)

**The question.** When may a Comline tool use the network, who decides, and
what must a fetched copy prove before it's used?

## Choosing and getting

Two different acts, which today are mixed:

- **Choosing** what a package depends on: `comline add`, and a future
`comline update`. It writes pins (a commit, a version, a hash) into
`config.idp`, and is the only act that may look for something new. It uses
the network only because the user asked.
- **Getting** the bytes a pin already names. The pin decides what they are;
the network only decides whether they're here yet.

Everything below is about getting.

## The default: on demand, with a hard switch

A build (`check`, `build`, `generate`, ...) fetches any pinned content it
doesn't have yet, as today.

- **`comline fetch`** fetches ahead of time exactly what a build would fetch,
and nothing else: the step to run before a machine goes offline, or in a CI
stage that's allowed out.
- **The offline switch** forbids all network access: `--offline` on any
command, `COMLINE_OFFLINE=1`, or `offline = true` in machine or
organisation configuration. A build that
is missing something then fails with one error listing *everything*
missing, not the first thing it tripped on, and saying how to supply it
(`comline fetch` elsewhere, a vendored copy, a bundle).
- **The language server never fetches**, offline or not. It shows a missing
dependency the way `check` would describe it.

## Package settings

A package can tighten this, for when its situation needs it: a package built
for a sealed customer, say, should behave strictly on every developer's
machine and in every CI, not only on the machines set up for it. The settings
go in the project's `comline.toml`: it's committed, so they travel with the
package; it's never frozen, so changing them is no new version; and only the
root package's is ever read, which is the precedence rule below.

```toml
[fetch]
mode = "explicit" # "on-demand" (the default) or "explicit"
require-hashes = true
```

- **Explicit fetching:** a build never fetches; only `comline fetch` does.
Builds stay offline by construction, at the cost of one step after a clone
or a pin change.
- **Required hashes:** every non-path dependency must declare its hash (see
below).

## Precedence: the strictest wins

The environment (the switch, machine and organisation configuration) and the
**root** package's settings combine, and the strictest applies:

- **Offline is absolute.** No package setting can allow the network back.
- **A package only tightens.** Its settings can make a build stricter than the
environment, never looser.
- **A dependency's settings apply to its own builds only.** Otherwise a
dependency could make every consumer's build stricter, or looser, without
them knowing.

| Environment | Root package | A build may fetch | Hash required (git) |
|---|---|---|---|
| online | default | yes, pinned content only | no (warning) |
| online | explicit fetching | no; `comline fetch` may | no (warning) |
| online | required hashes | yes | yes |
| offline | anything | no | yes |

## Hashes

A dependency's `hash` is the blake3 of its compiled content, and core already
checks a declared one. When it's **mandatory**:

- **A registry dependency: always.** A version string can be republished with
other content, so without a hash the pin names nothing.
- **A git dependency: when offline, or when the root package requires hashes.**
Otherwise the pinned commit already identifies the content, and today's
warning stays.
- **Any dependency with more than one source** ([supply](supply.md#several-sources-per-dependency)):
the sources don't share any other identity.

Offline is when the hash matters most: the bytes then come from a vendored
copy, a cache or a carried bundle, which have passed through other hands, and
the hash is what proves a copy is the pinned content.

A copy whose hash doesn't match is **an error, never a fallback**, naming
where it came from: it may be compromised.

## Options considered

| | On demand + switch (proposed) | Explicit fetch only |
|---|---|---|
| Day to day | as today | one more step after a clone or a pin change |
| A sealed environment | safe once the switch is set | safe by default |
| Strictness for one package | the package's "explicit fetching" setting | always |

On-demand with a switch keeps everyday use as it is; a mandatory hash where
it matters means getting can't change what is built; and a package that
needs the stricter behaviour asks for it, so it holds wherever the package is
built.

## Open questions

- Whether `comline fetch` takes `--for <target>`, to fetch only what one
target's generation needs.
82 changes: 82 additions & 0 deletions docs/docs/proposals/offline/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Offline and air-gapped use

Status: **draft** · two parts, read in order:
[fetching](fetching.md), then [supplying dependencies and tooling](supply.md)

**The question.** Nothing yet says how Comline works without internet
access. **What must hold, in every repository, so Comline can be used on a
machine or in a network that can't reach the internet, and so a build there
is exactly the build anywhere else?**

## The situations

All four are in scope:

| Situation | What it needs |
|---|---|
| **An offline developer machine** | Work from what was fetched earlier, or from a bundle, and be told clearly what's missing. |
| **An isolated build or CI network** | Build and generate from vendored copies or internal mirrors, never reaching outside. |
| **An air-gapped deployment** | Services built with Comline run in a sealed network: the runtime and generated code need nothing beyond the peer they talk to. |
| **The tooling into the gap** | The CLI, the language server, generators and docs brought into a sealed network, verifiably. |

## What it looks like today

| Part | Network today | Already offline-friendly |
|---|---|---|
| **core** (dependencies) | A `Git` dependency is fetched by the system `git` into a per-package `.comline/deps-cache/`. `Registry` sources exist in the config (with `hash` and `signature`) but nothing resolves them. | Pins are exact commits, and a checkout is reused. A mode that never fetches exists, which the language server uses. A declared `hash` (blake3 of the dependency's compiled content) is checked; without one, core warns and trusts the fetch. |
| **cli** | `check`, `add` and `build` fetch through core. It installs only with `cargo install --path .`: no release binaries, checksums or signatures. | Generators are built in, or local executables and `.wasm` files: nothing is downloaded. |
| **language server** | None. | It never fetches, and says "isn't fetched yet — run `comline check`". |
| **generated code** | Rust depends on `comline-runtime` by a git rev; TypeScript on `@comline/runtime` from npm. | Otherwise self-contained. |
| **runtime** | None beyond the peer it's given. | Fits air-gapped deployment as it is. |
| **specification** | The vendoring scripts clone from GitHub, only when a developer runs them. | Copies are committed, pinned and checked. |
| **docs** | The theme loads Google Fonts by default. | — |

Nothing reports usage or checks for updates.

## Rules that hold whatever the details

- **Choosing is separate from getting.** Choosing a dependency (`add`, a
future `update`) writes pins, and uses the network only when the user asks.
Getting bytes that are already pinned follows [the fetching
rules](fetching.md).
- **Identity is the hash, location is replaceable.** Where bytes come from
(a registry, git, a mirror, a vendored copy, a cache, a bundle) never
changes what is built. A copy is accepted only when it matches the pin.
- **Offline is absolute.** When the environment says offline, nothing touches
the network, whatever any package says.
- **Nothing calls home.** The runtime and generated code never reach anything
but the peer they're given. No tool sends telemetry or checks for updates.
The language server never fetches.

## Rollout

In stages, each useful on its own:

1. **Fetching rules in core and the CLI:** the offline switch, `comline
fetch`, the package settings and their precedence, and the hash rules.
2. **The shared cache and vendoring:** one hash-keyed cache per machine, in
place of the per-package deps cache, and `comline vendor`.
3. **Several sources per dependency, and mirrors:** ordered sources, machine
and organisation mirrors, and the root package's additions.
4. **Bundles:** export and import of a signed archive.
5. **Tooling into the gap:** release artifacts with checksums, signatures and
an SBOM, reproducible builds, and an offline docs bundle.
6. **Generated code's dependencies:** the runtimes published to crates.io and
npm, a hard requirement for 1.0. Until then, the docs say how to vendor
them.

## Where settings live

| Where | What | Frozen |
|---|---|---|
| `config.idp` | each dependency's identity (version or commit, and hash) and its `sources` | identity only; `sources` is never frozen |
| the project's `comline.toml` (committed, read only for the root package) | the fetch policy, and sources the root adds for transitive dependencies | never |
| machine and organisation config: a system file (`/etc/comline/config.toml`, `%PROGRAMDATA%`), a user file (`~/.config/comline/config.toml` and the platform's equivalents), or one named by `COMLINE_CONFIG` | `offline`, mirrors, the cache's location, trusted keys | — |
| the environment and flags | `COMLINE_OFFLINE=1`, `--offline` | — |

They combine, and the strictest applies.

## Remaining open questions

- Whether `comline fetch` takes `--for <target>`, to fetch only what one
target's generation needs.
163 changes: 163 additions & 0 deletions docs/docs/proposals/offline/supply.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# Supplying dependencies and tooling

Status: **draft** · part of [Offline and air-gapped use](index.md)

**The question.** How do dependencies, and Comline itself, get onto a machine
or into a network that can't fetch them, without that ever changing what is
built?

## Identity is the hash, location is replaceable

A dependency is identified by its pin: a version or a commit, and its content
hash. *Where* the bytes come from is a separate, replaceable thing. Every
route below is just another place to find the same hash-checked content, so
none can change a build: at worst one can fail to supply it.

## Several sources per dependency

A dependency may list several sources, in order, so it stays available
whatever happens to one of them:

```text
billing = {
version = "1.4.0"
hash = "blake3:…"
sources = [
{ registry = "comline" }
{ registry = "https://registry.acme.internal" }
{ git = "https://github.com/acme/billing", commit = "…" }
]
}
```

- **More than one source requires a hash.** A registry download and a git
checkout of the same package give the same content hash, and nothing else
they carry (a version string, a commit) says anything about the other.
- **An unreachable source, or one without the content, falls through** to the
next.
- **A source serving the wrong content is a hard error** naming it. Moving on
would still build the right bytes, but would hide what may be a
compromised source.

### Who can add sources

- **The package**, in its `config.idp`. The list is **never frozen**: only the
identity (version or commit, and hash) is, so moving a repository or adding
a source is no new version.
- **Machine or organisation configuration:** mirrors, as URL prefix rewrites
(`https://github.com/` → `https://git.acme.internal/`) or sources put ahead
of a package's own, without editing any package. Safe because the hash
decides.
- **The root package, for a transitive dependency,** in its `comline.toml`,
when that dependency's pin carries a hash: if C's only source is unreachable from your network and
the pin is in B's config, the root adds a source for C. It **adds, never
replaces**: content other than what B pinned would change what is built,
and possibly C's given-up guarantees. Replacing (a patch) is out of scope.

### Order

Machine and organisation mirrors first, then the root's additions, then the
package's own sources. Mirrors first keeps lookups inside the organisation:
each source tried learns the dependency's name. Offline, only local sources
count: the vendored copy, the cache, bundles.

The build reports which source served each dependency (verbose output and
`--json`). That's never frozen: location isn't identity.

## The shared cache

One content-addressed cache per machine, keyed by hash, in place of today's
per-package `.comline/deps-cache/`. Every route below feeds it, and a build
reads from it. Projects on the same machine share what's been fetched, and a
copy is checked against its hash on the way in.

- **Where:** the platform's cache directory: `$XDG_CACHE_HOME/comline`
(`~/.cache/comline`) on Linux, `~/Library/Caches/comline` on macOS,
`%LOCALAPPDATA%\comline\cache` on Windows. `COMLINE_CACHE`, or `cache-dir`
in machine configuration, moves it.
- **A read-only layer underneath:** an organisation can provision a shared
cache (`/opt/comline/cache`, say) that every user reads and none writes,
which suits a sealed network.
- **Cleaning is never automatic:** evicting silently could break a machine
that has since gone offline. `comline cache list` shows what's there, and
`comline cache prune` cleans by age or size.

## Vendoring

`comline vendor` writes every dependency a package needs, transitive ones
included, into a directory in the project, each with its pin and hash. It
travels with the repository through any channel that carries it, and a
change to a dependency shows up as a diff in review. A build uses a vendored
copy only when it matches its pin.

## Mirrors

An internal git server or registry mirror, set once per machine or
organisation, as above. Suited to organisations with many projects; a future
registry needs the same mechanism.

```toml
[[mirror]]
from = "https://github.com/"
to = "https://git.acme.internal/github/"

[[mirror]]
registry = "comline"
to = "https://registry.acme.internal/comline"
```

A mirror is **the only route** for what it covers: when it lacks something,
the original isn't tried, since every source tried learns what you depend on.
`fallback = true` on a mirror tries the original after it.

## Bundles

A single signed archive of dependencies, carried across the gap and imported
into the cache on the other side: `comline bundle export` on a connected
machine, `comline bundle import` on the sealed one. The same format can carry
the tooling.

- **Format:** a `.tar.zst` holding a `manifest.json` (each entry's kind,
dependency or tool, its pin, hash and size), the content stored by hash,
and a detached signature. Entries are in a fixed order with fixed
timestamps, so exporting the same set twice gives the same bytes.
- **On import,** each entry is checked against its hash before it enters the
cache, and the signature against the keys trusted in machine or
organisation configuration (`trusted-keys`).
- **An unsigned bundle is refused** unless `--allow-unsigned` is given. The
hash already proves a pinned dependency; the signature proves who
assembled the bundle, which matters for tooling nothing pins yet and for
an organisation's rule on what may cross.
- **Keys** are minisign's (Ed25519), the same as releases', so one tool checks
both.

## Generated code's dependencies

Generated Rust depends on `comline-runtime` by a git rev, and generated
TypeScript on `@comline/runtime` from npm. Each ecosystem has its own offline
tools (`cargo vendor`, registry mirrors, npm caches), which work best with
published packages.

- **Publishing the runtimes to crates.io and npm is a hard requirement for
1.0.** Generated code then names a version, not a git rev.
- **Until then**, the docs say how to vendor them for a sealed build:
`cargo vendor` handles git dependencies, and `npm pack` produces a tarball
to install from.

## Tooling into the gap

- **Release artifacts:** the CLI and the language server prebuilt for each
platform, with a checksums file, rather than `cargo install` from source
only.
- **Signatures:** minisign over the checksums file is the primary signature,
since it checks with one public key and no network. GitHub artifact
attestations (sigstore) are added on top, for provenance when checking from
a connected machine.
- **Reproducible builds:** anyone can rebuild an artifact from source and
compare bytes, as comline-typescript already does for its wasm core.
- **An SBOM per release**, for sealed environments' audits, as **CycloneDX
JSON**: what vulnerability scanners read, with mature Rust
(`cargo-cyclonedx`) and npm tooling. SPDX can follow if someone needs it.
- **An offline docs bundle:** the documentation as a self-contained archive,
with no outside assets (no Google Fonts), and generated schema docs
(`doc-html`) likewise.
4 changes: 4 additions & 0 deletions docs/mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,10 @@ nav:
- 'Proposal B': 'proposals/guarantee-marker/assumes.md'
- 'Proposal C': 'proposals/guarantee-marker/unverified.md'
- 'Proposal D': 'proposals/guarantee-marker/unchecked.md'
- 'Offline and air-gapped use':
- 'proposals/offline/index.md'
- 'Fetching': 'proposals/offline/fetching.md'
- 'Supplying dependencies and tooling': 'proposals/offline/supply.md'
- 'A shared vocabulary':
- 'proposals/shared-vocabulary/index.md'
- 'The catalog': 'proposals/shared-vocabulary/catalog.md'
Expand Down
Loading