Helios

One small program that runs your machines.

Builds, certificates, secrets, backups, health checks and alerts.
Everything a server needs for the apps it hosts, from a single static binary.

Quick start · Manifest · Isolation · Deploys · Backups · Commands · Docs

Renaming to Helios

Helios is the new name for Bedrock. The source module and primary command are now helios; bedrock remains a compatibility entrypoint. New projects use helios.yaml. Existing bedrock.yaml files are supported; if both defaults exist, select one with --manifest or remove the duplicate.

This source rename preserves the Bedrock installation layout, service names, Docker resources, sealed credentials and backup identities. Both commands use the same state and daemon. HELIOS_* configuration variables are accepted alongside their BEDROCK_* aliases; conflicting values are refused unless an explicit corresponding flag overrides them. Workloads receive both sets of app, workload and revision metadata variables.

The GitHub repository and release channel use kylebegeman/helios. Release builds produce Helios assets and identical legacy-named assets for older upgraders. See the migration record for the completed source and host rollout.

Why Helios

• One binary, nothing to install underneath. A static Go binary installs and operates the whole machine. There is no agent to babysit, no runtime to keep current, and no control plane to pay for.
• The manifest is the entire interface. An app describes itself in helios.yaml. Everything else follows from that file: DNS records, certificates, secrets, data services, health, backups and alerts.
• Isolation you get by default, not by remembering. Read-only root, no Linux capabilities, no privilege escalation, a non-root user, and a private network per app. The manifest holds the explicit ways out, and helios exposure shows what each container really got.
• Backups that are proven rather than hoped for. Every backup is quiesced and consistent, and helios drill restores it beside the app, starts the app on it and verifies it on a schedule.
• No pipeline required. git push helios main deploys and streams every step back. A push whose deploy fails is refused, so the branch on the machine is always what runs.

How a deploy works

flowchart LR
  M["helios.yaml"] --> DNS["confirm<br/>DNS"]
  DNS --> SEC["make<br/>secrets"]
  SEC --> DATA["ready<br/>data"]
  DATA --> BUILD["build<br/>images"]
  BUILD --> REL["run release<br/>workloads"]
  REL --> START["start beside<br/>the old"]
  START --> CHK["run<br/>checks"]
  CHK --> SW["switch<br/>the edge"]
  SW --> CERT["wait for<br/>certificates"]
  CERT --> CHK2["check through<br/>the edge"]
  CHK2 --> RET["retire the<br/>old revision"]
  RET -.->|"helios rollback"| START

Loading

The old revision is retained. helios rollback brings it back when the data compatibility contract permits it; unsafe data transitions require an explicit checkpoint restore. Every container receives HELIOS_APP, HELIOS_WORKLOAD and HELIOS_REVISION, along with the corresponding BEDROCK_* aliases.

Quick start

helios host setup                   # prepare a fresh Ubuntu or Debian machine
helios integration set cloudflare   # DNS records for your hosts
helios integration set storage      # Backblaze B2 or any S3 store, for backups
helios integration set email        # SMTP, for alerts
helios integration set pushover     # or ntfy: alerts that reach a phone
helios integration set loom_webhook # a Loom thread per critical incident

mkdir my-app && cd my-app
helios init my-app --host app.example.com  # write a starting helios.yaml
helios deploy .                     # build, check, switch the edge
helios status                       # health, traffic, errors, backups

helios launch https://github.com/you/site --host site.example.com  # or from a repository

Helios needs Ubuntu 22.04 or 24.04, or Debian 12 or 13, on x86_64 or aarch64.

An app's manifest

An app describes itself in helios.yaml at the root of its source. helios init writes a commented one for the shape you are building, so a new app never starts from a blank file:

helios init blog --kind static --host blog.example.com
helios init api --host api.example.com --postgres
helios init mailer --kind worker
helios init nightly --kind cron --schedule "0 4 * * *"

It only writes a file; nothing on a machine is touched. What it writes is parsed and validated before it reaches disk, so a generated manifest always loads. Here is one in full:

app: hello
description: What this is for
owner: personal
workloads:
  web:
    kind: web              # web, static, worker, cron or release
    build: {context: .}    # or image: ghcr.io/you/hello:1.2
    port: 8000
    routes:
      - host: hello.example.com
        dns: direct        # helios keeps the Cloudflare record (or proxied)
        # origin_pull: required  # with dns: proxied; see Authenticated origin pulls
      - host: example.com
        path: /hello       # a prefix; longest wins on a host
    env: {GREETING: hello}
    secrets: [API_KEY]     # names only; values live in helios's store
    health: {path: /healthz}
    resources: {memory: 256m, cpus: 0.5}   # memory defaults to 2g
    tmpfs: [/app/.cache]   # writable in memory; the root filesystem is read-only
    # user: "1000:1000"    # default: the image's user, or nobody for a root image
    # capabilities: [NET_BIND_SERVICE]     # default: none at all
    # writable_root: true  # the explicit way out of a read-only root
    # privileged: true     # for the rare workload that runs containers
  site:
    kind: static
    dir: public            # served by a file server
    routes: [{host: www.example.com}]
  nightly:
    kind: cron
    image: alpine:3.21
    schedule: "0 2 * * *"  # five fields, UTC
    command: [sh, -c, "echo tidy"]
    mounts: [{volume: files, path: /files}]
data:
  postgres: {version: "16"}  # DATABASE_URL reaches every workload
  volumes:
    files: {description: What people upload}
backup:                      # optional; an app with data is backed up
  schedule: "0 3 * * *"      #   nightly at 03:00 UTC (the default)
  drill: "0 4 * * 0"         #   restored and verified on Sundays (the default)
  keep: {daily: 7, weekly: 4, monthly: 6}
  verify: {sql: select count(*) from posts, at_least: 1}
checks:
  - url: https://hello.example.com/
    contains: hello

A cron workload runs on its schedule, in UTC, one run at a time, and a run still going when the next is due skips that one. helios jobs <app> lists its runs by ID, and helios logs <app> --job <id> shows what one printed, even after its container is gone. To pause it without a deploy, helios cron hold <app> <workload> --until <time> --reason <text> holds it until an RFC 3339 time at most 14 days ahead. A hold doesn't stop a run already going, and when it ends the workload runs at its next scheduled time, with no run to catch up on the ones it held. helios cron release <app> <workload> ends a hold early. helios status, helios jobs and helios ps show the hold and its reason, and the daemon sends it as a notice, then as a warning on its last day.

A larger app, with releases, singletons and derived secrets

An app with several parts, such as Loom's Core, uses a few more fields:

app: core
workloads:
  api:
    kind: web
    build: {target: server}      # workloads built alike share one image
    port: 4773
    routes:
      - host: core.example.com
        path: /api
      - host: runners.example.com
        port: 4774               # a route may reach another port
    aliases: [control]           # more names on the app's network; "api" is one
    singleton: true              # never two at once: the old one stops first
    grace: 60s                   # time to stop before it is killed (10s)
    secrets: [API_DATABASE_URL]
  worker:
    kind: worker
    build: {target: server}
    health:
      command: [node, bin/ready.js]   # run inside; exit 0 means ready
      timeout: 5m
    resources: {pids: 2048}
  migrate:
    kind: release                # runs once per deploy, before anything starts
    order: 1                     # lower first, then by name
    build: {target: server}
    command: [node, bin/migrate.js]
    secrets: [MIGRATION_DATABASE_URL]
    timeout: 10m
data:
  postgres:
    version: "17"
    image: pgvector/pgvector:0.8.0-pg17   # default postgres:<version>-alpine
    user: core_admin             # the superuser; default the app's name
    database: core
    init: deploy/postgres        # first-run scripts, from the source
    secrets: [API_PASSWORD]      # given to the first-run scripts
    database_url: false          # no automatic DATABASE_URL
  objects: {}                    # a MinIO of the app's own at {objects}:9000
secrets:
  generate:                      # made once, when missing
    API_PASSWORD: hex:32         # hex, base64 or base64url of N bytes; value:TEXT
  derive:                        # made again whenever a source changes
    API_DATABASE_URL: "postgres://core_api:{API_PASSWORD}@{postgres}:5432/core"
    MIGRATION_DATABASE_URL: "postgres://core_admin:{BEDROCK_POSTGRES_PASSWORD}@{postgres}:5432/core"

Generated and derived values stay on the machine; only their names appear anywhere. {postgres} and {objects} are the data services' names on the app's network, the object store's root user is MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, and {sha256:NAME} is the hex SHA-256 of a secret. A release that fails stops the deploy before anything new starts. A singleton that fails to come up is replaced by its predecessor again. A backup holds the database, every volume, the object store's files and the first-run scripts, so a restore on a new machine rebuilds the same roles before it loads the dump. helios run --secret NAME gives a one-off command one more secret, and helios run --stdin hands it this terminal's input without recording it anywhere. helios exec reads standard input only at a terminal or with -i, so in a script piped to bash -s it leaves the script's next lines alone. Piping or redirecting input into it needs -i before the --, as in helios exec site -i -- sh -c 'cat > /data/import.csv' < import.csv; without -i the command reads nothing.

One source can hold several apps, such as a product and a worker that runs other people's code apart from its data: helios deploy <dir> --manifest helios.worker.yaml deploys the other one, and helios secret copy <app> NAME <other-app> gives it a secret the first made, without showing it.

A port on the machine itself, past the edge

Traffic that isn't HTTP on a hostname, such as Headscale's STUN, is published on the machine's own address instead of routed through the edge:

app: headscale
workloads:
  server:
    kind: web
    image: headscale/headscale:0.26
    port: 8080
    routes: [{host: hs.example.com}]
    singleton: true              # required: two copies can't hold one port
    ports:
      - port: 3478               # the container's port
        protocol: udp            # tcp (the default) or udp
        # host_port: 3478        # the machine's port; default the same
        # address: 203.0.113.4   # one of the machine's addresses; default all

Only web and worker workloads publish ports, and only as singletons: a deploy starts the new container beside the old one, and the two can't hold one port, so a singleton's old container stops first. 22, 80, 443 and 5000 are the machine's own. A port another app on the machine already publishes is refused while the deploy is still a plan, and the plan lists what it will publish. One-off jobs, drills and previews publish nothing.

Docker opens a published port ahead of ufw's rules, so the firewall neither opens nor closes it: anyone can reach it. Deploys don't change the firewall, so add the rule yourself, once, on the machine:

ufw allow 3478/udp

That rule changes nothing about what is reachable; it makes ufw status say what is open. helios doctor lists every port an app publishes on a public address, and warns about one ufw has no rule for, or allows only from some sources, since Docker publishes it to everyone either way. To keep a port off the internet, give it a private address; ufw can't narrow it.

helios host setup --web-from cloudflare does not cover published ports. Its guard holds 80 and 443 alone, so a published port still answers anyone who knows the machine's address, and the doctor says so beside each one.

How apps are kept apart

Each app runs as if it were alone on the machine.

flowchart TB
  NET(["the internet"]) --> EDGE["Caddy edge<br/>certificates and routing"]
  EDGE --> A1["web"]
  EDGE --> B1["web"]
  subgraph APPA ["app: hello"]
    A1 --- A2[("postgres")]
    A1 --- A3["worker"]
  end
  subgraph APPB ["app: notes"]
    B1 --- B2[("postgres")]
  end

Loading

Its workloads share a network with each other and its database; the ones that serve traffic also share a network with the edge, and with nothing else. No app can reach another app's containers or database, or the edge's admin API, which is a socket only the machine reaches.

Default	What a workload gets
User	the image's user, or nobody when the image would run as root
Capabilities	none at all
Privilege escalation	refused
Root filesystem	read-only, with /tmp in memory
Memory	2 GiB
Processes	4096

The manifest fields above are the explicit ways out, and helios exposure shows each container as Docker runs it: its user, privileges, root filesystem, networks, published ports and mounts, and anything that shares a network it shouldn't.

Deploying without a pipeline

A route with dns: direct or dns: proxied gets its Cloudflare record from the deploy, which refuses a record that points at another machine; helios dns point <host> moves one on purpose. Retiring a route, or the app, removes its record. helios dns explains every host; helios dns audit finds records that point here with nothing routed, and helios dns drop <host> deletes one: only the A and AAAA records at exactly that name, only when each points at this machine, and only when no app here routes the host. A wildcard goes only when it is named itself.

An app is told who its visitor is in X-Forwarded-For and X-Real-IP: one address each, whatever the request claimed. Behind Cloudflare's proxy that is the visitor Cloudflare names, believed only on connections from Cloudflare's own ranges, which helios host setup keeps on every machine.

A proxied record hides the machine's address, but only until someone learns it. helios host setup --web-from cloudflare makes that not matter: only Cloudflare's proxy may reach 80 and 443, so the address answers nobody else on the web. Setup reads Cloudflare's published ranges and allows them in ufw before it removes the open rules. Docker forwards the edge's published ports past ufw, so setup also puts a guard in Docker's DOCKER-USER chain, run again at every boot before Docker starts. It keeps a copy of the ranges in /etc/helios for when the list can't be read, and helios host reconcile keeps it current. Every route must then be dns: proxied: setup refuses while an app routes a host any other way, and deploy, rollback and dns point refuse to add one. SSH stays reachable, keys only. Running setup again keeps the setting unless --web-from is given, and --web-from anyone opens the web again. The guard covers 80 and 443 only: a port an app publishes on the machine with ports: stays open to anyone, and helios doctor says so.

helios git allow <app> <public key> lets a key push that app and nothing else. Then, from the app's checkout:

git remote add helios bedrock@<machine>:<app>.git
git push helios main        # deploys, and streams every step back

A push whose deploy fails is refused, so the branch on the machine is what runs. helios deploy . --to bedrock@<machine> sends a working tree the same way, commit or not. helios git webhook <app> --repo git@github.com:you/app.git deploys when GitHub says the branch moved, with a deploy key and secret helios makes and shows once.

A push key says who may push; it doesn't say a person wrote what was pushed. helios git signers add <name> <public key> allows a person's SSH signing key, and helios git require-signed <app> then deploys that app only from a tip commit one of them signed, over push and webhook alike. An unsigned commit, a key nobody allowed, a GPG signature (GitHub signs the merges it makes that way) or a key not allowed to sign git commits is refused before anything is built, and the refusal names who may sign. A tree sent with --to bedrock@ has no commit to check, so it is refused too, and so is a push that adds a refs/replace/ ref, which could change what a signed commit deploys. To sign, set git config gpg.format ssh, git config user.signingkey ~/.ssh/id_ed25519.pub and git config commit.gpgsign true where you commit, and allow the signer before requiring signatures. Root is the break glass: helios deploy on the machine or --to root@, rollback, launch, preview up and recover aren't held to signatures. The policy lives in /etc/bedrock/git, and helios recover and helios move don't carry it yet, so run signers add and require-signed again on a recovered machine.

helios launch <repository> --host <hostname> takes a repository from nothing to running: it fetches the branch, works out what the source is (a Dockerfile, or a built site), writes the manifest it is missing and deploys it. What it guesses is deliberately narrow; anything else is refused with a suggestion.

helios preview up <repository> --branch <branch> --domain preview.example.com runs a branch as its own app beside the one it is a branch of: a derived name, its own containers and an empty database, at hostnames under the preview domain, with every route behind a sign-in (auth: loom, below). Pushing the same branch again updates it; helios preview ls lists them, helios preview down <preview> --data removes one and its data. Use helios preview pin <preview> to retain a branch, then review helios preview prune --older-than 7d --plan to clean up unpinned previews. Pins survive updates and rollbacks. Preview checks retain application and routed assertions; external checks that cannot be rewritten are reported.

A staged move starts with helios move rehearse <app> --to <target-hostname> and helios move test <source> --handoff rehearsal.json. The final capture, helios move final, persistently fences source writers. move stage restores that exact capture privately, then move cutover deliberately admits target writers and traffic. The source stays stopped across restarts. Both machines read the same backup repository, and move secrets preserves app credentials. See safe moves for planning downtime, retrying cutover and deliberately resuming or retiring the source. The older move out and move in commands remain available for manual copies.

helios remove <app> takes an app off the machine and keeps its volumes, database and sealed secrets, so it can come back with what it had. --data removes those too, and --secrets forgets only the secrets. An app removed without its data can be cleared later: helios remove <app> --data for an app that is no longer on the machine finds what it left behind (its sealed secrets, volumes, database and networks), names each one in the plan and removes them. Its backups in the storage bucket are never touched, and are the only way back.

Named machines and portable setup

Save and verify an SSH target, then select it explicitly for inspection or operations. Plans bind the connection, machine identity and uploaded source.

helios target set staging --ssh helios-staging --sudo
helios target verify staging
helios --target staging check . --host
helios --target staging deploy . --plan

Portable profiles pin the host configuration, binary version/checksum and integration names. helios --target staging host bootstrap --profile staging.json installs and verifies that profile with resumable client receipts. Selected integrations are transferred as recipient-bound age ciphertext while the destination keeps its own identity. See named targets and bootstrap for the complete workflow, and compatibility for upgrade preflight and state checkpoint guarantees.

A sign-in in front of a route

A route marked auth: loom is put behind a Loom Core's sign-in by the edge: the app behind it never sees a request from someone who is not signed in, and needs no auth code of its own. helios integration set loom names the Core's verify endpoint. A machine without that integration refuses to deploy a guarded route rather than serve it open, and previews rely on it.

Authenticated origin pulls

A dns: proxied host hides the machine's address behind Cloudflare, and helios host setup --web-from cloudflare lets only Cloudflare reach it, but any Cloudflare zone, someone else's included, can point at that address. With origin_pull on its routes, a host takes only the connections that present the client certificate Cloudflare holds for it, checked during the TLS handshake:

routes:
  - host: shop.example.com
    dns: proxied
    origin_pull: required    # optional while Cloudflare starts presenting it

Turning it on for a host goes in this order:

1. Mint the certificate. On the machine, or from your computer with --target, where the certificate is made locally and only its CA goes to the machine:

   helios edge origin-pull mint shop.example.com --out shop.example.com.pem

   This makes a CA and a client certificate it signs, both good for a year. The edge trusts the CA at once, and the CA's key is thrown away, so nothing can sign another certificate the edge takes. The file gets the certificate and then its key, readable by you alone. --out must name a file that doesn't exist yet, so no key is ever written over. If the machine keeps the CA but its edge doesn't reload, the file is still written and mint says so. With --target, if the machine then can't be asked whether it kept the CA, the file is kept too, and mint names the list and forget commands that check or drop the CA there before you upload anything.

2. Upload it in the Cloudflare dashboard, under SSL/TLS, Origin Server, Authenticated Origin Pulls, Per-hostname: choose Upload certificate, paste the file's CERTIFICATE block as the certificate and its PRIVATE KEY block as the private key, and associate the certificate with the hostname. Cloudflare presents it once the hostname is associated. Then delete the file: it holds the only copy of the key.

3. Set the zone's SSL/TLS encryption mode to Full (strict).

4. Deploy with origin_pull: optional. The edge checks a certificate that is presented and still lets a connection without one through, so nothing breaks while Cloudflare starts presenting it. A request whose certificate the edge checked is in its access log on the machine, with the certificate's name:

   docker logs --since 10m bedrock-edge 2>&1 | grep -F '"client_common_name":"shop.example.com"'

5. Only once requests through Cloudflare show up there, deploy with origin_pull: required. From then on a connection that doesn't present the certificate is refused before it sends a request.

A deploy, rollback, move or recovery cutover that would leave a host requiring origin pull with no CA the edge trusts is refused while it is still a plan, since the edge would refuse every connection to it. That includes a host another app on the machine requires, even when the app being deployed sets origin_pull: optional for it or leaves it out. helios edge origin-pull list shows each host's mode and the CAs it trusts, until when; --json adds each CA's PEM.

What to know before turning it on:

• Full (strict), never Flexible. A host with origin_pull, optional or required, answers plain HTTP on port 80 only with a 308 redirect to HTTPS, since there is no certificate to check over it. Under Flexible, Cloudflare reaches the machine over plain HTTP, hands that redirect back to the visitor, and sends the next request over plain HTTP again, round the same loop for good.
• 421 for a mismatched name. Once the edge checks certificates for any host on the machine, every HTTPS request to it, for any host, must name the same host in its TLS handshake (SNI) as in its Host header. One whose Host header differs from the name in its handshake gets 421 Misdirected Request. A connection whose handshake names no host, such as curl to the machine's address with only a Host header, fails during the TLS handshake and never gets an HTTP status. Without that, a connection made under another host's name, which asks for no certificate, could carry the protected host's Host header. Cloudflare sends the hostname in both; an Origin Rule that rewrites one without the other gets 421 too.
• Hosts deeper than one level. Cloudflare's Universal SSL certificate covers the zone and one level below it, shop.example.com but not eu.shop.example.com. A proxied host deeper than that needs an edge certificate at Cloudflare that covers it, an advanced or a custom one, or visitors never get as far as the machine.
• 10 per zone. Cloudflare allows 10 per-hostname certificates per zone, and each host gets its own, so a zone has at most 10 hosts with origin_pull, and rotating one needs a free place for its new certificate until the old one is deleted.
• Never the shared certificate. Cloudflare's global authenticated origin pull certificate is shared by every Cloudflare account, so it proves only that a connection came through Cloudflare, not through your zone. Don't give its CA to helios edge origin-pull trust; use the certificate minted for the host.
• Never on a dns: direct host. Its visitors connect to the machine themselves, with no certificate to present, so the manifest check refuses origin_pull without dns: proxied. Every route to a host within one app's manifest must set the same origin_pull, and an app's previews never set it, since Cloudflare holds no certificate for a preview's hostname.
• One host, several apps. Apps that route different paths of one host may set different origin_pull, so they can switch the host one deploy at a time, in either order. The host takes the strictest mode any of them asks for, so an app that relaxes a host another app still requires leaves it required.

To rotate, mint again into a new file, upload that certificate at Cloudflare, associate the hostname with it, and delete the old certificate there. Rotating always means minting again: nothing renews a certificate, and the CA that signed the old one can sign nothing more. helios keeps trusting the old CA beside the new one until you forget it, so the host keeps working while Cloudflare switches. helios can't see which certificate Cloudflare presents, so forget the old CA once Cloudflare presents the new certificate; mint prints the command when the host trusts an older CA. An old CA left to run out becomes a critical alert, and only forget clears it, since trusting or minting another CA never drops one:

helios edge origin-pull mint shop.example.com --out shop.example.com-2.pem
# Once Cloudflare presents the new certificate, drop the old CA:
helios edge origin-pull list
helios edge origin-pull forget shop.example.com sha256:<fingerprint>

forget without a fingerprint drops every CA the host trusts. It won't leave a host that requires origin pull with none, since the edge would then refuse every connection to it; deploy the host with origin_pull: optional or without origin_pull first.

The daemon warns 30 days before each CA a host trusts runs out. For an old CA beside a newer one, the warning says to forget it once Cloudflare presents the new certificate. A CA that runs out while the host still trusts it beside a newer one is a critical alert in either mode, since the edge then refuses the certificate Cloudflare may still present with it; it stays open until you forget that CA. A host that requires origin pull with no usable CA left is a critical alert too. A host that no app sets origin_pull for raises no alert, whatever its CAs, since the edge asks it for no certificate.

When a CA runs out, the daemon reloads the edge, so an optional host whose last CA ran out then checks no certificate; until that reload, the edge refuses the expired one. An edge that isn't answering then, or when the daemon starts after a CA ran out, is reloaded within a minute of answering. helios upgrade refuses a build without origin-pull support while any host sets origin_pull, since that build's edge would stop checking the certificate. Moving an app that requires it needs the CA on the other machine first; see safe moves.

A Loom thread for each critical incident

A critical incident can open a thread in Loom that already holds what was known when it began: the change before it, its logs, earlier incidents and upstream status. In Loom, make a scheduled task with a webhook trigger that starts a new thread for each run. Have it ask for approval, and give it no tools that write, because its evidence holds log lines that anyone who reaches an app can put words into. Set its signature check to header x-helios-signature, hex, prefix sha256=, with a secret of 16 to 512 characters. Its prompt should treat {{body.message}} and {{body.evidence}} as data, and stop when {{body.transition}} is test. Then on the machine:

helios integration set loom_webhook    # the task's relay URL and the secret
helios alerts test                     # probe the task; opens no thread
helios alerts test --leg loom_webhook  # a signed test; opens one thread

Only the task's URL on the T3 Connect relay is accepted, https://<relay>/v1/hooks/<endpoint>/<task>/<token>, with no query. Loom's own /api/hooks route is refused. Both values are kept secret, and an error never repeats them.

Loom tells an agent, not a person, so it never goes alone. With loom_webhook the only way set up, no alert goes out at all, and each incident says loom_webhook needs a person leg too; set email, pushover or ntfy, which helios alerts shows as NOT DELIVERED. This is checked when an alert goes out, not when the integration is set.

The task hears of an incident once, at its first critical notice, so a warning that grows critical is sent when it does. Warnings, daily repeats and recoveries never reach it. The body carries the incident's id, host, key, severity, message and opening time, and its evidence cut to fit 90,000 bytes. It is signed with the secret in x-helios-signature, as sha256= and the hex HMAC-SHA256 of the body.

Each incident is sent at most once. Before the request goes out, Helios writes a marker beside the incident, in /var/lib/bedrock/incidents/<id>/loom.json, and never sends that incident again while the marker is there. A request that never went out, one cut off while it was written, or a 429, takes the marker back, so it is tried again like any way that failed. It is tried until Loom takes it, even after the outage has recovered or eased to a warning, and until then helios alerts show <id> reads NOT DELIVERED. The daemon keeps that retry in memory, so a restart drops it for an outage that has already recovered or eased. Otherwise the marker ends in one of these, which helios alerts show <id> reads back:

• delivered: any 2xx.
• refused: a 3xx, 400, 401, 404, 405, 409, 410 or 413. The task turned it away, and it isn't sent again. It counts as this way's failed test, so host:alerts warns through the other ways until helios alerts test --leg loom_webhook passes or Loom takes a later incident.
• ambiguous: any other answer, or none, once the request went out, such as a 504 or a timeout. Loom may have opened the thread or may not. Helios won't risk a second one, so it isn't sent again: look for its thread in Loom. helios alerts show reads ambiguous 504, not resent.
• sending: the daemon stopped partway through. That too isn't sent again.

In Loom's connection settings, turn on "Hold webhook requests while this computer is offline". The relay then keeps a request for up to 24 hours and answers 202. Without it, an incident sent while Loom is offline gets a 503 or 504, which is ambiguous, and no thread opens. Loom checks a request the relay held only once it is back, so a held incident is delivered but clears no refusal, and a held probe or test shows nothing: helios alerts test says the relay held it and keeps the last result.

helios alerts test, and the daemon's Monday test, send the task a probe signed with a random key. A task that checks the signature turns it away with a 401 and opens no thread, and that is a pass: the relay, the URL and the signature check all work. A task that takes the probe would open a thread for anyone who has its URL, so the test fails and says how to set the check. --leg loom_webhook sends a test signed with the real secret, which opens a thread whose prompt should stop at once. Neither leaves a marker. A probe that passes never clears a refusal or a failed signed test; a signed test that passes does.

What Helios keeps an eye on

The daemon watches every app and the machine once a minute: containers running, health paths and checks answering through the edge, certificates valid and renewing, disk and memory, backups fresh (the machine's own too) and drills passing, each workload able to write the volumes it mounts (checked hourly), an operation holding the machine's operation lock for over 4 hours (backups, drills and deploys wait behind it), backup retention deferred for over 3 days, and any URL added with helios watch add (the other machine's sites, say). A problem has to hold for three rounds before it becomes an alert, one alert per app, and you hear once when it starts, once a day while it lasts, and once when it recovers. An alert that grows graver, a warning that becomes an outage, is told as soon as the graver state has held three rounds too, rather than at the next daily repeat. One that eases, an outage back to a warning, stays the outage until the milder state has held two rounds, so an outage that flaps is told once.

A machine that is down can't say so, so machines that run Helios watch each other with helios peer. Every minute each server asks the others for a short summary over the mesh, checks their edge over the mesh, and checks their public URLs from the outside. Together the three answers tell a machine that is down from a Helios that has stopped, a broken mesh or a failing edge, and one healthy machine sends the page for all of them. A peer that answers none of the three is critical once most of the machines watching it agree. The summary also shows what is quietly wrong on a peer: a backup over 36 hours old, an overdue restore drill, or a way of sending alerts that has delivered nothing in 8 days. A Mac can add its own view every minute and never pages. See peer watch.

Alerts go out every way that is set up, all at once, and every incident told in a round goes out together, so a slow way, such as Loom while it collects an outage's evidence, never holds back a page. Email goes through the email mail server, and through email2 only when the first one fails, so the same mail never arrives twice. Pushover or ntfy brings warnings and critical incidents to a phone. A critical incident rings through Pushover every minute until you acknowledge it, for up to an hour, and its recovery stops the ringing. A way that fails is tried again on its own, with the wait between tries growing to 15 minutes, so the ways that worked don't repeat themselves. ntfy needs no account and can run on your own server, but it can't ring until acknowledged, and a server of your own reaches an iPhone at once only with upstream-base-url: https://ntfy.sh in its config.

helios alerts test sends a test alert each way and says how each did; --leg pushover tests one. The daemon runs the same test every Monday at 09:15 UTC, and a way that fails becomes a warning the others deliver.

helios alerts show <id> shows one incident: when it opened and resolved, when you were told, any alert that wasn't delivered, and how its Loom thread went. --evidence adds what was known about it. For an app, that is the last deploy or rollback and its steps, the logs from the 10 minutes before the incident opened, and the app's signals over 6 hours. For the machine, it is the deploys and rollbacks in the hour before. Both get the key's earlier incidents and the status of Cloudflare and GitHub. Secrets, credentials, and private and mesh addresses are redacted, and text escaped too many times to search for a secret is left out. Logs, step notes and incident messages are left out when the secrets can't be read. The daemon keeps this evidence when it sends an incident to Loom, in /var/lib/bedrock/incidents/<id>/, at most 256 KiB. Otherwise it is collected and kept the first time you ask, and by then logs older than the journal keeps are gone. --json prints the incident, its Loom delivery and its evidence.

helios alerts review <id> prints a blameless review in Markdown for you to finish. Its timeline is filled in from the change before the incident, the operations around it, the alerts and the Loom delivery, and the key's earlier incidents. Log lines and step notes are left out of it. The blank review is docs/operations/incident-review.md.

A backup stops an app's writers for at most 30 minutes. A writer the backup stopped is not an alert until the pause has lasted 35 minutes, and then it is a critical one.

A disk with under 5 GB free is critical. The daemon also keeps how much of the disk is in use each hour for 90 days, and each minute of the last 6 hours in memory. It fits a trend through each that a few odd samples can't swing. A lone step each way, such as a cleanup or a large download, is set aside so only steady growth counts. The 12 samples either side of a step say its size, so a temp file that sits for a few samples can't pass for one. A passing bump, such as a temp file or a nightly dump, is smoothed over, and so is a step undone within 24 samples. Repeated steps, such as a nightly import, are read as the rhythm they are. A disk the 7 day trend fills within 14 days is a warning, and within 48 hours it is critical. A disk the 6 hour trend fills within 6 hours, as a runaway log does, is critical too, but only while usage still climbs at half that rate or more. Usage still climbs while it has reached its high, give or take the noise, within the last 40 minutes, or while it comes in bursts, three or more in a row, and the next is no later than one and a half times the longest wait between them so far. It has to climb both as sampled and with the steps set aside. A cleanup doesn't stop the climb: up to 40 minutes of a real fall don't count, and after it the climb counts again from the bottom. The rate counts only the rises, across whole bursts in the last 2 hours, or else the newest two, so a runaway that writes every hour reads the same between its bursts as at them. One download is one burst, so the disk goes quiet 40 minutes after it ends. A runaway that has slowed stays critical while its slower rate still fills the disk within 6 hours, and the alert then goes by that rate. The alert, about free disk space, says how fast the disk is filling and which app's data grew most. A disk with under 10% free is critical until there are 2 days of hourly samples. After that it is a warning while the 7 day trend lasts beyond 14 days or isn't growing.

Command	Shows
helios alerts	what is wrong now, and what was
helios peer ls	the machines this one watches, and the ones that may read it
helios status	health beside each app's requests, errors, p95, CPU, memory and disk
helios ls	the registry: owner, hosts, repository, last deploy, last backup
helios exposure	what each container may do and what reaches it
helios doctor	what to fix on this machine; changes nothing

Backups that restore

Backups go to one bucket per app with restic, encrypted with a password made once per storage account.

App backups drain Helios-managed jobs, stop running workloads gracefully, stop the object store, dump PostgreSQL, and snapshot the unchanged volume files. This is a maintenance window: the app is unavailable for the capture and upload, bounded to 30 minutes. Backups then restart exactly the containers that were running, with the object store first. Retention runs after service is restored. A forced shutdown aborts the capture instead of certifying an inconsistent snapshot.

A volume the app rebuilds or fetches again, such as a cache, build output or a local copy of data kept elsewhere, can stay out of its backups:

data:
  volumes:
    state: {}
    cache:
      backup: false

The backup's plan names each volume it leaves out. Such a volume is empty after every restore, drill and move: Helios creates it with nothing in it and says so, and a drill gives it to its workload's user as a deploy does. Keep anything the app can't rebuild in a volume that is backed up. A release's safety checkpoint still copies every volume, so a rollback puts each back as it was. The PostgreSQL dump is always backed up, and leaving out every volume of an app with no database fails validation; to back up none of it, set backup.off: true instead.

Every restic call a backup makes has a limit, so one stuck call can't hold the machine's operations: opening the repository and clearing stale locks 5 minutes each, a machine backup's upload 2 hours, forgetting old snapshots 15 minutes and the prune that follows 1 hour. A machine backup also copies each data app's pinned snapshot in with a 2 hour limit per app and archives the workload images with a 1 hour limit. In the step that copies them, connecting to Docker, and resolving and sizing the workload images, have a 2 minute limit each. A call that runs out of time is stopped, and restic gets a minute to release its repository lock. When retention fails or runs out of time after a snapshot was taken, the snapshot is kept and the backup still counts as good: helios backups shows "retention deferred" with the reason (so does helios status <app> for an app's last good backup), and the next backup applies retention again. Every backup prunes, even when it forgot nothing, and one prune rewrites at most 2 GB, so a repository with a lot to reclaim catches up over several backups. Retention deferred on every backup for over 3 days raises an alert. No successful machine backup for 36 hours raises a warning. Before each backup Helios runs restic unlock, which removes only locks restic considers stale (not refreshed for 30 minutes). It never removes a live lock, so if a stopped call couldn't release its lock, a backup started within the next 30 minutes can still find the repository locked and fail; the next scheduled run goes through.

Command	What it does
helios backup <app>	run one now
helios backup helios	preserve state, sealed secrets, images and pinned app recovery data
helios drill <app>	restore the latest snapshot beside the app, start it, verify, clean up
helios restore <app>	bring data onto a machine that doesn't run the app yet
helios backups	list what happened

helios drill uses an internal network with no outbound access, starts all long-running workloads before probing readiness, and never runs release or cron jobs. Privileged workloads cannot be safely drilled, and workloads that require external services may fail their drill readiness check. A drill also fails when a restored volume holds files its workload's user can't write, such as a database copied in by hand as root: that app would answer its health check and then fail its first save.

Before it restores anything, a drill checks that the snapshot fits with 5 GB or a tenth of the disk to spare, whichever is larger: the floors helios watch warns at. It checks every disk it writes to: its working directory, each scratch volume and the scratch database's volume. The size is what the snapshot's backup read, or restic's own count when the snapshot doesn't record it, plus a 4 KB block for each file, so a volume of many small files can't fill a disk its bytes alone would fit. A drill whose restore can't be counted or won't fit removes what it made and fails with the reason in its record, and the next drill waits for its scheduled time. A PostgreSQL database loaded from its dump takes more room than its snapshot, which the reserve only partly covers. helios rehearse, helios move test and the copy of each app's data in helios backup helios keep the same reserve. helios restore, a move's restore and a release rollback refuse only a snapshot that can't fit at all, say how much the disk keeps free, and go ahead without a check when restic can't count the snapshot.

Before it copies any app data, helios backup helios adds up every data app's pinned snapshot, as its backup recorded it, and the workload images. It checks the total fits in backups/machine with that reserve, and refuses a pin whose backup finished before the app's active revision. A backup of an app or the machine that fails while it prepares or snapshots removes what it staged. One that is stopped keeps it for its operation to resume: a resumed prepare stages it again from nothing, and a resumed snapshot snapshots the copy it kept, or fails and removes it if the stopped one had begun its upload. A failure after the upload began first stops any restic helper still reading the copy, such as one a killed daemon left running, and keeps the copy, saying so, while it can't. The next backup of that app or the machine replaces anything left there once it has stopped any such helper, and refuses the same way while it can't. A stopped backup's run shows as running in helios backups until the resumed attempt closes it with its own result. When its operation ends without resuming instead, because the plan changed or can't be rebuilt in the meantime, the run is closed as failed with that reason, as is one a killed daemon left. A move's capture records the run it began beside its staging, in moves/<app>/capture.run.json, so its steps, resumed or not, take up that run and never one another backup or capture left open, and its run is closed the same way when its operation ends without resuming.

helios drill <app> --skip-next --reason <text> skips the next scheduled drill when you know it can't run, such as while a disk is being resized. Its plan names the drill it skips and the one after it, and running it again skips the following one too. A skip is not a failed drill: helios backups lists it as skipped by operator with the reason, helios status shows it, and the daemon sends it as a notice. That notice becomes a warning when no drill has passed yet, or when the drill after the skip would find the last good drill more than 14 days old. Backups are never skipped this way; turning them off is a change to helios.yaml.

After restoring, a drill runs SQLite's integrity check on every SQLite database in the restored volumes, up to 64. A database that is corrupt or won't open fails the drill by its path in its volume, and the drill's record lists the sound ones. A database whose indexes sort by a collation its own app defines can't be checked without that app, so the record names it as not checked.

A workload that can't write its own volume is the same problem on the live app. The daemon raises it as an alert, and helios doctor names the files and the chown and restart that give them back. Helios never changes the ownership of files inside a volume on its own.

Restore the app's original secrets from the sealed machine backup before restoring data on a new machine. Helios refuses to generate replacement keys for restored data, which could make encrypted records unreadable.

Important

The pause journal survives daemon restarts: the daemon restores paused containers before accepting work, and a failed restart retains the journal and reports the error. Cron, helios run --stdin, helios exec and helios psql share the backup lock. Direct Docker or SQL writes and external database writers are outside this contract: do not run them during a backup. All app writers must be Helios-managed. Pick backup.schedule for an acceptable maintenance window, and note that an app must not use its own paused object store as its backup destination. Offsite storage remains the recovery target.

Machine recovery and release rehearsal

A new machine backup contains a self-contained catalog, immutable workload images and embedded app data from each app's last successful backup. Keep the source recovery identity, storage password and snapshot inventory separately. helios recover <source-host> --snapshot <id> --identity-file <protected-file> restores onto a clean target with its own identity and validates workloads privately. --rehearsal proves the restore in scratch isolation. Live traffic requires a separate recover cutover. See whole-machine recovery for preparation, reports, retries and abandonment.

helios rehearse ./candidate --snapshot <id> runs candidate release jobs and checks on isolated restored data, retaining a proof of the exact source, manifest, images and snapshot. A manifest can declare release.data_compatibility: backward-compatible or restore-required. Unknown and restore-required releases retain a quiesced checkpoint before changing existing data and block unsafe code-only rollback. rollback --restore-data <exact-checkpoint> deliberately replaces live data and discards writes since that checkpoint. See release safety for the full contract.

Secrets

The credentials Helios itself uses are integrations, kept sealed like any secret: helios integration set storage (Backblaze B2 or any S3 store), helios integration set email (SMTP, for alerts), helios integration set email2 (a second mail server, tried when the first fails), helios integration set pushover or helios integration set ntfy (alerts on a phone; the ntfy topic URL is kept secret, since anyone who knows it can read the topic), helios integration set cloudflare (checked against the API before it is kept), helios integration set loom (a Loom Core's verify endpoint, for routes behind a sign-in) and helios integration set loom_webhook (a Loom task's webhook URL and signing secret, for a thread per critical incident). Setting one again changes only what is answered; a blank answer keeps the current value. helios integration list shows nonsecret fields, which integrations are set and when each was last used. Credential values stay hidden.

An email integration's from can include a display name, such as Helios <helios@example.com>. Alerts keep that name in the message header and use only the address for the SMTP envelope. Each mail server has its own sender setting; changing email leaves email2 unchanged.

Moving a secret between machines

helios secret recipient initializes the machine identity if needed and prints only its public age recipient. Keep /etc/bedrock/secrets.key in your recovery process; automation never prints its private recovery key.

helios secret export core TOKEN runner --recipient age1... emits only ciphertext. Pipe that to helios secret import runner TOKEN on the receiving machine through pinned SSH connections. Transfers expire after ten minutes, are bound to the receiving app and name, and are idempotent. Integration credentials use explicit integration export and integration import transfers. helios move secrets <app> --to <age1...> does the same for every secret an app holds at once, which is what a move needs.

Validation and saved diagnostics

helios check . validates a manifest and its source paths offline, with field locations in errors. --json returns one structured report. helios check . --host adds deploy-plan preconditions for integrations, secret names, and route and port ownership. Deployment verifies builds, runtime readiness, DNS and certificates. Use doctor to inspect current host capacity; aggregate workload capacity is not predicted by the source check. The generated editor schema keeps authoring aligned with the Go parser.

Checks can declare context: application or context: edge. Guarded routes check application content directly and independently verify anonymous denial at the edge. An explicit edge check can use headers_from_secrets, mapping HTTP header names to app secret names, for an authenticated assertion. Credentials are loaded from the deployed revision and stay off logs and receipts; credentialed checks require HTTPS and refuse redirects to another origin.

helios alerts --limit N always includes every open incident plus N resolved incidents. Use --limit 0 for open incidents only. helios history --app api --kind app.deploy --failed --since 24h filters operations before applying the limit. helios history ID --output retrieves saved diagnostic tails, optionally with --step NAME or a one-based step position. Saved tails contain up to 40 lines per execution, rather than a complete log, and become available when an execution finishes.

App workloads and jobs log to systemd's journal, so their output outlives their containers. helios host setup, and helios host reconcile, keep the journal on disk for 14 days, at most 1G, with a rate limit loose enough for a noisy app; helios doctor shows the cap. A container made before Helios logged to the journal keeps its old log driver until it is replaced. The database, object store, backups and edge keep Docker's own logs, five files of 20m.

helios logs <app> [workload] --revision <id> shows a revision's output, from Docker while its containers exist and from the journal once they are removed, even after helios gc has forgotten the revision. --follow takes one workload, and a removed container has nothing more to follow. helios logs <app> --job <id> shows a job run's output, by the ID helios jobs shows. When the journal has nothing for a run, say one older than its 14 days, the 40 lines kept with the run are shown instead. A container removed before Helios logged to the journal took its output with it. Reading the journal takes root or the systemd-journal group. A machine on an older Helios says to upgrade it before --revision, --job, alerts show or alerts review work.

Commands

Machine	target · capabilities · host · daemon · doctor · upgrade · version · status · alerts · watch · peer
Apps	init · check · launch · deploy · preview · rollback · remove · move · ls · ps · logs · history · gc
Data	backup · backups · restore · recover · rehearse · drill · psql
Access	secret · integration · git · exec · run · jobs · cron
Network	dns · edge · exposure

Run helios <command> --help for any of them.

Every operation on the machine, from a deploy or a backup to a host setup or an upgrade, takes --plan to see it first and --digest to apply only what was seen; the settings commands (secret set, integration set, git allow, watch add) apply at once. Between reading a plan and applying it the manifest or the machine can move, so the digest is how an agent, or a person, applies the plan they actually approved:

helios deploy ./app --plan --json | jq -r .digest
helios deploy ./app --digest <hex>

A plan that has changed runs nothing and records nothing. --digest implies --yes, because naming the plan is the approval.

Deploy approvals also bind the source bytes, file modes and safe relative links. Helios captures that source privately before initialization or builds begin, so edits to the original checkout cannot change an operation already in progress. See deployment sources for recovery, cleanup and disk-space behavior.

Traffic from a hostname shared by several apps is omitted from per-app summaries: the edge counters identify the hostname, not which app handled each request. Dedicated-host traffic, workload resource usage and volume usage remain measured. During Docker or state outages, existing incidents remain open until a fresh observation can confirm recovery.

Build

make check   # gofmt, vet, test
make build   # bin/helios for this Mac
make linux   # bin/helios-linux-amd64 for the machines

One static binary, no runtime to install.

Cutting a release

Run tests and the lane proof, commit, then run HELIOS_RELEASE_SIGNING_KEY=<release key .pub> scripts/build-release.sh 0.7.16 /tmp/helios-release-0.7.16 from a clean checkout. It builds static Linux and macOS binaries for amd64 and arm64 plus SHA256SUMS, stamped with the version and exact source commit, and signs the sums into SHA256SUMS.sig. Publish those exact files on the matching GitHub release.

Signing. Releases are signed with an Ed25519 SSH key used for nothing else. Its private half stays in the key holder's SSH agent, so the build script only ever sees the public half. The script signs with ssh-keygen -Y sign -n helios-release, so the signature can't pass for one the same key made for anything else, and checks the result against release/allowed_signers. Without HELIOS_RELEASE_SIGNING_KEY it builds, leaves the sums unsigned and prints the command the key holder runs. An unsigned release is never published.

Verifying. Every build of Helios pins the release keys in internal/release/trust.go. helios upgrade --version and scripts/install-release.sh download SHA256SUMS.sig and refuse a release unless a pinned key signed its sums for helios-release. The check happens before any build is downloaded or run. helios upgrade prints the signer's fingerprint, and the upgrade plan and its receipt name it. To check a release by hand:

ssh-keygen -Y verify -f release/allowed_signers -I helios-release \
  -n helios-release -s SHA256SUMS.sig < SHA256SUMS

A checksum pinned from reviewed source vouches for a build by itself: helios upgrade --version 0.7.15 --sha256 <checksum>, HELIOS_SHA256 for the install script, or a bootstrap profile. That is how a release from before 0.7.16, which publishes no signature, still installs, and the upgrade says it is unsigned.

Rotating. Add the new key to internal/release/trust.go and release/allowed_signers (a test keeps the two equal), and publish that release signed by the old key. Machines that install it trust both keys, so the release after it can be signed by the new key alone. The old key leaves both lists the same way. If a release key may have leaked, stop trusting it: publish a release without it and install that release everywhere with a pinned checksum, since the leaked key could sign anything else.

Consumers pin the version and SHA-256 in reviewed source, verify before execution, and never pipe a downloaded script into a shell. An artifact replacement needs a new reviewed pin; do not overwrite published assets.

Status and docs

0.7 is the first Go release. Helios is the Go rewrite of Ophelia; the Python 0.6 line is archived on the ophelia-0.6 branch and the ophelia-0.6-final tag, and stays there.

Helios stays on 0.7.x. There is no 0.8 or 0.9: a finished feature is a patch release, cut when that feature is done rather than when a group of them is.

A machine already running helios upgrades itself. Helios 0.7.15 and later take the download only when a Helios release key signed the checksums published beside it:

helios upgrade --version 0.7.15

helios --target <name> host bootstrap --profile <file> provisions a machine from a reviewed portable profile. scripts/install-release.sh 0.7.5 <host>... remains available for initial binary installation.

Document	What it covers
docs/direction.html	what Helios is for, where it stands, and what is next
docs/ideas.html	the 37 ideas, and which are tabled
docs/plan.html	the 0.7 plan, delivered and closed
docs/blueprint.html	why the rewrite happened, written under an older name
docs/features/targets-and-bootstrap.md	named machines, portable profiles and integration transfer
docs/features/compatibility.md	API/state contracts and upgrade checkpoints
docs/features/machine-recovery.md	self-contained recovery and rehearsals
docs/features/safe-moves.md	source fencing and deliberate target cutover
docs/features/release-safety.md	release rehearsal and data-aware rollback
docs/features/deployment-sources.md	approved source captures, safe retries and cleanup
docs/features/peer-watch.md	machines that watch each other, and which one pages
docs/operations/incident-review.md	the blank review helios alerts review fills in
docs/technical/performance.md	benchmark workloads, repeated comparisons and optimization results
docs/technical/minio-image-availability.md	open object-service image distribution issue
lane/README.md	the proving ground

The disposable lane reports which operational workflows ran and which need public DNS, external storage or other explicit prerequisites. Unit checks and operational proofs are recorded separately.

Fresh pulls of the pinned MinIO object-service image currently return HTTP 401. The dependency issue records the evidence and required compatibility checks for a replacement.

License

Apache 2.0. See LICENSE and NOTICE.