Skip to content

isb 4.1.0: pin instance addresses on org bridges, net.failed, mattermost channels - #80

Merged
knowsuchagency merged 4 commits into
mainfrom
isb-stop-dhcp-lease-reuse-duplicate-ips-across-instances
Nov 9, 2026
Merged

knowsuchagency merged 4 commits into
mainfrom
isb-stop-dhcp-lease-reuse-duplicate-ips-across-instances

Conversation

@knowsuchagency

@knowsuchagency knowsuchagency commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

From the 2026-11-09 incident: a new instance in org exa (mcp-gateway-1-2ee0) got 10.83.115.43, which analytics-production-analytics-postgres-1-0a5e had held since 10/12; analytics ingestion and web broke from ~09:41Z until the gateway moved. incus' bridges lease for an hour and an OCI container never renews, so dnsmasq re-hands its address.

What changes

  • Pinned at create. isb pins eth0's ipv4.address when it creates an instance in an org project (sandboxes, stack replicas, build sandboxes), on an address no instance on the bridge holds (in state) or is pinned to, and records it in user.isb.ipv4.eth0. A per-process reservation covers creates in flight; after the create the address is checked again and moved before first start if another process took it. (incus accepts two NICs pinned to one address, then refuses to start either, so this check matters.)
  • Checked every minute by isb serve. Pins an unpinned instance at the address it has, in place (verified: same PID, same address); gives a stopped one a free address; adopts pins set by hand, which takes over the live stopgap's 59 overrides. Raises net.failed (matches *.failed, so the jessica channels hear it) when two instances hold one address, when a pinned instance has another address, or when one can't be pinned. It never moves an instance itself.
  • Diff rules. A NIC whose spec names no address is satisfied by isb's pin, and pruning never removes it.
  • Not done: ipv4.dhcp.expiry=infinite. With pins, lease expiry no longer matters. Changing the bridges would restart dnsmasq in every org and keep the leases of deleted instances forever.
  • Egress bridges (isbbrx*, one sandbox each) stay on DHCP.

Also: a mattermost notification provider

The slack provider accepts only hooks.slack.com, and Mattermost rejects the generic webhook JSON with a 400. mattermost posts {"text": ...} (title in bold) to an incoming-webhook URL on any server. It is https only, and every @ gets a zero-width space so event text never pings. Supported in the CLI (--mattermost), the tool, the web UI's channel dialog, and the docs.

Tests

  • Unit: subnet math, claims from incus JSON (including a VM's NIC found by MAC), the plan (incident replay, conflicts, adopt, drift, stopped pins), diff rules.
  • tests/ipam.rs against titan's incusd in a throwaway org (deleted afterwards): pin at create with unique addresses, a pruning plan is a no-op, an instance made behind isb's back is pinned in place without a restart, adopt, a stopped instance gets a free address, a hand-pinned duplicate is reported and not adopted.
  • Provider unit test: body shape, no mentions, https only, the URL never in errors, length cap.
  • scripts/check.sh, web lint and the web tests pass.

https://claude.ai/code/session_01LJ21XsZE86ywhpFMvYVWmE

From the 2026-11-09 incident (a new instance in org exa got the address
analytics-postgres had held since 10/12): incus' bridges lease for an hour
and an OCI container never renews, so dnsmasq re-hands its address.

- isb pins eth0's ipv4.address at create (sandboxes, stack replicas, build
  sandboxes) on an address no instance on the bridge holds or is pinned to,
  and records it in user.isb.ipv4.eth0. A per-process reservation covers
  creates in flight; after the create the address is checked again and
  moved before first start if another process took it (incus accepts two
  NICs pinned to one address, then refuses to start either).
- isb serve checks every org bridge each minute: pins an unpinned instance
  at the address it has, in place (no restart), gives a stopped one a free
  address, and adopts pins set by hand (the live stopgap's). It raises
  net.failed when two instances hold one address, when a pinned instance
  has another, or when one cannot be pinned. It moves no instance itself.
- The diff ignores isb's pin on a NIC whose spec names no address and never
  prunes it. Egress bridges (one sandbox each) stay on DHCP.

Tests: unit tests for the subnet, claims and plan; tests/ipam.rs against a
real incusd in a throwaway org.

Claude-Session: https://claude.ai/code/session_01LJ21XsZE86ywhpFMvYVWmE
…ver)

The slack provider takes only hooks.slack.com, and Mattermost answers the
generic webhook body with 400. `mattermost` posts {"text": ...} (the title
in bold) to an incoming-webhook URL on any server, https only, and puts a
zero-width space after every @ so event text never pings anyone. CLI
--mattermost, the channel_create tool, the web UI's channel dialog and the
docs know it.

Claude-Session: https://claude.ai/code/session_01LJ21XsZE86ywhpFMvYVWmE
@knowsuchagency knowsuchagency changed the title isb 4.1.0: pin instance addresses on org bridges, raise net.failed isb 4.1.0: pin instance addresses on org bridges, net.failed, mattermost channels Nov 9, 2026
@knowsuchagency
knowsuchagency marked this pull request as ready for review November 9, 2026 11:17
@knowsuchagency
knowsuchagency merged commit 7494021 into main Nov 9, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant