~

.changeset/apply-pnpm-patch-in-vite.md

@@ -0,0 +1,9 @@
+---
+"@inkandswitch/patchwork": patch
+---
+
+The vite plugin applies this repo's pnpm patch of `@automerge/automerge-repo` (`patches/@automerge__automerge-repo@<version>.patch`, shipped in the package as `dist/patches/`) to every copy of automerge-repo vite bundles for a site, in place of the two hand-written edits it carried before. A site installing from npm gets the same automerge-repo as this workspace: the real `detach` behind eviction, the `mesh` adapter role, and whatever else the patch holds.
+
+Covered: the page build's chunks, including the `/packages/@automerge/automerge-repo*.js` import-map chunks that patchwork's own protocol-handler worker imports from; the dev server's pre-bundled deps (an esbuild `onLoad` plugin in `optimizeDeps`); and module workers a site builds through vite (`new Worker(new URL("./x.ts", import.meta.url), { type: "module" })`), which vite bundles in a separate rollup pass with only `worker.plugins` — the `config()` plugin's worker config now sets `worker.plugins` to `[wasm(), patches({ complete: false })]`, so those bundles are patched too. A site that passes `worker: false` and writes its own `worker.plugins` has to add `patches({ complete: false })` to them itself.
+
+`patches()` fails the build if any file the patch edits never reached the bundler; `patches({ complete: false })` skips that check, for a worker bundle that may import none or only some of them. The patch is pinned to one automerge-repo version; a version bump fails the build until the patch is re-made against it.

.changeset/evict-after-handoff.md

@@ -0,0 +1,7 @@
+---
+"@inkandswitch/patchwork-bootloader": patch
+---
+
+The automerge protocol handler worker evicts the documents its Repo loaded once no `automerge:` handoff has been in flight for five seconds. A page load is a burst of handoffs through the same folder documents; they now stay hot across the load and are released after it, instead of living in the SharedWorker for as long as any tab is open. A headless `automerge:<folder>/path` redirect waits up to three seconds for the folder to hold every head a connected Subduction peer has advertised, since a re-found folder comes back from IndexedDB before its sync round lands.
+
+Eviction goes through `Repo.removeFromCache`, which this workspace's pnpm patch of `@automerge/automerge-repo@2.6.0-subduction.48` makes real: `removeFromCache` awaits each source's `detach`, and the Subduction source's `detach` persists unsaved commits, runs one sync round if no peer has them, drops its entry and `heads-changed` listener, and unsubscribes the ephemeral topic, so the document can be collected and a later `find` attaches afresh. The Vite plugin ships and applies this patch for consumers installing `@inkandswitch/patchwork` from npm.

.changeset/heads-announcements.md

@@ -0,0 +1,14 @@
+---
+"@inkandswitch/patchwork": patch
+"@inkandswitch/patchwork-bootloader": minor
+---
+
+Every Repo on the origin — each tab's `createRepo()` and the automerge protocol handler worker's, in both the plain and the keyhive branch — passes `headsChannel`, named `<storagePrefix>-heads`. When a tab persists commits it authored, its Repo posts the document's new heads on that BroadcastChannel; a sibling with the document open reloads it from the shared IndexedDB into its handle, and ignores announcements for documents it does not have open or heads it already knows. The channel also carries what the sync server holds: every tab talks to that server as the same identity, so one tab's `onRemoteHeads` observation is a fact for all of them, and each keeps the newest one per peer rather than the last one to arrive. Only a real observation is announced, never a relay of a relay. Every node keeps the same signer and peer id, its own socket to the sync server and the shared database; what changes is how a write in one tab reaches the others.
+
+The siblings mesh is gone with it: `siblingAdapters()` and the `@inkandswitch/patchwork-bootloader/siblings` export are removed, and no Repo passes `subductionAdapters`. Under one peer id the Subduction core never pushes a commit back to the sending peer's other connections, so the mesh links only added sync rounds.
+
+The option lives in this workspace's pnpm patch of `@automerge/automerge-repo@2.6.0-subduction.48`, which also carries three fixes: a `find` resolves from local storage as soon as shared storage holds the document, before the first server round; a `heads-changed` that saved nothing new (a reload, for instance) opens no server round; a document still initializing hydrates from local storage when a sync round fails while disconnected. A reload whose storage read fails is retried a few times, then logged once and left until the next announcement.
+
+A reload updates the handle, not the Subduction node's resident tree, so a commit that reached a tab only over the channel is one that tab cannot push: its hash is in the set every push is filtered against, and it is not in the tree a round reads. A containment backstop watches for that. When an entry's handle holds heads no peer has been seen holding, and a settling delay passes without a sibling reporting that the server has them, the tab writes those commits to storage a second time — which is what puts them in its tree — and then opens a round that can carry them. Each commit costs at most one such duplicate write, and a tab that is offline still does the write, so the reconnect round finds the tree already correct. The write is owed to the commit, not to the round: the cap on heal rounds gates the rounds alone, and a commit stops counting as stranded only once it has been stored or some peer has been seen holding it. Where every tab is online and pushing its own edits, the sibling's report arrives inside the settling delay and none of this runs.
+
+Consumers installing from npm get the unpatched fork, where `headsChannel` is ignored and tabs meet only through the server, until the fork is republished with these changes and the catalog pin is bumped.

.changeset/in-thread-indexeddb.md

@@ -0,0 +1,6 @@
+---
+"@inkandswitch/patchwork": patch
+"@inkandswitch/patchwork-bootloader": patch
+---
+
+IndexedDB is opened on the node's own thread: `createRepo` and the automerge protocol handler worker use `IndexedDBStorageAdapter` in place of `IndexedDBWorkerStorageAdapter`. Measured in `sites/bench`, the worker adapter bought no main-thread time (same boot, same cold load of 40 documents, 1ms flush latency either way) and cost a dedicated worker per tab, about 30 MB across three. The origin-wide signer is unchanged.

.changeset/lazy-keyhive.md

@@ -0,0 +1,14 @@
+---
+"@inkandswitch/patchwork": patch
+"@inkandswitch/patchwork-bootloader": patch
+"@inkandswitch/patchwork-elements": patch
+"@inkandswitch/patchwork-plugins": patch
+---
+
+`@automerge/automerge-repo-keyhive` is loaded only where keyhive is in use: `createRepo` and the automerge protocol handler worker `import()` it inside their keyhive branch, and `patchwork-elements` and `patchwork-plugins` no longer import it at runtime. Its entry module carries the keyhive wasm as a 3 MB base64 string, so the static imports put a 3.1 MB chunk in every tab's modulepreload list and in the worker whether or not the site enabled keyhive, at 7 to 10 MB of memory per tab, and 8 MB in the protocol-handler worker. The chunk is still emitted under `/packages/` and listed in the import map for tool code. Type imports are unchanged.
+
+`isKeyhiveDoc` in `patchwork-plugins`, and the keyhive access gates in `patchwork-elements`, decide from the document id's bytes: an id shorter than 32 bytes, or one whose bytes 16 through 31 are all zero, is a legacy document. They used to construct a keyhive `DocumentId` and take a throw as legacy, but that constructor is an ed25519 point decode and accepts about half of legacy padded ids, so about half of legacy documents went through `bestAccessForDoc`. This is the check behind ARK's `isUnprotectedDoc`, which it recommends over the deprecated `docIdFromAutomergeUrl`.
+
+When keyhive access to a document changes, `patchwork-elements` looks up the document's handle by its automerge document id before retrying. It used the keyhive `DocumentId` string, which is hex and never matched a handle, so an unavailable handle was never dropped before the retry.
+
+The vite plugin gives the worker chunks an empty module-preload dependency list. Vite wraps a dynamic import in a preload helper that touches `document` when it has dependencies to preload, and a worker has no `document`.

.changeset/one-automerge-wasm.md

@@ -0,0 +1,19 @@
+---
+"@inkandswitch/patchwork-bootloader": patch
+"@inkandswitch/patchwork": patch
+"@inkandswitch/patchwork-filesystem": patch
+"@inkandswitch/patchwork-elements": patch
+"@inkandswitch/patchwork-plugins": patch
+"@inkandswitch/patchwork-providers": patch
+"@inkandswitch/edge-handles": patch
+---
+
+Every tab and worker now runs one automerge wasm instance, streamed from `/automerge.wasm`.
+
+The bare `@automerge/automerge`, `@automerge/automerge-repo`, `@automerge/automerge-subduction` and `@keyhive/keyhive` specifiers resolve to their `/slim` builds everywhere: in the vite plugin's bundle, in the importmap a tool sees at runtime, and in the dev server's worker bundles. The fullfat entries embed and instantiate their own copy of the wasm on import, so a single value import of the bare name (there were four in our own packages) used to cost each tab a second automerge instance and a second, byte-identical `automerge.wasm` download. The `/packages/@automerge/automerge.js` chunk is no longer emitted; the bare name points at `/packages/@automerge/automerge/slim.js`.
+
+`initWasm` in the host and the protocol-handler worker hand the wasm-bindgen init a `Request` instead of buffering the bytes first, so both automerge and subduction go through `WebAssembly.instantiateStreaming`: no 5 MB transient copy, and the compiled module is eligible for Chrome's code cache.
+
+`@inkandswitch/patchwork-bootloader/externals` and `/externals-list` export the alias table as `slim`.
+
+`pnpm lint` (scripts/lint-slim-imports.mts, run in CI) fails on any import of a bare name in the table, type-only ones included, so the fullfat entries stay out of every bundle.

.changeset/skip-recaching-unchanged.md

@@ -0,0 +1,5 @@
+---
+"@inkandswitch/patchwork-bootloader": patch
+---
+
+The service worker no longer re-caches a passthrough response whose etag matches the copy it already holds. Every tab boot used to clone the whole bundle's responses and write them back to Cache Storage, holding a second copy of each body in the service worker's process until the write landed; with several tabs opening at once that peaked at a few hundred MB.

.github/workflows/ci.yml

@@ -32,6 +32,9 @@ jobs:
       - name: install dependencies
         run: pnpm install --frozen-lockfile
 
+      - name: lint
+        run: pnpm lint
+
       - name: build everything
         run: pnpm build
 

core/bootloader/package.json

@@ -26,10 +26,6 @@
       "import": "./dist/externals-list.js",
       "types": "./dist/externals-list.d.ts"
     },
-    "./siblings": {
-      "import": "./dist/siblings.js",
-      "types": "./dist/siblings.d.ts"
-    },
     "./signer": {
       "import": "./dist/signer.js",
       "types": "./dist/signer.d.ts"

core/bootloader/src/automerge-protocol-handler-worker.ts

@@ -3,16 +3,16 @@
 // does.
 //
 // It is a node like any tab's: the same IndexedDB, its own sync-server socket,
-// and the siblings channel to the tabs. Resolving requests is its whole job.
+// and the storage channel to the tabs. Resolving requests is its whole job.
 // When the service worker misses the cache for a request that looks like a URL
 // encoded URL, it broadcasts a HandoffRequestMessage on HANDOFF_CHANNEL; we
 // resolve the automerge URL, write the response into the service worker's
 // cache (keyed by a Request reconstructed to match the one it's holding), and
 // reply on the same channel.
 import { initializeWasm, hasHeads } from "@automerge/automerge/slim";
 // eslint-disable-next-line
-// @ts-ignore — initSync is a wasm-bindgen runtime helper not in the .d.ts
-import { initSync as initSubductionSync } from "@automerge/automerge-subduction/slim";
+// @ts-ignore — the default init is a wasm-bindgen runtime helper not in the .d.ts
+import initSubduction from "@automerge/automerge-subduction/slim";
 
 import {
   Repo,
@@ -21,21 +21,20 @@ import {
   stringifyAutomergeUrl,
   type AutomergeUrl,
   type DocHandle,
+  type DocumentId,
   type PeerId,
+  type StorageId,
 } from "@automerge/automerge-repo/slim";
 import { resolvePath } from "@inkandswitch/patchwork-filesystem";
 
-import { IndexedDBWorkerStorageAdapter } from "@automerge/automerge-repo-storage-indexeddb/IndexedDBWorkerStorageAdapter";
+import { IndexedDBStorageAdapter } from "@automerge/automerge-repo-storage-indexeddb";
 import { WebSocketWorkerClientAdapter } from "@automerge/automerge-repo-network-websocket";
-import {
-  initializeAutomergeRepoKeyhive,
-  initKeyhiveWasm,
-  type AutomergeRepoKeyhive,
-  type SyncServerSelection,
+import type {
+  AutomergeRepoKeyhive,
+  SyncServerSelection,
 } from "@automerge/automerge-repo-keyhive";
 
 import { DEFAULT_CLASSIC_SYNC_SERVER } from "./sync-config.js";
-import { siblingAdapters } from "./siblings.js";
 import { loadOrCreateSigner } from "./signer.js";
 import { keyhiveStorageName, storagePrefix } from "./storage.js";
 import { startWorkerControl } from "./worker-control.js";
@@ -86,12 +85,10 @@ function getRepo(): Promise<Repo> {
 
 async function buildRepo(): Promise<Repo> {
   log("fetching wasm");
-  const [automergeWasm, subductionWasm] = await Promise.all([
-    fetch("/automerge.wasm").then((r) => r.arrayBuffer()),
-    fetch("/subduction.wasm").then((r) => r.arrayBuffer()),
+  await Promise.all([
+    initializeWasm(new Request("/automerge.wasm")),
+    initSubduction(new Request("/subduction.wasm")),
   ]);
-  initSubductionSync(new Uint8Array(subductionWasm));
-  await initializeWasm(new Uint8Array(automergeWasm));
   log("wasm initialized");
 
   const { repo, hive } = syncServer.keyhive
@@ -104,21 +101,23 @@ async function buildRepo(): Promise<Repo> {
 }
 
 async function buildPlainRepo(): Promise<Repo> {
-  const storage = new IndexedDBWorkerStorageAdapter();
+  const storage = new IndexedDBStorageAdapter();
   return new Repo({
     signer: await loadOrCreateSigner(storage),
     storage,
     peerId:
       `${storagePrefix}-resolver-${Math.random().toString(36).slice(2)}` as PeerId,
     subductionWebsocketEndpoints: [syncServer.url],
-    subductionAdapters: siblingAdapters(),
+    headsChannel: `${storagePrefix}-heads`,
     enableRemoteHeadsGossiping: true,
   });
 }
 
 async function buildKeyhiveRepo(
   keyhiveSyncServer: SyncServerSelection
 ): Promise<{ repo: Repo; hive: AutomergeRepoKeyhive }> {
+  const { initKeyhiveWasm, initializeAutomergeRepoKeyhive } =
+    await import("@automerge/automerge-repo-keyhive");
   initKeyhiveWasm();
   const { hive, repo } = await initializeAutomergeRepoKeyhive({
     // ARK injects an `idFactory` deriving document ids from keyhive. A site
@@ -127,7 +126,7 @@ async function buildKeyhiveRepo(
       new Repo(
         syncServer.useIdFactory === false ? config : { ...config, idFactory }
       ),
-    storage: new IndexedDBWorkerStorageAdapter(keyhiveStorageName),
+    storage: new IndexedDBStorageAdapter(keyhiveStorageName),
     peerIdSuffix:
       `${storagePrefix}-resolver` + Math.random().toString(36).slice(2),
     automaticArchiveIngestion: true,
@@ -136,9 +135,9 @@ async function buildKeyhiveRepo(
     // the matching peer id. Omitting it defaults to "subduction".
     syncServer: keyhiveSyncServer,
     repo: {
-      storage: new IndexedDBWorkerStorageAdapter(),
+      storage: new IndexedDBStorageAdapter(),
       subductionWebsocketEndpoints: [syncServer.url],
-      subductionAdapters: siblingAdapters(),
+      headsChannel: `${storagePrefix}-heads`,
       enableRemoteHeadsGossiping: true,
     },
   });
@@ -244,6 +243,42 @@ function waitForHeads(
   });
 }
 
+const CATCH_UP_MS = 3_000;
+
+async function caughtUpWithPeers(
+  repo: Repo,
+  handle: DocHandle<unknown>,
+  signal: AbortSignal
+): Promise<void> {
+  if (!repo.isSubductionConnected()) return;
+  const peers = (await repo.connectedSubductionPeerIds()) as StorageId[];
+  const caughtUp = () => {
+    const states = peers.map((peer) => handle.isCaughtUpWith(peer));
+    return (
+      states.some((state) => state !== undefined) && !states.includes(false)
+    );
+  };
+  if (caughtUp() || signal.aborted) return;
+  await new Promise<void>((resolve) => {
+    const done = () => {
+      clearTimeout(timer);
+      handle.off("remote-heads", check);
+      handle.off("heads-changed", check);
+      signal.removeEventListener("abort", done);
+      resolve();
+    };
+    const check = () => {
+      if (caughtUp()) done();
+    };
+    const timer = setTimeout(done, CATCH_UP_MS);
+    handle.on("remote-heads", check);
+    handle.on("heads-changed", check);
+    signal.addEventListener("abort", done);
+    if (signal.aborted) done();
+    else check();
+  });
+}
+
 /**
  * Thrown instead of returning a Response when the request should fail as a
  * network error rather than resolve to something the caller can memoize.
@@ -270,6 +305,7 @@ async function resolveAutomergeUrl(
   // the headless req
   if (!heads) {
     const folder = await repo.find(maybeAutomergeUrl, { signal });
+    await caughtUpWithPeers(repo, folder, signal);
     const url = stringifyAutomergeUrl({ documentId, heads: folder.heads() });
     const location = `/${encodeURIComponent(url)}${path.length ? `/${path.join("/")}` : ""}`;
     return Response.redirect(location, 307);
@@ -325,8 +361,38 @@ function impatience(limit: number) {
   );
 }
 
+const EVICT_IDLE_MS = 5_000;
+let handoffsInFlight = 0;
+let evictTimer: ReturnType<typeof setTimeout> | undefined;
+
+function handoffStarted() {
+  handoffsInFlight++;
+  clearTimeout(evictTimer);
+}
+
+function handoffFinished() {
+  if (--handoffsInFlight > 0) return;
+  clearTimeout(evictTimer);
+  evictTimer = setTimeout(() => {
+    evictLoaded().catch((error) => console.error("eviction failed", error));
+  }, EVICT_IDLE_MS);
+}
+
+async function evictLoaded() {
+  const repo = await repoPromise?.catch(() => null);
+  if (!repo) return;
+  const ids = Object.keys(repo.handles) as DocumentId[];
+  log(
+    `evicting ${ids.length} document(s) after ${EVICT_IDLE_MS}ms without a handoff`
+  );
+  for (const id of ids) {
+    if (handoffsInFlight > 0) return;
+    await repo.removeFromCache(id);
+  }
+}
+
 async function handleHandoffRequest(message: HandoffRequestMessage) {
-  const { id, cachename, request } = message;
+  const { id, request } = message;
 
   let handoff: URL;
   try {
@@ -350,6 +416,17 @@ async function handleHandoffRequest(message: HandoffRequestMessage) {
     return;
   }
 
+  handoffStarted();
+  try {
+    await respondToHandoff(message, handoff);
+  } finally {
+    handoffFinished();
+  }
+}
+
+async function respondToHandoff(message: HandoffRequestMessage, handoff: URL) {
+  const { id, cachename, request } = message;
+
   let response: Response;
   try {
     log(`resolving handoff ${id} for ${handoff}`);

core/bootloader/src/externals-list.ts

@@ -37,3 +37,10 @@ const externals = [
   "solid-js/jsx-runtime",
 ];
 export default externals;
+
+export const slim: Record<string, string> = {
+  "@automerge/automerge": "@automerge/automerge/slim",
+  "@automerge/automerge-repo": "@automerge/automerge-repo/slim",
+  "@automerge/automerge-subduction": "@automerge/automerge-subduction/slim",
+  "@keyhive/keyhive": "@keyhive/keyhive/slim",
+};

core/bootloader/src/externals.ts

@@ -4,7 +4,7 @@ import { fileURLToPath } from "node:url";
 
 const require = createRequire(import.meta.url);
 
-export { default } from "./externals-list.js";
+export { default, slim } from "./externals-list.js";
 
 /**
  * pretend the import came from inside this package, so node_modules resolution

core/bootloader/src/service-worker.ts

@@ -344,7 +344,10 @@ async function servePassthrough(
       CACHEABLE_STATUSES.includes(result.status) &&
       /^https?:/.test(request.url)
     ) {
-      cacheInBackground(fetchEvent, cache, request, result.clone());
+      const etag = result.headers.get("etag");
+      if (!etag || etag !== cached?.headers.get("etag")) {
+        cacheInBackground(fetchEvent, cache, request, result.clone());
+      }
     } else {
       log(`not caching status ${result.status} for ${request.url}`);
     }