feat(doctor): amem doctor report and sync cadence without a daemon (Refs #40 F15, F16)

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- `amem doctor [--json]` and the sync cadence (agent-mem#40 F15, F16, AC8, task #424): one read-only report of duplicates and stale notes, pending candidates, capture freshness and hook registration per CLI, the brief's token cost, index equals rebuild, reader leases, the policy in effect and the ingest and sync cadence, with a stable `amem.doctor/v1` JSON shape pinned by a snapshot. `amem sync --if-due [--force]` syncs only when an hourly schedule and a failure backoff (15 minutes doubling to a day) allow it, only with a configured remote, bounded at 30 seconds, and every sync stamps `sync-state.json` (times and a count, never an error text). amem registers no sync hook and no resident process (owner decision O3). Doctor writes nothing into the home: every check runs on a scratch copy of the logs (the live index is only snapshotted read-only), and `sessionStartBrief` gains `measureOnly` for it. `sync --if-due` also stops git's transport children, stamps only real transport failures (also under `--json`) and ignores a future stamp. See `docs/contracts.md`.
+
 - Lexical recall gate (agent-mem#40 F14, AC7, task #423): the F14 candidates (query expansion from `about`-node words, 1-hop graph neighbours by append and by score blend, an `about`-node boost, pseudo-relevance feedback) were each scored with `amem eval` on the built-in set and on a new held-out set (`test/fixtures/eval-heldout.json`, written before any candidate ran). None raised MRR or hit@1 on both, so none is adopted and `store.search` is unchanged. Added the gate (`scripts/f14-gate.ts`, the pure `adopts()` rule, `test/f14-gate.test.ts` pinning the record) so a future candidate is judged the same way, and an optional `rank` on `evalSeeded` to score one. Recorded in `docs/contracts.md`.
 
 - Recall evaluation (agent-mem#40 F13, AC7, task #421): `amem eval [--set <file>] [--k 1,5,10] [--json]` replays a question set against the ranked `store.search` (what MCP and hub recall use, not the `amem search` substring filter) and reports recall@k, hit@k and MRR, with an `answerable` split, scored over the entries the store holds, that separates ranking from coverage. No model judge, no network, no embedding. The built-in synthetic fixture (`src/memory/eval-set.json`, 26 invented entries, long and short tier a day apart, and 32 questions) runs in a throwaway store and never touches the memory home; its scores (MRR 0.9063, hit@10 0.9375, recall@1 0.8281) are pinned in `test/eval.test.ts`, so a lexical change (F14) has to move them on purpose.

README.md

@@ -278,6 +278,12 @@ A private admission cursor rotates bounded inspection across pending records (at
 
 Successful native project identities have a private, bounded derived cache (128 cwd entries, 4 KiB each). A hit must match the current project scope, stricter Git-validity metadata and executable revision; a miss still uses the shared hardened Git resolver. It stores no capture or context, and does not guess an identity when Git fails. The hook uses the selected developer Git directly only when PATH selects Apple's root-owned shim; custom PATH binaries remain selected. Cache ceilings and invalidation are in [the P3 contracts](docs/contracts.md#hook-project-identity-cache).
 
+## Doctor and the sync cadence
+
+`amem doctor` prints one read-only report of this project's memory: duplicate and stale notes, pending candidates, capture freshness and hook registration for each CLI (a CLI with no hook is `manual`), the SessionStart brief's token cost, whether the index equals a rebuild of the logs, reader leases and the policy in effect. `amem doctor --json` is a stable machine-readable form (`amem.doctor/v1`). It writes nothing into the memory home (every check runs on a temporary copy of the logs) and creates nothing in a home you never used.
+
+Intake runs at the end of each Codex, Kimi and Pi session once `amem setup <cli> --apply` has registered it. Sync is off unless you ask for it: amem registers no sync hook and runs nothing between sessions. To sync on your own schedule, call `amem sync --if-due` from your shell (for example an EXIT trap in `~/.zshrc`; it needs a terminal). It syncs at most hourly, backs off 15 minutes doubling to a day after failures, stops after 30 seconds, and does nothing without a remote set by `amem sync --remote <private git url>`. `--force` skips the schedule and the backoff.
+
 ## Native transcript intake
 
 `amem ingest` reads existing Codex, Kimi or Pi transcripts and writes screened deterministic capture rows, never curated notes. Run it from the project to bind intake to that working directory; the CLI exposes no project/home override or policy bypass:

docs/contracts.md

@@ -355,3 +355,36 @@ agent-mem#40 F14 asks for query expansion from aliases and tags and for BM25 plu
 - **AC4, no candidate expires unseen.** `candidates.db` gains a nullable `shown_at` column, added in place when a `CandidateStore` opens, inside one immediate transaction that rechecks the column (several processes open the store at once, every `amem mcp` among them; and the switch to WAL is retried for a bounded time on lock contention). An older database keeps its rows, all unshown; a reader without the column treats every row as unshown and the stamp is a no-op, which only keeps candidates longer. A pending candidate is stamped only by `amem review walk`, which refuses to run unless stdout is a terminal (a redirect to a file or a pipe would stamp candidates nobody saw), and only AFTER the candidate's header and body were written to it (a write that fails stamps nothing and asks nothing, so a candidate the person could not read is never counted as shown). The SessionStart hook, the brief and `amem review list` never stamp: the hook cannot know the person read the context, and `list` prints an id, a kind and a date (no content), is run by scripts and cron, and `--json` may be piped to a program, so a stamp there would start the 7-day clock with nobody reading. The first value is kept. The 30-day age expiry deletes a decided (accepted or rejected) candidate, or a pending one shown at least 7 days ago; an unshown pending candidate is kept however old. The 500-per-project cap keeps rows in three tiers, unshown pending first, then shown pending (owed its grace), then decided, so it evicts decided rows before a shown one and shown ones before an unseen one. `put` prunes, counts, inserts and prunes again in ONE immediate transaction, so concurrent proposers cannot all pass the count and then let the cap evict an unseen candidate; when 500 unseen ones wait it throws `CandidateQueueFull` (`code: "candidate_queue_full"`, no candidate text, worded for an agent: ask the person to review), which the MCP server maps to one fixed sentence instead of its generic refusal. Re-proposing an existing candidate is not new. The existing retention test is amended to this behavior.
 - **`amem status`** adds `candidates: { count, oldestExpiresInDays, dueWithin7Days, unshown }` and a `warning` when any candidate is within 7 days of its 30-day age, or `candidates: { unavailable }` when `candidates.db` exists but cannot be read (the reader tries a plain read-only open, then an immutable one for a WAL database whose `-wal` and `-shm` are gone, and each form is proven with a query; the brief line fails open to no line). **`amem review walk`** writes each pending candidate (oldest first) to the terminal, stamps it, and only then asks accept, edit, reject, skip or quit, writing every outcome as it happens so an interrupted walk still shows what it did: accept runs the ordinary `review accept` (current policy, host shard checks), reject the ordinary `review reject`, edit opens the text (a `fail` candidate, and a general memory candidate proposed as a fail note, as the labelled template; a loose sentence is pre-filled under `Approach:`, while text that already carries any of the five labels opens exactly as written, so the edit cannot repeat a label), writes it through `note add`'s path as the person's own long-term note and rejects the candidate, skip leaves it pending, quit or end of input stops. Candidate text is printed with control and format characters flattened and bounded at 600 characters. A bare `amem review` still lists, so existing scripts hold.
 
+
+## Doctor and ingest/sync cadence (F15, F16, AC8)
+
+**`amem doctor [--json]`** is one report over the memory home, person-only like every other command (the agent-marker refusal runs first, so an agent shell never opens the home). Its sections, in this fixed order, each carry `status` `ok`, `attention` or `unavailable` (with a `reason`: a section that cannot be checked says so, never a silent zero), and the report carries the worst of them:
+
+| section | what it says |
+| --- | --- |
+| `duplicates` | notes of this project whose kind, title and text are identical after case and whitespace folding (ids only, never text), plus the imported-duplicate groups `amem dedupe` previews |
+| `stale` | notes whose every `path:` reference is gone from the repository (needs doctor run inside this project's repository, else `skipped`) |
+| `candidates` | the pending summary `amem status` shows; `attention` when one is within 7 days of expiry. Doctor never stamps a candidate as shown |
+| `capture` | per CLI (`claude`, `codex`, `kimi`, `pi`): `hook` `registered` (a setup receipt in the home's `setup/` directory, phase `applied`, naming that CLI) or `manual`; `capture` `fresh`, `stale` (older than 7 days) or `never`; the last capture time and row count. A registered hook that is not fresh, and any CLI that went stale, is `attention`; a CLI with no hook that never captured is reported, not flagged |
+| `brief` | the SessionStart brief's real size for this project (`tokens`, `budget`, coverage), `attention` when partial or over 90% of the budget |
+| `index` | the live index, snapshotted and brought level with the logs, equals a full rebuild of the same logs (both in the scratch copy). A curated split or generation home is `skipped` (its build already proves equality) |
+| `readers` | reader leases of live processes, of processes that are gone, and unreadable ones. Doctor takes none itself |
+| `policy` | the configuration in effect: which files, deny-path and PII-pattern counts, the byte budgets |
+| `cadence` | per CLI how intake runs (`hooks`, `session end` or `manual`) and the sync schedule (below) |
+
+**What "read-only" means.** Doctor writes nothing into the memory home: no file is created, changed, checkpointed or removed there, and it takes no reader lease. `test/doctor.test.ts` hashes every file of a populated home before and after the FIRST run (no warm-up), `-wal` files included; it normalises two kinds of SQLite bookkeeping and nothing else: a `-shm` file (wal-index scratch memory, which any read-only open changes) and an EMPTY `-wal` file. On Linux a database closed cleanly leaves neither, and a read-only open of it (the snapshot of the live index, the candidate and capture reads) creates them again, empty; macOS leaves them in place (Ubuntu CI showed this). A `-wal` holding data, every database, log, lock and sidecar are compared byte for byte, and the test first removes both so macOS takes the same path as Linux. A home nobody used stays absent, and so does a project the home has not seen. How:
+
+- Every check that needs a store runs on a private scratch copy in a temporary directory, removed afterwards: the project's (and its aliases') curated logs, the lifecycle and alias logs, and the project's work logs in a base of their own. The first version opened the live store through `curatedStore()`; a review probe showed that rewrites `index.db` and `search.db`, checkpoints both WALs and writes a reader lease, and that the brief's work list recreated its lock file and re-ran its migration.
+- The live shared-v1 index is only snapshotted (a read-only connection and `VACUUM INTO`), brought level with the copied logs the way the next reader would, and compared with a full rebuild of the same logs. A curated split or generation home builds and seals its index when it is created, so that section is `skipped` there (`amem rebuild` repeats the proof).
+- The brief is measured with `sessionStartBrief({ measureOnly })`, used by doctor alone: it takes no reader lease, reads the curated rows from the scratch store and the work list from the scratch work base (a work log with no index is still counted, which the first version under-reported), and opens this machine's capture database read-only and only when it exists. Without a scratch work base the work section is `unavailable`, never `complete`. It measures the brief the SessionStart worker builds; the native parent's deferred brief is smaller and is not what is measured. The hook path does not set `measureOnly`.
+- Duplicates and stale notes read at most the newest 100 000 notes of the project; a project with more says `truncated`. More than 64 MiB of logs is `unavailable`, not a partial answer.
+
+`skipped` means nothing to check here, with the reason (no curated notes yet, not run inside the repository, a split home); it does not degrade the overall status. `unavailable` means a check could not run.
+
+**`--json` is stable (AC8).** `amem.doctor/v1`. `test/doctor-snapshot.test.ts` pins every key, the type of every value and the fixed vocabulary against `test/fixtures/doctor-snapshot.json`, for a populated and for an unused home. Adding a key or changing a type or a word changes the fixture, so a reviewer sees it.
+
+**F16, cadence without a daemon.**
+
+- *Ingest.* Already done by `amem setup` (#417): the SessionEnd hook of Codex and Kimi, and Pi's `session_shutdown` extension, run `amem ingest <cli>` within the hook budget (3 s, 2.5 s for Pi) as a short-lived process with no watcher. Claude captures through its seven bounded hooks. Doctor reads the setup receipts and reports a CLI with no registered hook as `manual`, with the command to register it. "Registered" is evidence from the receipt (`hookEvidence: "setup receipt"`): a hook the person removed by hand afterwards still shows as registered until `amem setup <cli> --apply` or `--undo` runs again.
+- *Sync (owner decision O3: off by default).* amem registers **no** sync hook and starts **no** resident process. `amem sync --if-due` is the opt-in entry the person calls from their own shell (for example an EXIT trap); it needs a terminal like every mutating command, so an unattended scheduler is not a caller. **A CLI's SessionEnd hook cannot run it, deliberately (claude's F16 ruling, final):** a hook has no terminal and may carry an agent marker, and the person-only gate on a network and git write is not weakened for one flag. The person who wants sync at the end of a session puts the command in their own shell, not in a hook amem registers. It does nothing without a configured remote (no `.git` is created), nothing when it is not due, and it stops a running git step after 30 seconds: git runs in its own process group for that call, so the stop reaches the transport children git started (ssh, a credential helper) and not git's pid alone. Because that group no longer receives what the terminal sends to amem, amem turns Ctrl-C (SIGINT), SIGTERM, SIGHUP (the terminal closed, which is also how an EXIT trap's shell goes away) and SIGQUIT into the stop (`on`, not `once`, so a second Ctrl-C is not the default action that would end amem before git), and a process that exits for any other reason signals every live git group first (an exit handler, installed when the first deadline git starts). Tested with the real binary under a pty for each of the four signals, three Ctrl-Cs in a row and a group SIGHUP, and with a process that dies by an uncaught throw. The one thing nobody can handle is SIGKILL of amem itself (or a power loss): the step timer dies with amem, so a git already running is left with nothing to stop it and runs until it finishes or its transport gives up. The next sync takes over `sync.lock` once amem's pid is gone and removes `index.lock`, which can then land under that live git. This is the existing hazard of any killed sync, and `--if-due` does not widen it; it is not closed here (`ponytail`: a group-aware stale-lock check in `sync.ts`, if a killed sync is ever seen to corrupt the index). The 30 seconds bound the git steps. It does not bound the `gh` visibility lookup (its own 15 s limit) or the cleanup `git merge --abort` after an interrupted merge, which deliberately ignore the stop so the repository is left clean (up to 60 s). After a success the next attempt is due an hour later; after a failure the wait is 15 minutes and doubles with each further failure up to a day. `--force` (only with `--if-due`) ignores the schedule and the backoff. The only state is `sync-state.json` beside the logs: two times and a failure count, never an error text (a git error can carry a remote URL), mode 0600, written by every `amem sync`, plain or scheduled. A damaged or hand-edited stamp, and one dated in the future, read as no history. Only a transport failure counts: the real result is read (also under `--json`, which returns a failed sync instead of throwing), and a refusal no remote ever saw (another sync or mutation holds the lock, no remote is configured, the policy declines a note, the remote is public, a usage error) is not stamped, so it neither backs the schedule off nor makes a contended run look like a failed one. The read-modify-write of the stamp is not locked: two syncs finishing together can lose one count, which at worst delays a backoff by one step (`ponytail`: a lock if a scheduler ever runs several at once). Doctor reports the remote as `configured` or `none` (never the URL; read from the active profile's own repository: the shard cache of a split home, `generations/<G>/.git` of a generation home, `.git` of shared-v1), the last attempt and success, the failure count and the next attempt, and is `attention` when a remote has had no success for 7 days or has failed 3 times in a row.
+- *Option A, rejected.* A detached bounded `amem sync` child started from the SessionEnd hook would hand network and git work to a hook. `AGENTS.md` forbids a model or network call on the capture path and a resident process, and `test/hooks-network.test.ts` runs the executable hooks, the SessionStart brief, native ingest and the capture append under network tripwires and observes no attempt, so a sync wired into them would have to change that contract. A detached child outlives the hook's own budget and cannot be accounted to it. The choice stays with the person: a hook is not asked to do it. `test/sync-cadence.test.ts` greps the setup, hook and ingest sources for any sync wiring and the cadence module for any process or timer.