feat(maintenance): backfill stored connector documents into the memory tree

[YellowSnnowmann]

Summary

Adds MemoryMaintenance::backfill_connector_trees — re-files already-stored connector documents into the memory tree.

tinyhumansai/openhuman#6007 (this repo's #134) fixed the routing: connector items now reach mem_tree_chunks as they sync. It could not fix the records already on disk, and neither can a re-sync — the per-item sync gate treats an ingested document as done, so it fetches nothing and creates no tree rows. On the profile that reported the bug that is ~3000 documents, fully embedded in the namespace store and invisible to tree recall, the memory graph, and the source row's ingest status. Recovery today means removing and re-adding the account, which re-fetches everything at the provider's expense.

Tracked as tinyhumansai/openhuman#6012.

Shape

The walk lives in tinymemory-core::backfill, next to the funnel the sync path uses, and the provider is a thin conversion. That placement is the point: a backfilled row and a freshly-synced row have to be the same row, and they only stay that way while one function writes both. The alternative — reconstructing {toolkit}:{connection_id}:{item_id} at a second call site — is precisely what caused #6007.

Namespaces are rebuilt from the source registry, not parsed out of list_namespaces(). The registry is what the writers used (accept_source_items composes source:{toolkit}:{connection_id} from the same row), so reconstructing it the same way cannot drift; parsing a namespace string back into halves has to guess where the toolkit ends.

Legacy skill-{toolkit} namespaces are the awkward half. store_skill_sync takes an _integration_id it never persists, so pre-migration documents record no connection at all — and the tree scope needs one. Where the registry holds exactly one connection for the toolkit there is only one possible answer and it is used; where it holds several, those documents are skipped and the reason is reported. A wrong attribution in a memory system is worse than a missing one.

One change outside the new module

ingest_connector_item_into_tree returned (), which cannot distinguish "I just treed this" from "the tree already had it" — and that distinction is the whole difference between a backfill that reports progress and one that reports nothing. It now returns Option<IngestResult>: None is the skip contract (no resolvable scope), Some(result) carries already_ingested.

Both existing callers drop the payload, so their behaviour is unchanged. The funnel's own test now asserts None on a blank scope rather than discarding the value — a skip that started answering Some would otherwise be silently counted as work done.

Properties worth knowing

• Idempotent by construction, not by bookkeeping. The ingest pipeline answers already_ingested when its transaction persists nothing, so a second pass writes nothing and an interrupted pass loses nothing. There is no watermark to keep, which is why limit bounds cost rather than carrying a cursor — resuming is just calling again.
• Expensive, and therefore explicit. list_documents carries no content column, so each document needs its own read, and each ingest embeds its chunks. dry_run reports what a pass would examine while writing nothing — the honest way to show the size of the job before paying for it. Nothing calls this automatically: firing it on upgrade would quietly spend a user's embedding budget, which is its own bug (openhuman#5324).
• Four counters, not one. "Did nothing" has three different causes a caller must tell apart — the tree already held everything, nothing could be addressed, or there was nothing to look at. Collapsing them would make an unresolvable account read exactly like a fully backfilled one.
• Corruption still aborts. Per-document failures are tolerated and counted, reusing corruption::escalate_or_count; a corrupt store fails every later document identically, so continuing would burn the whole limit reproducing one error (openhuman#5820).
• The null driver refuses, matching every other mutating Maintenance member — "backfilled nothing" must not read as work done by a driver that stores nothing.

Test plan

• cargo fmt --all -- --check
• cargo clippy --all-targets --all-features -- -D warnings
• cargo test --all-features

Five new cases in crates/tinymemory-core/src/backfill_tests.rs:

• a stored document reaches the tree at exactly gmail:conn-1:msg-1 with path_scope gmail:conn-1, having first asserted the tree is empty — that precondition is #6007;
• a second pass reports already_present: 1, writes nothing, and adds no chunks;
• targets are rebuilt from the registry, including the legacy namespace when it is unambiguous;
• the legacy namespace is skipped, by name, when a toolkit has several connections, while both current namespaces stay addressable;
• a dry run counts without writing, and a bounded pass reports more_pending rather than reading as complete.

Release note for the host

This adds a contract member, so it needs a minor bump rather than a patch — the host cannot call backfill_connector_trees until a release carries it and modules::registry is re-pinned. Calling it against an older pinned artifact answers Unsupported, which is strictly worse than the feature being absent, so the openhuman side (module forwarder, guard family, RPC surface) is deliberately not opened until this lands and ships.

No new capability family — this is a member of the existing Maintenance family, so Capability::ALL is unchanged and the host's capability-count assertion is unaffected. METHODS moves 142 → 143.

Refs tinyhumansai/openhuman#6012, tinyhumansai/openhuman#6007

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 5d4bb7a5-0906-4faf-a76f-9e74eadb5bf4

Warning

Your free Security trial is over. An organization admin can upgrade to Advanced for continuous pull request security review or dismiss this notice.

---

_{Comment @coderabbitai help to get the list of available commands.}

[tinysweeper]

How this change flows

0 changed behaviours across 5 relationships. 4 surrounding behaviours are shown (60 graph nodes walked). 38 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["resolve_targets"]:::impacted
  n1["...e_registry_including_the_legacy_namespace"]:::impacted
  n2["...ed_when_a_toolkit_has_several_connections"]:::impacted
  n3["Result"]:::impacted
  n0 -->|uses| n3
  n1 -->|calls| n0
  n1 -->|tests| n0
  n2 -->|calls| n0
  n2 -->|tests| n0
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff

Loading

_{Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.}

[tinysweeper]

tinysweeper found nothing blocking. Approving.

             $0.0443 · 370,350 in / 11,044 out · 86,272 cached (23%) · openrouter/openai/text-embedding-3-small, deepseek/deepseek-v4-flash, z-ai/glm-5.2 · 756 embedded
critique:    $0.0172 · 170,644 in / 5,023 out  · 17,837 cached (10%) · deepseek/deepseek-v4-flash, z-ai/glm-5.2
security:    $0.0238 · 160,750 in / 5,812 out  · 68,435 cached (43%) · deepseek/deepseek-v4-flash, z-ai/glm-5.2
tests:       $0.0020 · 23,519 in  / 151 out    · 0 cached (0%)       · deepseek/deepseek-v4-flash
description: $0.0013 · 15,437 in  / 58 out     · 0 cached (0%)       · deepseek/deepseek-v4-flash