fix(sources): route connector items into the memory tree

[YellowSnnowmann]

Summary

MemorySourceSink::accept_source_items wrote namespace documents and vector chunks but never mem_tree_chunks. A Composio sync therefore landed thousands of documents and embeddings while every tree-backed surface — tree recall, the Memory Tree graph, and the source row's ingest status — still reported zero. None of those read memory_docs.

Reported downstream as tinyhumansai/openhuman#6007, from a profile showing 2120 source_gmail* docs and 7986 vector chunks against 0 Gmail rows in mem_tree_chunks, next to a pre-migration profile with 908.

Root cause

This is the #5473 reconnect, regressed. #5473 added tree ingest to HostSyncAdapter::ingest_document_into_memory_tree, on the SkillDocSink path. The connector migration then built a second ingest path — accept_source_items in tinymemory-tinycortex — which never learned the tree half. LocalDocumentSink (folder sync) still went through ingest_document_with_scope, which is why local sources kept working and only connector sources looked empty.

So the fix is not a second call to the same rules — it is one copy of them:

• ingest_connector_item_into_tree owns the {toolkit}:{connection_id} tree scope, the {…}:{item_id} per-item key, and the blank-half guard. That key pair is what OpenHuman's source_id_prefix counts a Composio source's ingest by ("{toolkit}:{connection_id}:"), so a drift there empties the source status and the memory graph together — silently, because the documents and vectors are still written.
• ingest_connector_item_tolerated owns the failure policy, which is why corruption::escalate_or_count stays pub(crate): callers get the policy, not the primitive, so a new call site cannot quietly implement a weaker one. A tree failure is counted and warned, never fatal — propagating lets one deterministically-poisonous item stall a whole connection and re-fetch the page (real Composio spend) on every retry. Corruption still aborts the run (openhuman#5820).

HostSyncAdapter now delegates, so both paths share the rules. The pre-existing #5473 test is unchanged and still green, which is what makes that refactor behavior-neutral.

The gate

Tree ingest is gated on source_kind == "composio" and a non-empty {toolkit}:{connection_id} split. Both halves, deliberately:

• The kind gate is load-bearing. OpenHuman derives a toolkit-shaped ingest prefix only for SourceKind::Composio and keys every other kind mem_src:{id}:. Treeing a non-Composio source would write rows no status or graph surface can count — the same invisible-write bug, wearing a different source kind.
• The shape gate is the existing ":conn" guard: a blank half yields a scope with no platform prefix, which no retrieval kind resolves.

forget_source had to learn the same key

Not in the original report, but it ships here because this change is what makes it reachable. forget_source deleted chunks with delete_chunks_by_source, which matches a source id exactly — it never matches {source_id}:{item_id}. Once items are treed per-item, a user who disconnected Gmail would keep their documents cleared and every synced message still retrievable in the tree.

The prefix sweep is unconditional on purpose: it matches nothing for a source that never had per-item rows, and a shape gate here could drift out of step with the write gate, which is the direction that leaks. The existing bare-id delete_orphaned_source_tree still lands, because the items' shared path_scope is the bare source id.

Test plan

• cargo test -p tinymemory-core --lib engine::sync — 15/15, including the untouched #5473 guard and a new test for the funnel's blank-connection_id half (a blank toolkit was already covered through the sink path; no sink path can produce a blank connection, but accept_source_items can).
• cargo test -p tinymemory-tinycortex --test full_provider_conformance — 33/33.
  • The existing non-connector source ("drive-1"/"drive") now asserts zero tree rows. That assertion is the gate test.
  • A new connector case asserts the exact per-item id gmail:conn-1:msg-1 and path_scope gmail:conn-1, then asserts forget_source takes those rows with it.
• cargo fmt --all --check

Notes for the reviewer

• This needs a release to reach users. OpenHuman only dev-depends on tinymemory-tinycortex; the engine ships inside the prebuilt TinyBus module, pinned at 1.13.7. The host-side re-pin is a separate openhuman PR gated on a patch release cut from this.
• Connector ingest now embeds twice — once into vector_chunks, once into mem_tree_chunks. On the reported profile that is ~8000 additional chunk embeddings for one account, which touches the embedding budget openhuman#5324 added warnings for. Pre-existing on the old sink path, but the blast radius grows; worth measuring on a real Gmail sync.
• Existing affected users need a re-sync, and the per-item sync gate means already-synced messages may not be re-fetched — recovery may require removing and re-adding the source. A backfill that reads existing memory_docs namespaces and creates the tree rows is deliberately not in scope here and will be filed separately.

Refs tinyhumansai/openhuman#6007

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 8f91b122-cada-48b0-8083-10d178f503cf

Warning

Your free Security trial is over. An organization admin can upgrade to Advanced for continuous pull request security review or dismiss this notice.

---

_{Comment @coderabbitai help to get the list of available commands.}

[tinysweeper]

Requesting changes: 1 lane(s) blocking, worst finding is high.

Fix or reply to the findings below and push. The next review clears this automatically once they are gone — you should not need to dismiss anything by hand.

             $0.0195 · 173,062 in / 3,962 out · 18,996 cached (11%) · openrouter/openai/text-embedding-3-small, deepseek/deepseek-v4-flash, z-ai/glm-5.2 · 703 embedded
critique:    $0.0096 · 73,860 in  / 2,579 out · 11,044 cached (15%) · deepseek/deepseek-v4-flash, z-ai/glm-5.2
security:    $0.0069 · 66,174 in  / 522 out   · 7,952 cached (12%)  · deepseek/deepseek-v4-flash, z-ai/glm-5.2
tests:       $0.0018 · 20,398 in  / 376 out   · 0 cached (0%)       · deepseek/deepseek-v4-flash
description: $0.0012 · 12,630 in  / 485 out   · 0 cached (0%)       · deepseek/deepseek-v4-flash

[tinysweeper]

Move tree_item capture after the move of item fields into input

tree_scope.map(|_| …) captures title and item.content by value at this point, even when tree_scope is None and the closure is never called — because clone() is called eagerly on item.content (not inside the closure body) to produce the tuple, and title on line 2210 was already bound from item.title into a local (also a move). The effect is that item.content and item.title are consumed (moved) before the NamespaceDocumentInput construction on lines 2219–2233 that also references item.title (title is swallowed by the earlier binding) and item.content. This will fail to compile because the fields are used after partial move. The fix is to compute the clone inside the closure, so that fields are only borrowed (not moved) when tree_scope is None.

[RULE] use-after-move ·

[YellowSnnowmann]

Not a use-after-move — this compiles, and the premise of the finding is inverted.

The finding states the clones are "called eagerly on item.content (not inside the closure body) to produce the tuple". They are inside the closure body. The whole tuple expression is the closure body:

tree_scope.map(|_| (item.item_id.clone(), title.clone(), item.content.clone()))

|_| (…) has the tuple as its body, so nothing in it evaluates unless map calls it. Because the body only calls .clone(), the closure captures item and title by shared reference, not by value; Option::map takes the closure by value and either invokes it or drops it, so the borrow ends when map returns. item.item_id, item.content and title are then free to move into NamespaceDocumentInput on the next statement. That is also exactly the property the comment above the line claims ("only a tree-scoped source pays the clones") — for None, map never runs the body and no clone happens.

Verified on this SHA rather than argued: cargo check --all-targets is clean, and cargo test -p tinymemory-tinycortex --test full_provider_conformance passes 33/33 — including the two new cases that exercise both the Some and None sides of this exact map.

The suggested fix ("compute the clone inside the closure") is what the code already does, so there is no change to make here.

[tinysweeper]

How this change flows

4 changed behaviours across 24 relationships. 6 surrounding behaviours are shown (60 graph nodes walked). 38 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["HostSyncAdapter<br/>changed"]:::changed
  n1["...pe_item_is_skipped_for_memory_tree_ingest<br/>changed"]:::changed
  n2["..._is_tolerated_and_skill_store_is_retained<br/>changed"]:::changed
  n3["...nd_tool_rule_state_transitions_round_trip<br/>changed"]:::changed
  n4["from_workspace_dir"]:::impacted
  n5["join"]:::impacted
  n6["with_config"]:::impacted
  n7["memory_fixture"]:::impacted
  n8["...nal_reader_lists_and_reads_a_local_folder"]:::impacted
  n9["assert"]:::impacted
  n1 -->|uses| n0
  n1 -->|calls| n4
  n1 -->|tests| n4
  n1 -->|calls| n5
  n1 -->|tests| n5
  n1 -->|calls| n6
  n1 -->|tests| n6
  n2 -->|uses| n0
  n2 -->|calls| n4
  n2 -->|tests| n4
  n2 -->|calls| n5
  n2 -->|tests| n5
  n2 -->|calls| n6
  n2 -->|tests| n6
  n2 -->|calls| n9
  n3 -->|calls| n9
  n7 -->|calls| n4
  n7 -->|calls| n5
  n8 -->|uses| n0
  n8 -->|calls| n4
  n8 -->|tests| n4
  n8 -->|calls| n5
  n8 -->|tests| n5
  n8 -->|calls| n6
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff

Loading

_{Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.}