[phernandez]

Why

Basic Memory Cloud is adding a fair-use limit for legacy plans (basic-memory-cloud#2974): past a daily chunk budget, a workspace's notes are still saved and full-text indexed, but their embeddings wait for the next UTC day. Two things were missing in core:

1. Clients cannot tell how current semantic search is. An edited note keeps its earlier ready vector chunks, which look the same as current ones, so nothing in the schema showed that a note's embeddings lag its content.
2. Cloud cannot tell our CLI and MCP server from unknown scripts. Every client that calls Cloud sent httpx's default python-httpx/x.

What changed

semantic_index_current_as_of on search responses

• New SearchResponse.semantic_index_current_as_of: datetime | None. Semantic (vector) search reflects every note in the searched projects changed before this time. The value is the oldest last-modified time (entity.updated_at) among markdown notes whose embeddings are not current. It is null when every note is current, when semantic search is unavailable (the keyword-only host fallback, [BUG] Fatal crash on startup when python.org Python 3.12 is present alongside Homebrew on macOS #711), or when the scope is empty.
• New column entity.embedded_checksum (migration 0044): the checksum whose embeddings a vector-sync pass finished. A note is behind when this column is NULL (never embedded), differs from checksum (edited since), or the last pass left the note part-way (vector_sync_deferred_at).
  • SearchRepositoryBase._sync_entity_vectors_internal, the one boundary every sync entry point goes through, reads checksums before the pass reads search rows and stamps them on the entities the pass finished. A write that lands during a pass leaves the note behind (the safe direction) rather than stamping content the pass never saw.
  • SearchService stamps notes that opted out with embed: false. They owe no embeddings, so they never hold the freshness value back.
  • The backfill treats every entity not left part-way as current. If it set NULL instead, notes that owe nothing (opted out, or with no text) would read as behind until rewritten. A note that really was behind at upgrade reads as current until its pending pass runs.
• A partial index ix_entity_project_embedding_not_current (project_id, updated_at) WHERE <not current> holds only the notes that are behind, so the lookup is one indexed MIN per search, with no per-result work. The predicate is one shared SQL constant, and both the index and the query use it, because SQLite only uses a partial index when the query repeats its predicate. A test asserts the SQLite plan uses it.
• Populated by the project search route, the scoped (multi-project) search route, and the MCP all-projects merge (least current database wins). It passes through SearchClient unchanged.
• bm_search text output adds Semantic results current as of YYYY-MM-DD HH:MM UTC. when the value is set. JSON output carries the field. Output is unchanged when everything is current. Documented in bm-search(3).
• Read cache: vector-chunk changes already invalidate content-wide reads (ReadCacheChange.search_index), so cached search responses do not keep a stale value.

CLI User-Agent

• basic_memory/user_agent.py defines BASIC_MEMORY_USER_AGENT = "basic-memory/<version> (cli)". It is set as a client default on _cloud_client (the /proxy client behind get_client, used by project sync and WebDAV upload), get_cloud_control_plane_client (WebDAV transfer and upload), cli/commands/cloud/api_client.py, and CLIAuth. A per-request User-Agent from a caller still wins, because httpx merges request headers over client headers.
• There is no existing way to tell the MCP server from the CLI in-process, so both send (cli). rclone (push/pull) is an external binary and is unchanged.

Schema / deploy

• Alembic 0044_entity_embedded_checksum: adds a nullable column, runs one UPDATE backfill, and creates a partial index. It runs on SQLite and Postgres. Cloud picks it up with tenant schema migrations at deploy.

Testing

• just fast-check: clean (ruff and ty).
• New real-database tests on SQLite and Postgres (testcontainers):
  • tests/repository/test_semantic_index_freshness.py: all current gives None; an edited note with its pass held back gives its modified time (older current notes do not count) and returns to None after the pass; a never-embedded note; a part-way (deferred) note; an opted-out note; a stamp older than the checksum; non-markdown files; semantic unavailable gives None; the SQLite plan uses the partial index.
  • tests/api/v2/test_search_semantic_freshness.py: field on the project and scoped routes; null without semantic search.
  • tests/mcp/test_tool_bm_search.py: no note when current; JSON field and text line for one project and for all_projects.
  • tests/test_entity_embedded_checksum_migration.py: backfill, index, downgrade (no duplicated generated columns).
  • tests/test_cloud_user_agent.py: header on the wire (httpx MockTransport) for the proxy, control-plane, API, and auth clients; a caller's header wins.
• Load-bearing checks: removing the stamp fails the freshness tests, and removing the header fails the User-Agent tests.
• Wider SQLite sweep (tests/services tests/indexing tests/mcp tests/cli tests/index tests/repository tests/api/v2 test-int/mcp): all pass. Postgres: freshness, readiness, deferred-resume, bm_search, API, and migration tests pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_016N9c6ouiT4Wid3arGR41Dv

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-05T00:01:24.149847Z	26984ab	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d44bf1933f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Clear prior deferral markers when stamping opted-out notes

If a previously deferred or failed note is changed to embed: false, stamping its checksum here does not make it current: the new freshness predicate also treats any non-null vector_sync_deferred_at as behind. _clear_entity_vectors() only deletes vector rows, and both opt-out paths bypass the repository sync boundary that clears deferral markers. Consequently, subsequent syncs and reindex passes keep reporting this opted-out note as behind indefinitely, pinning freshness for the entire searched scope even after all eligible notes finish embedding. Clear the obsolete deferral marker when completing opt-out cleanup, or exclude opted-out notes from the freshness calculation.

AGENTS.md reference: AGENTS.md:L185-L187

Useful? React with 👍 / 👎.

[phernandez]

Fixed in 26984ab. Both opt-out paths now go through SearchService._settle_opted_out_entities. It clears vector_sync_deferred_at through record_entity_vector_deferrals(completed_entity_ids=...) and then stamps the checksum. Regression test test_opting_out_a_part_way_note_settles_it (deferred note, then embed: false, then freshness is None) fails without the fix and passes with it on SQLite and Postgres.