fix(memory): scope LanceDB identities across projects

CHANGELOG.md

@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- **App and project scope identifiers now use a portable lowercase filesystem
+  contract.** Values are limited to 1–128 lowercase ASCII letters, digits,
+  underscore, dot, hyphen, `@`, or `+`. Trailing dots, Windows device names,
+  default-directory aliases, and runtime-owned app names are rejected with HTTP
+  422. Public response IDs and sender/owner validation remain unchanged.
+  Existing roots containing nonconforming scope names require coordinated
+  migration of both source directories and retained scoped SQLite state before
+  upgrade; `cascade rebuild` does not perform that name migration.
+- **LanceDB row identity is now scoped by app, project, and owner.** Existing
+  roots with conforming scope names require an offline
+  `everos cascade rebuild --yes`. A storage-generation marker, lifecycle locks,
+  and serialized bootstrap keep legacy or incomplete projections fail-closed.
+
+### Fixed
+
+- Prevented owner-local memory identifiers in separate projects from
+  overwriting each other in LanceDB. The repair covers episodes, atomic facts,
+  foresights, agent cases, agent skills, and user profiles while retaining the
+  historical public response identifiers.
+
 ## [1.2.3] - 2026-08-07
 
 **Background maintenance that fails loudly instead of quietly.** A soak run on

QUICKSTART.md

@@ -235,8 +235,11 @@ Your extracted memory is a normal Markdown file under the memory root:
     └── lancedb/
 ```
 
-Markdown is canonical; SQLite and LanceDB are derived indexes. You can read,
-edit, diff, and version the memory files without a database client.
+Markdown is canonical for extracted memory, and LanceDB is its rebuildable
+search projection. SQLite also contains buffered and coordination state that is
+not reconstructable from markdown. You can read, edit, diff, and version the
+memory files without a database client, but should use `cascade rebuild` rather
+than deleting `.index` manually.
 
 ## Upgrade capabilities when you need them
 

docs/api.md

@@ -138,7 +138,7 @@ storage. This is the same rule users see when reading rendered output:
 
 #### Other conventions
 
-- **Server-generated IDs** follow `<owner>_<kind>_<YYYYMMDD>_<NNN>`,
+- **Public response IDs** follow `<owner>_<kind>_<YYYYMMDD>_<NNN>`,
   e.g. `alice_ep_20260528_00000001` for an episode, `alice_af_...`
   for an atomic fact. See
   [storage_layout.md §4](storage_layout.md) for the encoding.
@@ -156,6 +156,20 @@ for the agent track). The default scope materialises on disk as
 added only for the literal id `"default"` so the default space stays
 visually distinct from user-named scopes).
 
+Scope identifiers are lowercase because they are used as raw directory
+segments. Rejecting case variants prevents two logical scopes from resolving
+to the same directory on case-insensitive filesystems.
+
+This is a stricter contract than earlier releases. Before upgrading an existing
+root, audit both its scope directories and retained SQLite state for uppercase,
+reserved, trailing-dot, or device-name scopes. If any exist, do not start the
+new binary or run `cascade rebuild` yet. This release has no general supported
+in-place scope-name migration. Use a fresh-root markdown import or a separately
+reviewed, deployment-specific migration that updates paths and retained SQLite
+state together. Rebuild only reconstructs LanceDB; it does not rewrite retained
+SQLite scope values. See the
+[cascade runbook](cascade_runbook.md#legacy-nonconforming-scope-names).
+
 A `/search` or `/get` query never crosses scopes — different
 `(app_id, project_id)` pairs are isolated.
 
@@ -166,8 +180,14 @@ Both fields share the same validation:
 | Type | `string` |
 | Default | `"default"` |
 | Length | 1–128 chars |
-| Charset | `^[a-zA-Z0-9_.-]+$` |
-| Rejected literals | `"."` and `".."` (path-traversal guard) |
+| Charset | `^[a-z0-9_.@+-]+$` |
+| Rejected literals | `"."`, `".."`, trailing dots, Windows device names, and reserved storage/configuration names |
+
+For `app_id`, the reserved storage/configuration names are `.index`, `.tmp`,
+`.lock`, `.projection.lock`, `default_app`, `everos.toml`, and `ome.toml`.
+For `project_id`, `default_project` is reserved. These restrictions apply only
+to app and project scopes; sender and owner identifiers retain their existing
+mixed-case validation contract.
 
 ## Errors
 

docs/architecture.md

@@ -203,8 +203,13 @@ Three-piece observability:
     └── knowledge/                                      # global shared knowledge
 ```
 
-System-managed entries (`.index/`, `.tmp/`) and `ome.toml` live directly
-under the memory root.
+System-managed entries (`.index/`, `.tmp/`, `.lock`, and `.projection.lock`)
+and the `everos.toml` / `ome.toml` configuration files live directly under
+the memory root.
+
+`app_id` and `project_id` use a lowercase portable filesystem grammar because
+they are stored as raw path segments. Runtime/configuration names are reserved
+at the app level so user memory cannot overlap system-managed root entries.
 Full tree + frontmatter chassis: [storage_layout.md](storage_layout.md) and
 [how-memory-works.md](how-memory-works.md). Frontmatter has 4-tier field
 protection (L1 read-only / L2 system / L3 business / L4 user).

docs/cascade_runbook.md

@@ -131,34 +131,129 @@ everos cascade rebuild          # prompts for confirmation
 everos cascade rebuild --yes    # non-interactive
 ```
 
-> **Stop the `everos server` first.** Unlike `cascade sync`, rebuild
-> **drops and recreates** the LanceDB tables. A running daemon holds
-> cached table handles that would keep pointing at (and writing to) the
-> dropped dataset, corrupting the rebuild. This is the one cascade
-> command that is **not** safe to run alongside a live server.
+> **Stop the `everos server` and other mutating cascade commands first.**
+> Unlike `cascade sync`, rebuild **drops and recreates** the LanceDB tables.
+> A running daemon or CLI writer holds cached table handles that would keep
+> pointing at the dropped dataset. Current servers and `sync`, `fix --apply`,
+> and `backfill` hold the shared projection lifecycle lock for their complete
+> LanceDB lifetime. Rebuild requires the exclusive side plus the legacy OME
+> lock, and refuses to start while any such process is active. During an
+> upgrade, do not launch an older binary concurrently with rebuild: an old
+> process does not participate in the projection-lock protocol until its OME
+> guard has been acquired.
+>
+> After a memory root has been rebuilt for storage generation 2, do not run an
+> older EverOS binary against that root at any time. Older binaries ignore the
+> generation marker and can write legacy row IDs while it still says
+> `READY(2)`. If that happens, stop every EverOS process and run
+> `everos cascade rebuild --yes` with the current binary before restarting.
 
 What it does, in order:
 
-1. **Drops** every business LanceDB table (`drop_business_tables`) and
-   evicts them from the connection cache.
-2. **Recreates** them empty from the current schema + FTS indexes
-   (`ensure_business_indexes`).
-3. **Clears** the cascade queue (`md_change_state.reset_all`) so every
-   md file re-enqueues as `added` on the next scan.
-4. **Re-scans + drains** (`sync_once`): re-embeds and re-inserts every
-   md entry.
+1. **Establishes the exclusion precondition** described above. If either the
+   projection lifecycle lock or legacy OME lock is not free, rebuild exits
+   before changing the marker, queue, or LanceDB tables.
+2. **Atomically publishes `REBUILDING` for storage generation 2** before
+   opening the stores or performing any destructive work. This invalidates
+   a previously valid `READY` marker before a crash can expose a partial
+   rebuild.
+3. **Clears** the cascade queue (`md_change_state.reset_all`) so every md
+   file re-enqueues as `added` on the next scan.
+4. **Drops** every business LanceDB table (`drop_business_tables`) and
+   evicts it from the connection cache.
+5. **Recreates** the tables from the current schema and builds the FTS
+   indexes (`ensure_business_indexes`).
+6. **Re-scans and drains** (`sync_once`, followed by additional drains
+   while work remains): re-embeds and re-inserts every md entry.
+7. **Requires a clean completion state**: `pending == 0`,
+   `failed_retryable == 0`, and `failed_permanent == 0`. If any count is
+   nonzero, rebuild fails and leaves the marker as `REBUILDING`.
+8. **Atomically publishes `READY(2)`** only after all completion checks
+   pass. Only then does the command print `rebuild complete`.
+
+If the process crashes, is interrupted, or encounters an error after
+`REBUILDING` has been published but before `READY(2)` is published, the marker
+remains `REBUILDING`. The API server and mutating cascade commands fail closed
+rather than serving or changing a partial index. Correct the underlying error
+and rerun `everos cascade rebuild` with the server stopped; the rebuild
+recovery path deliberately does not require a `READY` marker.
+
+### Storage-generation gate on startup
+
+Generation 2 changes the value-level identity stored in LanceDB row primary
+keys. A schema check cannot distinguish generation-1 keys from generation-2
+keys, so startup validates `.index/lancedb/.storage_identity.json` before it
+opens LanceDB or runs schema/index migrations.
+
+Startup and mutating cascade commands acquire the shared projection lifecycle
+lock before checking the marker and keep it until their LanceDB handles close.
+They accept only `READY(2)` and refuse to proceed when the marker is:
+
+- missing on a memory root that contains source markdown or LanceDB artifacts;
+- malformed or contains unknown fields or JSON types;
+- `READY` for any generation other than 2; or
+- `REBUILDING`, including after an interrupted or failed rebuild.
+
+A marker-less root is initialized directly as `READY(2)` only when its
+extracted-memory source and projection are empty: it has neither source
+markdown nor LanceDB artifacts. Retained SQLite state is not proof of a fresh
+root, so existing installations must still perform the scope audit below. A
+bootstrap lock serializes first-time marker, schema, and index creation across
+API and mutating CLI processes. For every blocked state above, stop the server
+and rerun `everos cascade rebuild`. The read-only `cascade status` and
+`cascade fix` listing remain available through SQLite so operators can inspect
+the queue without opening LanceDB.
+
+Scope directories must use the lowercase portable grammar documented in
+[`api.md`](api.md#scopeid-app_id-and-project_id). Existing roots with only
+conforming scope names can proceed directly to the generation-2 rebuild.
+Nonconforming roots require the separate migration described below.
 
 It deliberately **skips `verify_business_schemas`** — the drift it
 recovers from would otherwise trip that guard on startup before the
 rebuild could run (chicken-and-egg).
 
 Why not a bare `rm`:
 
-| Recovery | Re-populates `done` entries | Preserves `unprocessed_buffer` |
+| Recovery | Current result | Preserves `unprocessed_buffer` |
 |---|---|---|
-| `rm -rf .index/lancedb` | ❌ scanner skips `done` rows → empty index | ✅ |
-| `rm -rf .index` | ✅ | ❌ deletes un-extracted messages |
-| `everos cascade rebuild` | ✅ | ✅ |
+| Remove `.index/lancedb` | Unsupported; existing-root startup fails the generation gate | Yes |
+| Remove `.index` | Unsupported; startup fails the generation gate and SQLite-only state is lost | No |
+| `everos cascade rebuild` | Rebuilds all source markdown and publishes `READY(2)` | Yes |
+
+### Legacy nonconforming scope names
+
+Earlier releases accepted scope names that are unsafe or ambiguous on common
+filesystems, including uppercase letters, trailing dots, Windows device names,
+and names now reserved for runtime state such as `.index` and `.tmp`.
+
+Before upgrading an existing root, inspect both the source tree and retained
+SQLite data. If any `app_id` or `project_id` is nonconforming:
+
+1. Stop every EverOS process and take a complete backup of the memory root.
+2. Do not run the new server or `cascade rebuild` against that root yet.
+3. Do not hand-edit only the directory names or only the database. This release
+   does not provide a general supported in-place scope-name migration.
+4. Either import the extracted markdown into a fresh root with unique accepted
+   scope names, or use a separately reviewed, deployment-specific migration
+   that updates the business-scope directories and every matching `app_id`,
+   `project_id`, stored markdown-path reference, and serialized scope value in
+   all SQLite databases as one transactionally planned maintenance operation.
+   This includes scope values embedded in OME `run_record.event_payload` JSON,
+   not only ordinary columns in `system.db`.
+5. After that migration completes, run `everos cascade rebuild --yes` with the
+   current binary, then restart.
+
+`cascade rebuild` resets the cascade queue and reconstructs LanceDB, but it
+intentionally preserves `unprocessed_buffer`, `memcell`,
+`conversation_status`, knowledge, cluster, and reflection state in SQLite. It
+therefore cannot perform the scope-name migration by itself. There is no
+automatic migration for these legacy names in this release.
+
+For a legacy app literally named `.index` or `.tmp`, do not rename the whole
+directory: those paths also contain runtime-owned state. Prefer a fresh-root
+import; any attempt to separate business content from those directories needs
+its own reviewed migration rather than an ad hoc filesystem move.
 
 ## Recovery paths
 
@@ -186,14 +281,13 @@ and it would detonate later inside `merge_insert` as an opaque
 `LanceError(IO): Spill has sent an error` (EverOS #337). The type check
 turns that into this clean startup error.
 
-Recover with **`everos cascade rebuild`** (documented above). Do **not** just
-`rm -rf ~/.everos/.index/lancedb`: that clears the vectors but leaves
-`md_change_state` marked `done`, so the scanner skips every already-
-indexed file and the index comes back **empty**. And do **not**
-`rm -rf ~/.everos/.index`: that also deletes `unprocessed_buffer`
-(messages received but not yet extracted — not rebuildable from md).
-`cascade rebuild` is correct on both counts. Markdown is the source of
-truth, so no memory content is lost.
+Recover with **`everos cascade rebuild`** (documented above). Do **not** remove
+`~/.everos/.index/lancedb`: that also removes the storage-generation marker,
+so current startup fails closed when the root contains source markdown or
+projection artifacts. Do **not** remove `~/.everos/.index`: that additionally
+deletes `unprocessed_buffer` messages that have not yet become markdown.
+`cascade rebuild` preserves that SQLite-only state, rebuilds every source
+entry, and publishes `READY(2)` only after completion checks pass.
 
 ### inotify watch-limit exhaustion (Linux)
 

docs/how-memory-works.md

@@ -24,19 +24,22 @@ This is the narrative companion to the reference docs: see
 ## The storage stack
 
 Three embedded pieces, each owning what it is best at. Markdown is the
-**source of truth**; the other two are **derived and rebuildable**.
+**source of truth for extracted memory**. LanceDB is rebuildable; SQLite also
+contains operational state that markdown does not contain.
 
 | Layer | Backed by | Holds | Rebuildable? |
 |---|---|---|---|
 | **Markdown + YAML frontmatter** | plain `.md` files | the memory content itself — the only portable, human-editable asset | — (it *is* the truth) |
-| **SQLite** (`aiosqlite`) | `.index/sqlite/*.db` | system state, audit log, the cascade queue, the boundary buffer, OME engine state | ✅ from markdown |
+| **SQLite** (`aiosqlite`) | `.index/sqlite/*.db` | system state, audit log, the cascade queue, the boundary buffer, OME engine state | No, not completely |
 | **LanceDB** (Arrow) | `.index/lancedb/*.lance` | vector + BM25 + scalar columns for retrieval | ✅ from markdown |
 
 !!! note "The one rule that follows from this"
-    Delete the entire `.index/` directory and **no memory is lost** — it
-    rebuilds from the `.md` tree. There is no separate "export"; the
-    markdown *is* the export. (How to trigger a rebuild:
-    [Operating it](#operating-it).)
+    Markdown is the source of truth for extracted memory, but `.index/` also
+    contains SQLite-only state such as unprocessed buffered messages. Never
+    delete `.index/` or `.index/lancedb` manually. Use
+    `everos cascade rebuild`, which preserves SQLite state and rebuilds the
+    LanceDB projection from markdown. There is no separate markdown export;
+    the markdown tree is the extracted-memory record.
 
 ## Storage paths
 
@@ -45,11 +48,17 @@ The default memory root is **`~/.everos/`** (override with
 inside the memory root as `everos.toml` (generated by `everos init`).
 
 Memory is partitioned by **`<app_id>/<project_id>`** *before* the
-user-visible directories, so different `(app, project)` spaces never share
-a directory or cross in search. The reserved id `"default"` materialises as
+user-visible directories. In the managed layout, without operator-created
+symlink aliases, different `(app, project)` spaces do not share a directory or
+cross in search. The reserved id `"default"` materialises as
 `default_app` / `default_project` on disk (so a default space stays
 visually distinct from a user-named one).
 
+App and project identifiers use lowercase portable filesystem names. This
+prevents case variants from collapsing onto one directory on macOS or Windows.
+Reserved runtime/configuration names are rejected for the app component; see
+the [ScopeId contract](api.md#scopeid-app_id-and-project_id).
+
 ```
 ~/.everos/                                  ← memory root (EVEROS_ROOT)
 ├── default_app/                            ← <app_id>   ("default" → default_app)
@@ -71,7 +80,7 @@ visually distinct from a user-named one).
 │       │           └── skill_<name>/SKILL.md        (+ references/ scripts/)
 │       └── knowledge/                      ← shared / global
 │
-├── .index/                                 ← system-managed, rebuildable (gitignore)
+├── .index/                                 ← system-managed runtime state (gitignore; do not delete)
 │   ├── sqlite/
 │   │   ├── system.db                       state / audit / cascade queue (md_change_state) / buffer / LSN
 │   │   ├── ome.db                          Offline Memory Engine state
@@ -304,9 +313,10 @@ The CLI ([cli.md](cli.md)) is intentionally small:
       the whole index, run `everos cascade rebuild` — it drops the
       LanceDB tables and re-indexes from md, re-populating even entries
       the queue already marked `done` and preserving un-extracted
-      buffered messages. (A bare `rm -rf <memory-root>/.index/lancedb`
-      is **not** enough: the cascade queue still shows those files
-      `done`, so the scanner skips them and the index comes back empty.)
+      buffered messages. Do not remove `.index/lancedb` manually: current
+      startup fails closed because that also removes the required storage
+      generation marker. Do not remove `.index`, which additionally deletes
+      SQLite-only state that markdown cannot reconstruct.
       For an incremental catch-up, use `everos cascade sync`.
     - **Flush** is an HTTP endpoint (`POST /api/v2/memory/flush`), not a
       CLI command — it forces *extraction* of the session buffer, which is

docs/knowledge.md

@@ -47,9 +47,10 @@ Each level corresponds to a different granularity of API:
 
 ## Storage layout
 
-Every document is a self-contained directory. Markdown files are the
-single source of truth; SQLite and LanceDB are derived indexes built
-automatically by the cascade daemon.
+Every document is a self-contained directory. Markdown files are the source of
+truth for extracted document content, and the cascade daemon builds its search
+projection automatically. SQLite also holds metadata and broader operational
+state, so the `.index` tree is not disposable as a whole.
 
 ```
 ~/.everos/<app>/<project>/knowledge/