Skip to content

feat(chat): sync group rosters on device and search them locally - #915

Closed
bmc08gt wants to merge 7 commits into
mainfrom
feat/roster-search
Closed

bmc08gt wants to merge 7 commits into
mainfrom
feat/roster-search

Conversation

@bmc08gt

@bmc08gt bmc08gt commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator

The @mention picker needs to search a group's members as the user types, and a group can be larger than one GetRoster page. This keeps each opened group's full roster on device with a token index, so search is a local index range scan with no new RPC. Android is building the same behaviour to the same spec. There's no picker UI in this PR.

Sync (RosterSync)

  • Opening a group chat syncs its roster in the background when it has never had a full read, or when the roster version has moved past the stored watermark. Nothing syncs at launch.
  • A full read pages 100 at a time (up from 50) and stops at 20 pages (2,000 members); search uses whatever is held.
  • Catch-up reads from the first page, which lists the newest joins first, and stops at the first member at or below the watermark. That usually takes one page. If the device then holds more members than member_count, someone left unseen, and a full read runs in the background to reconcile. The pending flag is persisted, so a killed app resumes it on the next open.
  • Stream joins and leaves apply per member by Member.version. A leave keeps the row as a tombstone, so a page that trails the stream can't bring the member back.
  • Members missing from a full read are dropped only when the read completed, every page reported the same roster version, and the stored version hasn't moved past it.
  • Opening the picker re-reads the first page, since profile changes don't bump the roster version. Storing a fetched profile re-tokenizes that user in every group.

Search (RosterSearchSource, local implementation only)

  • Tokens are the words of the display name plus the username. Both the tokens and the query are folded the same way: NFKD, lowercase, then combining marks removed. "eri" finds "Érica".
  • Every query word has to be a prefix of some token. The range upper bound is q + U+10FFFF, so a name with an emoji right after the prefix still matches.
  • Ranking: speakers among the chat's newest 50 held messages come first, newest first. An exact username match on a one-word query comes next. The rest sort by folded name, then raw name, then user id.
  • The signed-in user and blocked users are excluded. An empty query returns only recent speakers.
  • Callers depend on the protocol, so a server-side search can replace or back up the local one later.

Schema

  • Three new tables: roster_member, roster_token (plus an index) and roster_sync. schemaVersion stays at 43. createTablesIfNeeded() already creates missing tables with IF NOT EXISTS on every open, so existing stores gain them empty and keep their data. Each group then does a full read the first time it opens.
  • The version gate moves into Database.discardStoreIfOutdated so it can be tested. A store recorded at or above the code's version is kept, and a later bump still deletes the whole file.
  • hard-rules.md and CLAUDE.md now say a change that only adds new tables, which start empty and fill from the server, doesn't bump. Column changes to existing tables still bump.

ChatBubbleViewTests "Raising with a shape casts the lift shadow along that path" fails in the full FlipcashTests run on main as well, so it's unrelated to this change.

The mention picker needs to search a group's members as the user types,
and the only roster held today is the few members chat metadata embeds.
Add three tables, bump the schema to 44:

- roster_member (conversationId, userId): the full roster. A leave keeps
  the row with isMember false and the leave's version, so a GetRoster page
  that trails the stream can't bring the member back.
- roster_token (conversationId, token, userId): one row per normalized
  display-name word plus the username. The primary key leads with
  (conversationId, token), so a prefix lookup is a range scan.
- roster_sync: per-group version, member count, last full sync, cap and
  gap state. A group with no row is untracked.

RosterSearchText holds the normalization Android mirrors: lowercase,
NFKD, drop general category M. The prefix upper bound is prefix + U+10FFFF
rather than U+FFFF, because SQLite compares UTF-8 bytes and a name
followed by an emoji sorts above prefix + U+FFFF.

FlipClient.getRoster paged to the end in one call; replace it with
getRosterPage so the caller can cap the loop, and raise the page size
from 50 to 100.
RosterSync pages Chat.GetRoster into the store when a group is opened and
it has never been fully synced, holds fewer members than member_count, or
the stream skipped a roster version. It stops after 20 pages (2,000
members). Groups are not synced at launch; stream roster updates are
written only for groups already opened.

LocalRosterSearch sits behind RosterSearchSource so a server search can
replace it later. A member matches when every query word prefixes one of
their tokens. Ranking: senders of the chat's newest 200 held messages,
most recent first; then an exact username match; then by normalized
display name. The signed-in user is excluded, and an empty query returns
recent speakers. prepare(chatID:) refreshes the first roster page, since
profile edits don't bump the roster version.
A missed roster update used to force a full re-read. GetRoster pages
newest joins first and each Member carries the roster version of its
join, so a catch-up now reads from the first page and stops at the first
member at or below the stored watermark; usually one page. Leaves don't
show up that way, so the page's member_count is checked after the merge.
If more members are held than counted, a reconcile is recorded and runs
as a background full read; the persisted flag resumes it on the next open
if the app dies first.

A full read drops a missing member only when it reached the end, every
page reported the same roster version, and the held row is not newer.
A failed page records nothing.

Search now matches Android: NFKD, then lowercase, then strip Mn; split on
whitespace and Z; recent speakers come from the newest 50 held messages;
names tie-break on folded then raw display name, then user id. Blocked
users are excluded, and a cached profile re-tokenizes that user in every
tracked group, since profile edits don't move the roster version.
Android found two leave gaps in its roster store. Neither exists here:
refreshRosterProfile touches only rows with isMember set and never
writes version or isMember, and applyRosterUpdates skips a change at or
below the held version before it touches the row or its tokens. These
tests hold both in place, and add a code-point sort case that doesn't
depend on fullwidth letters, which NFKD folds to ASCII.
The 43 -> 44 bump deleted every store on upgrade. The three roster
tables are new and start empty, and createTablesIfNeeded() already
creates them with IF NOT EXISTS on every open, so a v43 store gains
them in place. Each group has no roster_sync row yet and does a full
read the first time it opens.

The version check moves into Database.discardStoreIfOutdated so its
keep/delete decision is testable: a store at or above schemaVersion
is kept, and a later bump still deletes the whole file, roster
tables included.
createTablesIfNeeded() runs on every open with IF NOT EXISTS, so a new
table that starts empty and fills from the server reaches existing
stores without deleting them. Changes to an existing table's columns
still bump.
@bmc08gt bmc08gt self-assigned this Sep 30, 2026
@bmc08gt
bmc08gt marked this pull request as draft October 1, 2026 18:35
@bmc08gt

bmc08gt commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator Author

Parked: Chat.GetRoster is staying private for now, so roster sync would fail on every group chat. The mention picker landed without it in #942, using Chat.GetMentionSuggestions. This can come back as a second RosterSearchSource implementation once the roster is available.

@bmc08gt bmc08gt closed this Oct 2, 2026
@bmc08gt
bmc08gt deleted the feat/roster-search branch October 5, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant