Skip to content

feat(library): POST /status for the clients' missing-items pass - #53

Merged
GianniCarlo merged 8 commits into
mainfrom
feat/library-status
Sep 29, 2026
Merged

GianniCarlo merged 8 commits into
mainfrom
feat/library-status

Conversation

@GianniCarlo

@GianniCarlo GianniCarlo commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Why

Both apps find what the server is missing by comparing their local paths with GET /keys (synced keys only).

  • Moves get undone. A path that went stale on the device (the item was moved or renamed on another one) reads as missing. The client re-uploads it at that path, and PUT / treats a known uuid at a new key as a move, so the item goes back. For a folder, its whole subtree goes with it.
  • Two cases aren't covered at all:
    • books imported while a signed-in account's subscription had lapsed;
    • LITE books waiting for their file after an upgrade to PRO.

What

POST /v1/library/status takes { uuids: [...] }, every uuid in the client's local library in one body, and answers { unknown, unsynced }.

  • unknown: no row has the uuid, active or deleted.
    • The client matches these through /uuids first. A legacy row at that key takes its uuid; a conflict makes the client adopt the server's.
    • Then it registers them like an import. Nothing holds the uuid, so nothing can move.
    • A deleted item keeps its uuid, so a book deleted on another device isn't registered again.
  • unsynced: active books with no file in S3. PRO clients upload these through /upload/* by uuid, never by path. LITE ignores the list.
  • Input: uuids come back spelled as they were sent (Postgres lowercases, iOS sends uppercase). Strings that aren't uuids are left out of both lists, so one bad local uuid can't fail the request.
  • Query:
    • the list goes in as one uuid[] parameter, because whereIn would hit Postgres' 65,535-parameter cap;
    • it runs as two queries (active rows, deleted rows) so each uses its partial index;
    • a failed read is a 500 (LibraryLookupError), never an empty answer, which would read as "register everything".
  • Body size: only this route parses a 5 MB body (about 130k uuids). jsonBody keeps body-parser's 100 KB on every other route and leaves this path to the route. The route parses its body only after checkSubscription and requireCloudData, so unauthenticated callers, and subscribers on a tier without cloud access, never get a large body buffered. The path is matched the way Express routes (any case, trailing slash ignored).
  • Access: open to PRO and LITE (requireCloudData), and not audited (it's a read).
  • /keys is deprecated: still served for the shipped builds that use it; new clients use /status.

The client rules are in docs/multipart-uploads.md ("The missing-items pass"). Accepted exception: an item from before March 2026 (before uuids existed) that another device deleted under its own uuid, or none, reads as unknown and comes back.

Deploy

Ship this together with #50/#51, before the iOS build that uses it: that build's first sync depends on /status.

One migration, 20260929120000_library_items_active_not_null: library_items.active was nullable, and a NULL row was an invisible third state (every read filters on active = true).

  • Backfill: NULL rows become false, i.e. deleted, which is what they already were to clients. true would bring them back into users' libraries and could collide with the active-only unique indexes.
  • Lock-safe: it runs outside a transaction, so no step scans the ~1.6M rows while holding an exclusive lock:
    1. add CHECK (active IS NOT NULL) NOT VALID (enforced on new writes);
    2. backfill;
    3. VALIDATE (scans without blocking reads or writes);
    4. SET NOT NULL (proved from the check, no second scan);
    5. drop the check.
  • Lock timeout: each ALTER runs in its own short transaction with SET LOCAL lock_timeout = '5s'. Its brief exclusive lock then gives up instead of queueing behind an open transaction and stalling every query on the table behind it. The migration is safe to re-run.
    • Checked locally: with another session holding a lock, a step fails after about 5 s, and it succeeds once the lock is released.
    • Production runs Postgres 17.9, so SET NOT NULL can skip the second scan.
  • Down drops NOT NULL; the backfilled rows stay false.

Tests

  • LibraryServiceItemsStatus.test.ts:
    • how rows sort into unknown / unsynced / neither, and that deleted rows count as seen;
    • an active row plus deleted rows sharing a uuid;
    • NULL synced, another user's uuid, spelling and duplicates, invalid strings;
    • 70k uuids in one request, and a failed read;
  • LibraryDBItemsByUuids.test.ts: the DB-level uuid filter.
  • libraryItemsActive.test.ts: active defaults to true and rejects NULL. The migration was also run up, down (a NULL row inserted), and up again locally: the row came back false, the column NOT NULL, the check gone.
  • jsonBody.test.ts (supertest): a 1 MB body parses on /status in any spelling and gets a 413 on any other route.
  • The validation schema, plus a controller test (500, and only a count is logged).
  • 462 tests on Postgres 17. The repo's reviewer rubric was run locally until a round came back with no findings (4 rounds).

The apps find what the server is missing by comparing their local paths
with `GET /keys`. A path that went stale on the device (the item was
moved or renamed on another one) reads as missing, and re-uploading it
there moves the item back: `PUT /` treats a known uuid at a new key as a
move, a folder's whole subtree included. `/keys` also can't show books
imported while an account's subscription had lapsed, or LITE books
waiting for their file after an upgrade to PRO.

`POST /v1/library/status` takes every uuid in the client's library in
one body and answers `{ unknown, unsynced }`:
- `unknown`: no row has the uuid, active or deleted. The client registers
  those, after matching them through `/uuids`. Nothing holds the uuid, so
  nothing can move, and a book deleted elsewhere counts as seen.
- `unsynced`: active books with no file in S3. PRO clients upload those
  by uuid, never by path.

Uuids come back spelled as sent, and strings that aren't uuids are left
out. The list goes in as one `uuid[]` parameter (a `whereIn` would hit
Postgres' 65,535-parameter cap), queried as its active and deleted
halves so each uses its partial index. A failed read is a 500, never an
empty answer, which would read as "register everything". The JSON body
limit goes to 5 MB (about 130k uuids). Open to PRO and LITE.

`/keys` is deprecated: still served for the builds that use it.
Comment thread src/server.ts Outdated
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

✅ Claude PR Review — PASS

Adds POST /v1/library/status (the missing-items pass by uuid), a route-scoped 5 MB JSON parser with every other route kept at 100 KB, and a lock-safe migration making library_items.active NOT NULL (NULL becomes false).

  • Authorization: the route has checkSubscription + requireCloudData, and both halves of LibraryDB.getItemsByUuids filter on the caller's user_id (a test covers another user's uuid). The uuid array literal is built only from strings that pass isValidUUID's strict hex regex, and it's bound as a parameter, so there's no injection.
  • Body size: the path match in jsonBody covers Express' case and trailing-slash forms. largeJsonBody runs only after the tier checks. recordSyncOperation reads the body only on finish, and only for audited routes, which don't include /status.
  • Previous finding: the two-snapshot issue is fixed; the lookup is now one UNION ALL statement.
  • Migration: nothing reads a NULL active as live, and the unique partial indexes cover only active = true, so backfilling NULL to false can't collide. Each step runs in its own short transaction with lock_timeout.
  • Minor: there's no per-user rate limit on this heavy (5 MB, ~130k-uuid) route. That's acceptable for subscribed callers on a weekly cadence.

Findings: no findings

Previously raised

Finding Status
src/services/db/LibraryDB.ts:90 (info) ⚠︎ moved ✅ verified fixed in 01ef1f9

Converged: nothing new this round, and every earlier finding is settled.

Model claude-opus-5-5 · run log · 0 new · 0 carried over · 1 verified closed · 0 resolved · advisory (a human should still review). Findings are de-duplicated across pushes; an earlier finding closes only when the verification pass judges it against the current code — fixed, no longer applicable, accepted by a maintainer, or a duplicate of a finding reported on this push.

Scope the 5 MB JSON limit to POST /v1/library/status. The global parser
runs before the rate limiter and auth, so the raised limit let any
request, unauthenticated ones included, make the server buffer and parse
a 5 MB body. `jsonBody` now keeps body-parser's 100 KB everywhere and
leaves this one path to the route, which parses with 5 MB after
`checkSubscription`.
Walk the requested uuids once, filing each as unknown or unsynced, instead of filtering the list twice. An unsynced book is always a seen one, and the else-if says so.
Comment thread src/api/LibraryRouter.ts Outdated
Check the tier before parsing the 5 MB status body: requireCloudData now runs before largeJsonBody, so a subscriber on a tier without cloud access gets the 403 without the server reading a large body first.
Match the own-body route the way Express routes (case-insensitive, trailing slash ignored), so a spelling the route accepts isn't parsed first at 100 KB and then rejected for a large library.
Comment thread src/services/db/LibraryDB.ts
Make library_items.active NOT NULL. Every read filters on active = true, so a NULL row was an invisible third state, which POST /status would have answered as unknown. NULL rows are backfilled as deleted (false), which is what they already were to clients (true could also collide with the active-only unique indexes). The migration runs outside a transaction so nothing scans the table under an exclusive lock: a NOT VALID check (enforced on new writes), the backfill, VALIDATE, then SET NOT NULL proved from the check.
Comment thread src/database/migrations/20260929120000_library_items_active_not_null.ts Outdated
Give each ALTER in the active NOT NULL migration its own short transaction with SET LOCAL lock_timeout = '5s'. Their brief ACCESS EXCLUSIVE locks otherwise wait behind any open transaction on library_items, and every later query on the table queues behind them. A blocked step now fails fast, and the migration is safe to re-run. Production runs Postgres 17.9, so SET NOT NULL is proved from the validated check.
Comment thread src/services/db/LibraryDB.ts Outdated
Read both halves of getItemsByUuids in one UNION ALL statement, so they share one snapshot: as two statements, a row changing state or committing between them showed up in neither and read as unknown. Each branch still uses its partial index.
@GianniCarlo
GianniCarlo merged commit 99e1e43 into main Sep 29, 2026
4 checks passed

This branch was successfully deployed

1 active deployment
reviewer — 01ef1f96 Deployed Sep 29, 2026 by GianniCarlo via review #108
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant