feat(library): POST /status for the clients' missing-items pass - #53
Merged
Merged
Conversation
The apps find what the server is missing by comparing their local paths
with `GET /keys`. A path that went stale on the device (the item was
moved or renamed on another one) reads as missing, and re-uploading it
there moves the item back: `PUT /` treats a known uuid at a new key as a
move, a folder's whole subtree included. `/keys` also can't show books
imported while an account's subscription had lapsed, or LITE books
waiting for their file after an upgrade to PRO.
`POST /v1/library/status` takes every uuid in the client's library in
one body and answers `{ unknown, unsynced }`:
- `unknown`: no row has the uuid, active or deleted. The client registers
those, after matching them through `/uuids`. Nothing holds the uuid, so
nothing can move, and a book deleted elsewhere counts as seen.
- `unsynced`: active books with no file in S3. PRO clients upload those
by uuid, never by path.
Uuids come back spelled as sent, and strings that aren't uuids are left
out. The list goes in as one `uuid[]` parameter (a `whereIn` would hit
Postgres' 65,535-parameter cap), queried as its active and deleted
halves so each uses its partial index. A failed read is a 500, never an
empty answer, which would read as "register everything". The JSON body
limit goes to 5 MB (about 130k uuids). Open to PRO and LITE.
`/keys` is deprecated: still served for the builds that use it.
✅ Claude PR Review —
|
| Finding | Status |
|---|---|
src/services/db/LibraryDB.ts:90 (info) ⚠︎ moved |
✅ verified fixed in 01ef1f9 |
Converged: nothing new this round, and every earlier finding is settled.
Model claude-opus-5-5 · run log · 0 new · 0 carried over · 1 verified closed · 0 resolved · advisory (a human should still review). Findings are de-duplicated across pushes; an earlier finding closes only when the verification pass judges it against the current code — fixed, no longer applicable, accepted by a maintainer, or a duplicate of a finding reported on this push.
Scope the 5 MB JSON limit to POST /v1/library/status. The global parser runs before the rate limiter and auth, so the raised limit let any request, unauthenticated ones included, make the server buffer and parse a 5 MB body. `jsonBody` now keeps body-parser's 100 KB everywhere and leaves this one path to the route, which parses with 5 MB after `checkSubscription`.
Walk the requested uuids once, filing each as unknown or unsynced, instead of filtering the list twice. An unsynced book is always a seen one, and the else-if says so.
Check the tier before parsing the 5 MB status body: requireCloudData now runs before largeJsonBody, so a subscriber on a tier without cloud access gets the 403 without the server reading a large body first.
Match the own-body route the way Express routes (case-insensitive, trailing slash ignored), so a spelling the route accepts isn't parsed first at 100 KB and then rejected for a large library.
Make library_items.active NOT NULL. Every read filters on active = true, so a NULL row was an invisible third state, which POST /status would have answered as unknown. NULL rows are backfilled as deleted (false), which is what they already were to clients (true could also collide with the active-only unique indexes). The migration runs outside a transaction so nothing scans the table under an exclusive lock: a NOT VALID check (enforced on new writes), the backfill, VALIDATE, then SET NOT NULL proved from the check.
Give each ALTER in the active NOT NULL migration its own short transaction with SET LOCAL lock_timeout = '5s'. Their brief ACCESS EXCLUSIVE locks otherwise wait behind any open transaction on library_items, and every later query on the table queues behind them. A blocked step now fails fast, and the migration is safe to re-run. Production runs Postgres 17.9, so SET NOT NULL is proved from the validated check.
Read both halves of getItemsByUuids in one UNION ALL statement, so they share one snapshot: as two statements, a row changing state or committing between them showed up in neither and read as unknown. Each branch still uses its partial index.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Both apps find what the server is missing by comparing their local paths with
GET /keys(synced keys only).PUT /treats a known uuid at a new key as a move, so the item goes back. For a folder, its whole subtree goes with it.What
POST /v1/library/statustakes{ uuids: [...] }, every uuid in the client's local library in one body, and answers{ unknown, unsynced }.unknown: no row has the uuid, active or deleted./uuidsfirst. A legacy row at that key takes its uuid; a conflict makes the client adopt the server's.unsynced: active books with no file in S3. PRO clients upload these through/upload/*by uuid, never by path. LITE ignores the list.uuid[]parameter, becausewhereInwould hit Postgres' 65,535-parameter cap;LibraryLookupError), never an empty answer, which would read as "register everything".jsonBodykeeps body-parser's 100 KB on every other route and leaves this path to the route. The route parses its body only aftercheckSubscriptionandrequireCloudData, so unauthenticated callers, and subscribers on a tier without cloud access, never get a large body buffered. The path is matched the way Express routes (any case, trailing slash ignored).requireCloudData), and not audited (it's a read)./keysis deprecated: still served for the shipped builds that use it; new clients use/status.The client rules are in
docs/multipart-uploads.md("The missing-items pass"). Accepted exception: an item from before March 2026 (before uuids existed) that another device deleted under its own uuid, or none, reads asunknownand comes back.Deploy
Ship this together with #50/#51, before the iOS build that uses it: that build's first sync depends on
/status.One migration,
20260929120000_library_items_active_not_null:library_items.activewas nullable, and a NULL row was an invisible third state (every read filters onactive = true).false, i.e. deleted, which is what they already were to clients.truewould bring them back into users' libraries and could collide with the active-only unique indexes.CHECK (active IS NOT NULL) NOT VALID(enforced on new writes);VALIDATE(scans without blocking reads or writes);SET NOT NULL(proved from the check, no second scan);SET LOCAL lock_timeout = '5s'. Its brief exclusive lock then gives up instead of queueing behind an open transaction and stalling every query on the table behind it. The migration is safe to re-run.SET NOT NULLcan skip the second scan.false.Tests
LibraryServiceItemsStatus.test.ts:unknown/unsynced/ neither, and that deleted rows count as seen;synced, another user's uuid, spelling and duplicates, invalid strings;LibraryDBItemsByUuids.test.ts: the DB-level uuid filter.libraryItemsActive.test.ts:activedefaults to true and rejects NULL. The migration was also run up, down (a NULL row inserted), and up again locally: the row came backfalse, the column NOT NULL, the check gone.jsonBody.test.ts(supertest): a 1 MB body parses on/statusin any spelling and gets a 413 on any other route.