diff --git a/README.md b/README.md index feb0076..52875bd 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,14 @@ user's reading position quietly stops syncing. - **[`reference-server/`](reference-server/)** — how to run the reference implementation locally, which was itself undocumented. - **[`coverage.mjs`](coverage.mjs)** — fails if a requirement in `SPEC.md` has - neither a test nor a written reason why it cannot have one. Runs in CI, so - the spec and its test suite cannot drift apart quietly. + neither a test nor a written reason why it cannot have one, and if a proposed + requirement belongs to no optional feature. Runs in CI, so the spec and its + test suite cannot drift apart quietly. + +One section is not observed behaviour of a released server: **§5.8 describes an +open, unmerged pull request** and is pinned to a branch commit. It is an +optional feature, so a server that does not implement it skips those +requirements and stays conformant. This is **not** an official KOReader project and carries no endorsement from it. It is a description of observed behaviour. Where it and the reference @@ -68,16 +74,25 @@ writes to your server: see [Running the verifier](#running-the-verifier) below. ## Results -Measured 2026-09-21. `verify.mjs` against each server, with the profile flags +Measured 2026-09-23. `verify.mjs` against each server, with the profile flags each one's design calls for. -| Implementation | Version | Passed | MUST failed | SHOULD failed | Verdict | -|---|---|---:|---:|---:|---| -| `koreader/koreader-sync-server` (reference) | `koreader/kosync:latest`, OpenResty 1.29.2.3, gin 0.2.0 | 49 | **0** | 2 | conformant | -| `pid1/tsundoku` | branch `main` | 47 | **0** | 2 | conformant | +| Implementation | Version | Passed | MUST failed | SHOULD failed | Skipped | Verdict | +|---|---|---:|---:|---:|---:|---| +| `koreader/koreader-sync-server` (reference) | `koreader/kosync:latest`, OpenResty 1.29.2.3, gin 0.2.0 | 49 | **0** | 2 | 20 | conformant | +| `pid1/tsundoku` | branch `main` | 47 | **0** | 0 | 22 | conformant | + +Neither server implements §5.8, and nor does any server you can download, so both +runs skip its twenty `[K-ID-…]` requirements. tsundoku's run skips two more, the +registration assertions, because it closes kosync self-registration by design and +the run declared `--registration-off`. -tsundoku's run skips the two registration assertions, because it closes kosync -self-registration by design and the run declared `--registration-off`. +Measured 2026-09-23 against `koreader/koreader-sync-server` PR #55 (branch +`multi-identifier-aliases` at `49dbd38`, built locally on OpenResty 1.29.2.3), +where the probe does find §5.8: **71 passed, 0 MUST failures, 0 SHOULD failures, +0 skipped**. The 51 outside §5.8 are what the published image is scored on, where +it passes 49 and fails two SHOULD — that branch also carries the unreleased fix +for the document-id defect below. The reference server's two SHOULD failures are a genuine defect in it, not a disagreement about the protocol: `PUT` accepts any document id, while `GET` @@ -166,6 +181,13 @@ of synthetic document ids derived from `--document`. Those ids are constant across runs by design, so a hundred runs leave five rows rather than five hundred. Point it at a test account. +The `identifiers` feature of §5.8 is the exception: it allocates **fresh digests +on every run**, because an alias, once created, is never repointed, so a second +run over the same digests would resolve through the first run's aliases and +assert nothing. Against a server that implements it, each run leaves roughly +twenty document and alias keys behind; against one that does not, it leaves the +single record its probe writes. + It **never probes `DELETE /users/me`**, because a conformant server would delete the account under test. diff --git a/SPEC.md b/SPEC.md index 0b9c45e..40d6207 100644 --- a/SPEC.md +++ b/SPEC.md @@ -31,6 +31,11 @@ This document describes the behaviour of those two programs. It is a KOReader project.** Where this document and the reference implementations disagree, the implementations are right and this document has a bug. +**§5.8 is the one exception, and it says so at its head.** It describes an +open, unmerged pull request rather than released behaviour, it is pinned to a +branch commit rather than to the table above, and nothing in it is part of +conformance. Every other section describes code that ships. + The reference server was **built and run** for this document: `docker.io/koreader/kosync:latest` (image `db2a16684d0b`, OpenResty 1.29.2.3, `X-Framework: gin/0.2.0`), under Podman on arm64, on 2026-09-21. Claims marked @@ -496,6 +501,9 @@ Push a reading position. | `device_id` | string | no | opaque per-device id | | `metadata` | object | no | see §7.6 | +An unmerged proposal adds one more optional field, `identifiers`; it is +described in §5.8 and is not part of conformance. + `api.json` lists `document`, `progress`, `percentage`, `device` and `device_id` as `required_params` and `metadata` as an `optional_param` (`koreader plugins/kosync.koplugin/api.json:23-44`) — but `required_params` is @@ -532,6 +540,10 @@ code **2004**, message `Field 'document' not provided.` {"document": "", "timestamp": 1790005119} ``` +**[K-PUT-7]** The server **MUST NOT** answer **202**. `api.json` declares it as +an accepted status, and no client treats it as success, so a 202 is a push the +client queues for retry while the server believes it stored the position. + **[K-PUT-4]** `timestamp` is **generated by the server** as Unix epoch **seconds** (`syncs_controller.lua:246`: `local timestamp = os.time()`). Any client-supplied `timestamp` in the body is ignored — the reference client never @@ -662,6 +674,10 @@ cannot serve on `GET`, or accept a wider character set on both. | `device_id` | string | stored (omitted if the pushing client never sent one) | | `timestamp` | number | stored | +An unmerged proposal adds an optional `ids` query parameter and two more +response fields to this endpoint; it is described in §5.8 and is not part of +conformance. A request that does not send `ids` is answered as below. + `syncs_controller.lua:193-226`. Each field is emitted only if the Redis hash returned a non-`null` value for it, and `document` is added **only if at least one other field is present** (`:221-224`): @@ -787,6 +803,485 @@ succeed (`syncs_controller.lua:64-72`). A retry with the now-stale key returns **[K-PWD-2]** Stored progress is preserved. Document keys are not touched. +### 5.8 Optional identifier matching — **PROPOSED, NOT MERGED** + +> **Status: proposed.** Everything in this section describes +> `koreader/koreader-sync-server` **pull request #55**, which is **open and not +> merged**. No released server implements it. If #55 is merged in a different +> shape, this section is wrong and not the server. +> +> Every `[K-ID-…]` requirement is `MAY`, under the optional feature +> `identifiers` (§12.5). Implementing the section is optional and a server that +> ignores the fields below stays conformant; a server that implements it is +> held to all of them, so `MUST` below means "if you implement this". + +| | | +|---|---| +| Upstream PR | [`koreader/koreader-sync-server#55`](https://github.com/koreader/koreader-sync-server/pull/55) | +| Wire shape described here | branch `multi-identifier-aliases` at `0c0f5ad4eedf31668a6ca7646ed8ae9a9c4ed853` (`pid1/koreader-sync-server`, 2026-09-22) | +| Built and run for this section | that branch under OpenResty 1.29.2.3 with Redis 8.10.2, `GIN_ENV=test`, on 2026-09-22 | + +Citations in this section are to that branch, not to the pinned reference +commit of §1.1, and are written `identifiers.lua:7` for +`lib/identifiers.lua`. + +**The problem.** `document` is one digest of one file (§8). A reader who +recompresses an EPUB, re-downloads it from another shop, or converts it gets a +different digest and loses the position, even though it is the same book. +There is no way to say "this file is also known as" in v1. + +**The shape.** Both progress endpoints take an optional, ordered list of +identifiers. A request that names none is answered exactly as it is today. + +#### Request + +`PUT /syncs/progress` takes an optional `identifiers` array beside the existing +fields: + +```json +{"document": "C1", + "identifiers": [{"type": "content", "value": "C1"}, + {"type": "structure", "value": "S1"}, + {"type": "filename", "value": "F"}], + "percentage": 0.32, "progress": "/body/DocFragment[20]/body/p[22]", + "device": "my kpw"} +``` + +`GET /syncs/progress/:document` takes the same list flattened into one `ids` +query parameter, because a GET has no body and repeated query parameters are +not reliably ordered (`identifiers.lua:92-94`): + +``` +GET /syncs/progress/C1?ids=content:C1,structure:S1,filename:F +``` + +An entry is a `{type, value}` pair. `type` is an **opaque label chosen by the +client**: the server stores and echoes it without interpreting it, so a new +kind of identifier needs no server change (`identifiers.lua:1-3`). + +##### Type registry + +`type` is opaque **to the server**, which stores and echoes it without +interpreting it. It is not opaque to clients: `progress_match` is a trust +signal, and a client decides whether to follow another device's `progress` +string on the strength of a shared label. Two clients that compute the same +label differently therefore disagree about what they have agreed on, so a label +without a recipe is not interoperable. + +| `type` | value | changes when | +|---|---|---| +| `content` | the document's own digest over the file bytes, as §8 derives it | any byte of the file changes | +| `structure` | md5 over the spine, as defined below | the edition, or the chapter list or its order, changes | +| `filename` | md5 of the file name, as §8.5 derives it | the file is renamed | +| `metadata` | md5 of `"title:" + title + "\n" + "authors:" + authors`, normalised as below. A title with no author yields nothing. **Always weak** (`[K-ID-17]`) | the work's title or authors change | + +#### Weak identifiers + +**[K-ID-16]** An entry in the `PUT` body **MAY** carry `"weak": true`. It means +the client does not consider that identifier sufficient to claim a record that +already exists. The flag appears **only in the request body**: adoption is a +property of a write, so the `ids` parameter of a read has no equivalent and its +grammar is unchanged. + +**[K-ID-17]** A `PUT` whose walk resolves **only** through a weak entry **does +not adopt that record.** It writes under `document`, creating it if absent, +exactly as though nothing had matched, and answers with that digest. A record +found through a strong entry is adopted as before. Reads are unaffected: a weak +identifier resolves on a `GET` like any other, and `progress_match` says what the +position is worth. + +This is what separates seeding from binding. A copy that shares only a weak +identifier is shown the other copy's position and can seek to its percentage, +and it still keeps a record of its own; it cannot overwrite the position of the +record it read from, and it cannot merge two works that happen to be tagged +alike. Without it, `progress_match` tells a reader not to trust a weak match +while nothing tells a writer not to clobber on one. + +A client **MUST** mark as weak any identifier that can match a different work. +`metadata` is always weak. `content` and `structure` never are. + +Five things the rule would otherwise leave to each implementation: + +- **"Only through a weak entry" means the entry the walk stopped on.** The walk + stops at the first hit (`[K-ID-5]`) and does not continue past a weak one + looking for a strong hit further down. Under `[K-ID-5b]` the two readings + coincide, since weak entries sort last; they differ only for a client that + breaks the ordering rule. +- **`weak` is a boolean.** Anything else is rejected with **403** code **2003**, + like any other malformed entry. Reading a non-boolean as strong is exactly the + clobber the flag exists to prevent. +- **`match` on a non-adopting write is the create rule** (`[K-ID-8b]`): the type + of the entry whose value equals `document`, not the weak type that resolved + the walk. +- **Weakness governs adoption, not registration.** A weak entry at or below the + match is still registered as an alias, whether or not the write adopted. That + is what lets a later read be seeded at all. +- **The stored writer list does not record the flag.** `progress_match` reports + the shared type and leaves the judgement to the reader, so weakness is not + recoverable after the write. +- **A weak entry that is itself the `document` still adopts.** `[K-ID-8]` + requires the list to contain an entry equal to `document`, and nothing stops + that entry being the weak one — a client whose configured document digest is + its weakest identifier is in exactly that position. Writing to the record you + are addressed by is not claiming another's, so `[K-ID-17]` does not fire when + the entry that resolved the walk carries the `document` value. + +**[K-ID-15]** A client **offers only the types it can compute honestly.** The +registry is not a set a client has to fill: a type whose recipe it cannot follow +for the book in hand is omitted, not approximated. A client that can derive no +identifier but `document` itself has nothing to say and sends no list at all, +which is the request in §5.4 exactly as it was. Substituting a value that merely +resembles a registered type is worse than omitting it — a digest over a title +dressed up as `filename` is the `metadata` type this section removed, and +`[K-ID-12]` will make one bad match permanent. + +**[K-ID-14]** A client **MUST NOT** treat a `progress_match` type it does not +recognise as sufficient to follow a `progress` string. An unrecognised label +carries no guarantee about the file the position was written against, so it is +`percentage` that applies, not the stored position. + +The same applies to anything **derived** from that string, not only to following +it. A client that resolves the stored position against the local book to compare +it with the local one — for conflict detection, for a preview, for deciding +whether to prompt — is resolving a position from a file it may not share, and the +number it gets back looks authoritative. Gating only the jump leaves that number +free to suppress a conflict prompt, at which point the position is overwritten +with no one asked. Where the match is not one the client may follow, `percentage` +is the only comparable quantity. + +`structure` is computed from the OPF, and two clients have to compute it +identically or the label means nothing. The recipe, exactly: + +1. Take the `` element's `unique-identifier` attribute and find the + `` whose `id` matches it — the first, if several do. Trim it. + If the result is empty, or no element matches, fall back to the first + `` whose trimmed text is non-empty. That value is the first + line; if there is none, there is no first line. Trimming removes the XML + whitespace characters (space, tab, CR, LF) and nothing else. +2. Walk `` in document order. For each ``, resolve its `idref` + against `` and take that item's `href`: **not** percent-decoded, + **not** resolved against the OPF directory, **not** reduced to a basename. + Strip a `#fragment` if one is present. Each is a line, in spine order. +3. Join the lines with `\n`, with no trailing newline, and take the md5 of the + UTF-8 bytes. + +A container with no spine, or one that is not an OPF-bearing archive, has no +`structure` digest and the identifier is omitted rather than guessed. + +The rest of the recipe, because two implementations will otherwise each make a +defensible choice and disagree: + +- **Every value here is the one an XML parser yields, with entity references + expanded** — the identifier, the hrefs and `full-path` alike. An `href` written + `a&b/ch2.xhtml` contributes the thirteen characters `a&b/ch2.xhtml`. Raw + source bytes are used nowhere. Expanding is what a parser does without being + asked; keeping a value raw means going out of its way, and a rule that is + harder to obey is one two implementations will obey differently. +- **Element names are matched on the local name**, so `` and + `` are the same element and `` under a default Dublin Core + namespace counts as ``. **Attribute names are matched + literally and unprefixed** — `href`, `id`, `idref`, `full-path`, `media-type`, + `unique-identifier`. An unprefixed attribute is in no namespace, so `opf:href` + is a different attribute rather than the same one, and the package format + writes these unprefixed. +- **An element's value is its complete character data, concatenated**, not the + first text node. A parser may split a run at an entity reference or a buffer + boundary, and two that split differently must still agree. +- **Only the five predefined XML entities and numeric character references are + expanded**: `& < > " '`, `&#nnn;` and `&#xHH;`. Any other + undeclared reference makes the document not well-formed, and a digest taken + from one is undefined rather than wrong — a conforming parser rejects the + document where a lenient one may expand an HTML name it happens to know. +- A package that puts `` outside `` is malformed, and resolves + no spine entry, so it has **no** `structure` digest rather than one computed + from whatever was found elsewhere. +- **At least one spine line is required.** A digest of the identifier alone is + not a `structure` digest, so a package whose ``s all dangle has none, + the same as one with no `` at all. +- Two `` elements carrying the same `id`: the **first**, as for + ``. +- **An `href` is not trimmed.** Step 1 trims the identifier and nothing else + does; leading or trailing space inside the attribute is part of the value. +- **An `` whose `idref` resolves to no manifest item contributes no + line**, and does not invalidate the digest. So does one that resolves to an + `` carrying no `href`: no href, no line, rather than an empty one. +- **`linear="no"` items are included.** They are spine entries, and the + renderer's `DocFragment` numbering counts them. +- `` is looked up within ``, `` within ``, and + `` within ``. +- **The OPF is the first `` whose `media-type` is + `application/oebps-package+xml`**, or the first `` of any type when + none declares it. Its `full-path`, as the parser yields it, is the archive + member name; if no member matches, there is no digest rather than a guessed + one. Preferring + the declared package document keeps this from disagreeing with whatever a + reader already opens for a multi-rendition container. + +`structure` deliberately covers **no file contents.** The tools that motivate +this section rewrite them: CrossPoint's EPUB optimizer re-encodes images to +JPEG through a canvas, injects a stylesheet into every chapter's `` and +rewrites `` when it splits an image, so every entry's bytes change +while the spine does not. A digest over entry CRCs survives recompression and +fails against the very tool this exists for. The spine href list is also exactly +what an xpointer counts — `/body/DocFragment[N]` is the Nth spine entry — so a +`structure` match says precisely that the position's chapter index means the +same thing here. + +The `metadata` normalisation, because three implementations disagreed on it: + +- **The authors are the `` elements of the OPF**, in the order the + document gives them, taken from each element's **text** and never from an + `opf:file-as` attribute. `` is not an author. Each is + normalised on its own, then they are **sorted** and joined with `;`. +- **Case folding is ASCII only.** `A`–`Z` fold to `a`–`z` and nothing else does. + A full Unicode fold is not available on every client — a microcontroller has no + case table and is not going to carry one — and an identifier two clients fold + differently is worse than one that folds less. The cost is a miss on a title + whose case varies outside ASCII, which for a weak identifier is the safe + direction. +- **Whitespace is the XML set**, space, tab, CR and LF, as in the `structure` + recipe. Runs collapse to one space and the ends are trimmed. No other + character is whitespace. +- A value that normalises to nothing is absent, and an absent title or an empty + author list yields no `metadata` digest at all. + +`metadata` is the only identifier that survives a **spine change**, so it is the +only one that reaches a different edition or a re-chunked conversion. It is also +the only one that can match two genuinely different works, which is why it is +never a binding identifier: it seeds a reader, it does not claim a record. See +`[K-ID-16]` and `[K-ID-17]`. + +| Rule | Value | Citation | +|---|---|---| +| Maximum entries per request | **8** | `identifiers.lua:7` | +| `type` pattern | `^[a-z][a-z0-9-]*$`, at most 32 characters | `identifiers.lua:8,14` | +| `value` pattern | `^[A-Za-z0-9][A-Za-z0-9-_.]*$`, at most 128 characters | `identifiers.lua:9,15` | +| Duplicate `type` in one list | rejected | `identifiers.lua:82-85`, `:119-122` | +| Empty list | rejected | `identifiers.lua:64-66` | +| Order | the client's **order of preference**, strongest first — required, see `[K-ID-5b]` | `identifiers.lua:49-50` | + +**[K-ID-8]** The list **MUST contain an entry whose `value` equals `document`** +(`syncs_controller.lua:197-202`). A list that does not is rejected with **403** +code **2003**, on both the write and the read. This is what keeps `document` +meaning *"the identifier I would send if you only took one"*: the record stays +addressable by the digest a client naming no identifiers sends, and a read that +names none follows no alias to reach it (`[K-ID-1b]`). + +**[K-ID-8b]** That entry **need not be first.** Position in the list carries +preference, not identity. When nothing resolves, the record is created under +`document` rather than under the first entry, and `match` is that entry's type +(`syncs_controller.lua:104-109`). Pinning it to the first position would force a +client whose `document` digest is its weakest identifier to offer that one first +and be matched on it — which is what KOReader sends when its document matching +is set to the filename, and it would report `progress_match: "filename"` for a +copy whose content is byte-identical. + +**[K-ID-9]** A list of more than **8** entries is rejected with **403** code +**2003** (`identifiers.lua:67-69`, `:103-105`). Exactly 8 is accepted. Each +identifier costs a lookup on the read path and a possible alias write on the +write path, so the cap is what bounds one request's work. + +#### Response + +**[K-ID-2]** A `PUT` that names identifiers returns **200** with exactly three +fields (`syncs_controller.lua:500-504`): + +```json +{"document": "C1", "match": "content", "timestamp": 1790124995} +``` + +`document` is the **canonical** digest the record is stored under, which is not +necessarily the one the request sent. There is no `progress_match` on a write: +the writer is the request itself. + +**[K-ID-3]** `match` is the `type` of the identifier that **resolved the +lookup**, reported with the *caller's* own label. For a record being created it +is the first entry's type, since that is what the record is created under +(`syncs_controller.lua:98-100`). + +A `GET` that names identifiers returns today's fields plus two: + +| Field | Type | Meaning | +|---|---|---| +| `match` | string | the identifier type that **found** the record | +| `progress_match` | string | the strongest identifier the caller shares with whoever **wrote the current `progress` string**, or `"none"` | + +**[K-ID-6]** `match` and `progress_match` answer different questions and +routinely differ. A reader can match a record on its own content digest — +`match: "content"`, the record is literally its file — while the position +stored there was written by a different book that only shared the file name: +`progress_match: "filename"`. The first says *we found your book*, the +second says *how much you should trust this xpointer*. Only `progress_match` +bears on whether the position can be followed +(`syncs_controller.lua:350-361`, `identifiers.lua:180-195`). + +`progress_match` is computed against the identifiers stored **beside the +progress string they were written with** (`identifiers_for`), not against +whatever the record last saw, so identifiers are never attributed to a string +their owner did not write (`syncs_controller.lua:426-431`, `:356`). Two +consequences: + +- **[K-ID-6b]** `"none"` when the caller and the writer share no value at all. +- **[K-ID-6c]** When the current `progress` was written by a request that named + **no** identifiers, the record is treated as written by the digest it is + stored under, so `progress_match` equals `match` (`syncs_controller.lua:360`). + +**[K-ID-1]** A request that names **no** identifiers gets **exactly today's +response**: `{document, timestamp}` on the write, the §5.5 body on the read, +**with neither `match` nor `progress_match`**, and the read **follows no +alias** — it looks up the literal `document` and nothing else +(`syncs_controller.lua:378-386`). An existing client sees no change of any +kind. + +**[K-ID-11]** A document that matches nothing, under any identifier offered, +is still **200 with an empty body** — `[K-GET-3]` is unaffected, and there are +no matching fields to report on a miss (`syncs_controller.lua:318-320`, +`:346-348`). + +#### Resolution and aliases + +Identifiers other than the record's own become **aliases**: per account, +`user::alias:` holding `:`. + +Resolution walks the caller's list **in order**, and for each entry tries the +value as a document first and as an alias second, stopping at the first hit +(`syncs_controller.lua:84-97` on the write, `:131-141` on the read). + +**[K-ID-5]** Because the walk is in the caller's order, the same three +identifiers offered strongest-first and weakest-first resolve through different +entries and report different `match` values. The list is a preference order, +not a set; a server that resolves in its own order gives two clients different +answers about the same book. + +**[K-ID-5b]** A client **MUST** order its identifiers strongest first — most +specific to the file in hand, least specific last. The registry table above is +in descending strength order for the types it names. This is not a stylistic +preference: `[K-ID-12b]` is the protection against a wrong match becoming +permanent, it is expressed in terms of list position, and a list that opens with +its weakest identifier matches at the first position and so registers every +identifier behind it. **Measured 2026-09-23** against the branch above with the +`[K-ID-12b]` rule in place: two books sharing only a weak identifier, offered +weakest-first, merge and stay merged after the weak identifier is corrected — the +same failure `[K-ID-12b]` exists to prevent, reproduced with nothing but a +different list order. A server cannot check this (§12.4). + +**[K-ID-4]** A copy that shares any identifier with a record the account +already holds resolves to that record, and the response carries the +**canonical** digest so the next request can address it directly. + +**[K-ID-7]** An alias **MUST NOT shadow a document that exists in its own +right.** The walk tests `EXISTS document:` before consulting the alias +table, and an alias is created only for a value that is not already a document +(`syncs_controller.lua:87-88`, `:104`). A weak identifier can therefore fail to +match, but it cannot move one book's position onto another book's record. + +**[K-ID-12]** An alias is **created, never repointed.** The write path writes +an alias only when none exists, or when the one that exists points at a +document that is gone (`syncs_controller.lua:115-120`). A digest that has +resolved to a record keeps resolving to it. + +**[K-ID-12b]** An identifier the caller ranks **above** the one that matched is +**not registered at all** (`syncs_controller.lua:110`). Matching on a weak +identifier is a guess; registering the caller's stronger digests as aliases to +that guess would make a wrong one permanent, because `[K-ID-12]` never repoints +and no endpoint unlinks. Two books a library has tagged alike share only their +weakest identifier, and without this rule the second one pushed takes the first +one's record, overwrites the position stored there, and cannot be separated +again by correcting the tagging — its own content digest has been glued to the +other record on the way through. Confining a wrong guess to the identifier that +made it is what makes it recoverable. On a **create** the record is the caller's +own, so every identifier it offers describes it and all of them are registered. + +What this rule recovers is the **copy**, not the position: the intruding push +still overwrote what was stored on the other record, and nothing restores that. +It also has a cost on the benign path. A recompressed copy that matched on +`structure` never has its own `content` digest registered, on that push or any +later one, so it depends on the weaker identifier continuing to match and is +orphaned if the spine later changes. That trade is deliberate, and it is a trade. +The rule is only as good as `[K-ID-5b]`, which a server cannot enforce. + +**[K-ID-10]** Aliases are namespaced per account, like positions +(`[K-ISO-1]`). **[K-ID-13]** They live under the account's `user::` +prefix, so `DELETE /users/me` removes them with everything else +(`[K-DEL-1]`). + +#### [measured] + +Against the branch above at `49dbd38` on 2026-09-23, `GIN_ENV=production`, +plaintext listener, account `reader`. `C1` is the original, `C2` a recompressed +copy of it, and `C3` a different book that happens to carry the same file name. + +``` +$ curl -X PUT .../syncs/progress -d '{"document":"C1", + "identifiers":[{"type":"content","value":"C1"}, + {"type":"structure","value":"S1"}, + {"type":"filename","value":"F"}], + "percentage":0.32,"progress":"/body/DocFragment[20]/body/p[22]", + "device":"my kpw"}' +{"document":"C1","timestamp":1790176644,"match":"content"} + +# the recompressed copy: different content digest, same spine +$ curl '.../syncs/progress/C2?ids=content:C2,structure:S1,filename:F' +{"percentage":0.32,"device":"my kpw","progress_match":"structure", + "progress":"\/body\/DocFragment[20]\/body\/p[22]","document":"C1", + "timestamp":1790176644,"match":"structure"} + +# a different book sharing only the file name writes, and takes the record +$ curl -X PUT .../syncs/progress -d '{"document":"C3", + "identifiers":[{"type":"content","value":"C3"}, + {"type":"structure","value":"S3"}, + {"type":"filename","value":"F"}], + "percentage":0.5,"progress":"/body/DocFragment[3]/body/p[9]","device":"pb"}' +{"document":"C1","timestamp":1790176644,"match":"filename"} + +# the original reads again: found by its own digest, written by the other +$ curl '.../syncs/progress/C1?ids=content:C1,structure:S1,filename:F' +{"percentage":0.5,"device":"pb","document":"C1", + "progress":"\/body\/DocFragment[3]\/body\/p[9]","progress_match":"filename", + "timestamp":1790176644,"match":"content"} + +# the same read naming no identifiers: today's body, no new fields +$ curl '.../syncs/progress/C1' +{"percentage":0.5,"device":"pb","document":"C1","timestamp":1790176644, + "progress":"\/body\/DocFragment[3]\/body\/p[9]"} + +# a list that names the document nowhere +$ curl '.../syncs/progress/C1?ids=filename:F' +403 {"code":2003,"message":"Invalid request"} +``` + +Redis after that sequence — one document and **two** aliases, all under the +account: + +``` +user:reader:alias:F user:reader:alias:S1 +user:reader:document:C1 user:reader:key +``` + +`C3` and `S3` are absent, and that is `[K-ID-12b]`. The third push matched on +`filename`, the weakest identifier it offered, so the two digests it ranked above +that one were not registered. Had they been, the second book would be reachable +only through the first book's record for good, and correcting the file name would +not undo it. What the rule does not undo is the overwrite: `C1` now holds the +other book's position, and the `my kpw` reading at 0.32 is gone. + +#### Running the checks + +```bash +node verify.mjs --base-url http://127.0.0.1:8080 \ + --user alice --password hunter2 +``` + +There is no flag. The verifier pushes one position naming identifiers and reads +the answer: a `match` field means the feature is implemented and the family is +checked, and its absence means the family is recorded as `SKIP`. + +The identifier checks allocate **fresh digests on every run**, unlike the rest +of the suite, which reuses stable ids by design. They have to: `[K-ID-12]` says +an alias is never repointed, so a second run over the same digests would +resolve through the first run's aliases and assert nothing. + + --- ## 6. Error model @@ -1723,6 +2218,11 @@ Full options, profile flags and the recipe for standing up the reference server are in [`README.md`](README.md) and [`reference-server/`](reference-server/). +The `[K-ID-…]` requirements of §5.8 do not decide that verdict. They describe an +unmerged proposal and are `MAY`, under the optional feature `identifiers` +(§12.5): a server that does not implement them skips them and stays conformant, +and a server that does is held to all of them. + ### 12.2 Checklist for a new server Work down this list; each row names the requirement it satisfies. @@ -1764,7 +2264,7 @@ Work down this list; each row names the requirement it satisfies. - [ ] `GET` always includes `timestamp`. `[K-FLD-13]` - [ ] Accept `metadata` without failing; do not clear it when omitted. `[K-FLD-10]`, `[K-FLD-11]` - [ ] Answer within 5 seconds. `[K-SYNC-7]` -- [ ] Never emit `202`. §5.4 +- [ ] Never emit `202`. `[K-PUT-7]` **Errors** @@ -1791,6 +2291,12 @@ It fails if a requirement is defined here with neither an assertion nor a §12.4 row, and it fails if `verify.mjs` asserts an identifier this document does not define. It runs in CI. +Requirements defined in a section whose heading says **proposed** are counted +separately, and `coverage.mjs` additionally fails if one of them belongs to no +optional feature. `verify.mjs` refuses at run time to record a requirement under +a feature's prefix without naming that feature, so a proposed requirement cannot +reach `MUST` and be charged to every server. + ### 12.4 Requirements the verifier cannot check | Requirement | Why not | @@ -1799,6 +2305,7 @@ define. It runs in CI. | `[K-URL-1]` unversioned paths | True by construction: every other assertion is issued against exactly these paths, so the suite could not pass if they were versioned. | | `[K-AUTH-1]` credentials in `x-auth-user`/`x-auth-key` | Same: every authenticated assertion sends only these two headers. A server reading credentials elsewhere fails all of them. | | `[K-AUTH-2]` the key is `md5(password)` | Client-side. Checkable only by inspecting a client. The verifier computes the MD5 itself and so exercises the server's half. | +| `[K-FLD-1]` `document` is an opaque non-empty string without a colon | Asserted in consequence rather than directly: every authenticated assertion addresses a document, so an empty one fails them all, and the character set is probed as `[K-DOC-ID-1]` and `[K-DOC-ID-2]`. The colon is refused by the reference's own read route (§14.3) before the field rule can be observed. | | `[K-CT-1]` clients send valid JSON | Client-side. | | `[K-CT-4]` clients must not dispatch on response `Content-Type` | Client-side. The server half is `[K-CT-3]`, which is asserted. | | `[K-REG-1]` `password` is pre-hashed | Not directly observable: a correct server and a double-hashing server both accept the registration. The verifier asserts the consequence instead — register with key `K`, then authenticate with key `K` — which a double-hashing server fails. | @@ -1812,6 +2319,11 @@ define. It runs in CI. | `[K-FLD-12]` `timestamp` is epoch seconds | Asserted as `[K-FLD-14]`, which checks the value is within a day of now and so would catch milliseconds. | | `[K-FLD-8]` device self-detection | Client-side. The server precondition — that `device_id` round-trips — is `[K-FLD-7]`. | | `[K-SYNC-1]` … `[K-SYNC-6]` conflict resolution | Entirely client-side; the server never compares anything. The verifier asserts the server-side preconditions instead: `timestamp` present (`[K-FLD-13]`), in seconds (`[K-FLD-14]`), server-generated (`[K-PUT-4]`), and last-write-wins (`[K-PUT-5]`). | +| `[K-ID-12]` an alias is created, never repointed | Not observable over HTTP: the alias table is private, and every wire consequence of it is already asserted — resolution is stable (`[K-ID-4]`), a document that exists in its own right is never shadowed (`[K-ID-7]`), and an identifier above the match is never registered (`[K-ID-12b]`). Proposed; see §5.8. | +| `[K-ID-5b]` identifiers are ordered strongest first | Client-side, and not inferable from the wire: the server has no idea what any type means, so a weakest-first list is indistinguishable from a strongest-first one whose strongest identifier happens to be the one that matched. Checkable only by inspecting a client. Proposed; see §5.8. | +| `[K-ID-15]` a client offers only the types it can compute honestly | Client-side, and invisible from the server: a list of two is indistinguishable from a client that could only manage two, and a value that merely resembles a registered type is a correct-looking string the server has no way to judge. Checkable only by inspecting a client. Proposed; see §5.8. | +| `[K-ID-14]` a client must not follow a position on an unrecognised `progress_match` | Client-side. The server reports the type and acts on it in no way, so nothing over the wire distinguishes a client that honours this from one that ignores it. Checkable only by inspecting a client. Proposed; see §5.8. | +| `[K-ID-13]` aliases are removed with the account | Same reason as `[K-DEL-1]`: probing `DELETE /users/me` would delete the account under test. Proposed; see §5.8. | | `[K-DOC-1]` … `[K-DOC-10]` document identity | Not a server behaviour: `document` is opaque to a server, and four of the eight surveyed servers never compute it (§11.1). Checked by `vectors/check.mjs` against the golden vectors of §8.6, not over HTTP. | @@ -2002,6 +2514,10 @@ questioning the code's existence (`syncs_controller.lua:17`): | Is there any client that sends a `timestamp` on PUT? | Not among those surveyed. The reference server would ignore it. | | Is `202` reachable on any server? | Not among those surveyed. | | Is error code `100` reachable? | **No**, in practice — §3.3. Every HTTP client sends a default `Accept`, so the reference server answers `101`. | +| Will `identifiers` (§5.8) be merged, and in this shape? | **Open.** `koreader/koreader-sync-server#55` is unmerged as of 2026-09-27. §5.8 is pinned to a branch commit for that reason. | +| What identifier **types** should a client send? | `content`, `structure`, `metadata` and `filename`, strongest first, with the recipes in the §5.8 registry. `metadata` is always weak (`[K-ID-17]`). The server treats `type` as an opaque label and never interprets it (`identifiers.lua:1-3`), so the set can grow without a server change — but `progress_match` makes the label a trust signal between clients, so a label without a recipe is not interoperable (`[K-ID-14]`). None of the four is computed by any released KOReader build. | +| How should a client pick the `document` when it holds several identifiers? | The one it would send if the server took only one. §5.8 requires one entry to equal `document` and not that it be first (`[K-ID-8b]`), so a client addressed by its filename digest ranks that entry last. | +| Does an alias ever expire? | **No, in the proposal.** Aliases are removed only with the account (`[K-ID-13]`). A long-lived account accumulates one alias per distinct identifier it has ever offered, and nothing in the branch bounds that. | | Does the official deployment carry the `[K-GET-1a]` router defect? | **[unverified]** — the published image does (measured). Whether `sync.koreader.rocks` runs that image is unknown. | --- diff --git a/coverage.mjs b/coverage.mjs index 51f4dac..91a62b9 100644 --- a/coverage.mjs +++ b/coverage.mjs @@ -6,8 +6,14 @@ * being prose. This fails if either: * * - a requirement is defined in SPEC.md but is neither asserted by verify.mjs - * nor listed in SPEC.md §12.4 as untestable, with a reason; or - * - verify.mjs asserts an identifier that SPEC.md does not define. + * nor listed in SPEC.md §12.4 as untestable, with a reason; + * - verify.mjs asserts an identifier that SPEC.md does not define; or + * - a PROPOSED requirement belongs to no optional feature, which would hold + * every server to an unmerged proposal. + * + * A requirement is proposed if it is defined under a heading whose text says + * so. Those are counted separately, because "SPEC.md defines N requirements" + * should not quietly grow by things no released server implements. * * node coverage.mjs # report and exit non-zero on a gap * node coverage.mjs --list # also print the full mapping @@ -21,7 +27,27 @@ const list = process.argv.includes("--list"); // A requirement is DEFINED by a bold identifier at the head of a statement: // **[K-GET-3]** ... or **[K-GET-3] An unknown document ...** -const defined = new Set([...spec.matchAll(/\*\*\[(K-[A-Z0-9-]+)\]/g)].map((m) => m[1])); +const DEFINITION = /\*\*\[(K-[A-Z0-9-]+)\]/g; +const defined = new Set([...spec.matchAll(DEFINITION)].map((m) => m[1])); + +// PROPOSED requirements: defined under a heading that says "proposed". The +// heading owns everything up to the next heading of the same or higher level. +const proposed = new Set(); +for (const m of spec.matchAll(/^(#{2,4}) .*$/gm)) { + if (!/proposed/i.test(m[0])) continue; + const rest = spec.slice(m.index + m[0].length); + const next = rest.search(new RegExp(`^#{1,${m[1].length}} `, "m")); + for (const d of (next === -1 ? rest : rest.slice(0, next)).matchAll(DEFINITION)) proposed.add(d[1]); +} + +// verify.mjs registers each optional feature with the identifier prefix its +// requirements share, and refuses at run time to record one of them without +// naming the feature. What that cannot catch is a proposed requirement attached +// to no feature at all, which would be checked as MUST against every server. +const optional = [...verifier.matchAll(/requirements:\s*"(K-[A-Z0-9-]+)"/g)].map((m) => m[1]); +const unattached = [...proposed] + .filter((id) => !optional.some((prefix) => id.startsWith(prefix))) + .sort(); // ASSERTED by the verifier. const asserted = new Set([...verifier.matchAll(/id:\s*"(K-[A-Za-z0-9-]+)"/g)].map((m) => m[1])); @@ -57,7 +83,12 @@ for (const line of table.split("\n")) { // An assertion may refine a requirement with a suffix (K-CT-3 -> K-CT-3b), and // §12.4 may cover a family with a trailing dash (K-SYNC- covers K-SYNC-4). -const assertedFor = (id) => asserted.has(id) || [...asserted].some((a) => a.startsWith(id)); +// +// A refinement suffix is never a digit: without that, K-ID-16 counts as covered +// by an assertion for K-ID-1, and every requirement numbered past 9 is checked +// against the wrong one. +const refines = (a, id) => a.startsWith(id) && !/^\d/.test(a.slice(id.length)); +const assertedFor = (id) => asserted.has(id) || [...asserted].some((a) => refines(a, id)); const untestableFor = (id) => { if (untestable.has(id)) return untestable.get(id); for (const [key, reason] of untestable) { @@ -77,7 +108,7 @@ for (const id of [...defined].sort()) { } } -const undefinedIds = [...asserted].filter((a) => !defined.has(a) && ![...defined].some((d) => a.startsWith(d))).sort(); +const undefinedIds = [...asserted].filter((a) => !defined.has(a) && ![...defined].some((d) => refines(a, d))).sort(); if (list) { const w = Math.max(...rows.map((r) => r[0].length)); @@ -91,6 +122,7 @@ const counts = rows.reduce((a, [, s]) => ((a[s] = (a[s] ?? 0) + 1), a), {}); process.stdout.write( `SPEC.md defines ${defined.size} requirements: ` + `${counts.asserted ?? 0} asserted, ${counts.untestable ?? 0} declared untestable, ${gaps.length} unaccounted for.\n` + + `${proposed.size} of them are PROPOSED (unmerged upstream), and belong to an optional feature that a server is probed for.\n` + `verify.mjs makes ${asserted.size} distinct assertions.\n`, ); @@ -101,6 +133,12 @@ if (gaps.length) { for (const id of gaps) process.stdout.write(` ${id}\n`); process.stdout.write(`\nEither assert them in verify.mjs, or add a row to SPEC.md §12.4 saying why not.\n`); } +if (unattached.length) { + bad = true; + process.stdout.write(`\nProposed requirements that belong to no optional feature:\n`); + for (const id of unattached) process.stdout.write(` ${id}\n`); + process.stdout.write(`\nRegister a feature in verify.mjs whose \`requirements\` prefix covers them, or\nevery server is held to an unmerged proposal.\n`); +} if (undefinedIds.length) { bad = true; process.stdout.write(`\nAsserted by verify.mjs but not defined in SPEC.md:\n`); diff --git a/verify.mjs b/verify.mjs index a845ea4..2d7e625 100644 --- a/verify.mjs +++ b/verify.mjs @@ -123,15 +123,16 @@ const KEY2 = USER2 ? (opts.secondKey ?? (opts.secondPassword && md5(opts.secondP * to the verifier to be scored correctly will be described wrongly, and the * resulting page of failures says nothing about the server. * - * key short identifier, used in the report and in `--json` - * title what the feature is, in a noun phrase - * section the section of SPEC.md that defines it - * probe async, returns a boolean or { present, note } + * key short identifier, used in the report and in `--json` + * title what the feature is, in a noun phrase + * section the section of SPEC.md that defines it + * requirements the identifier prefix its requirements share + * probe async, returns a boolean or { present, note } */ const features = new Map(); -function feature({ key, title, section, probe }) { - features.set(key, { key, title, section, probe, present: undefined, note: undefined }); +function feature({ key, title, section, requirements, probe }) { + features.set(key, { key, title, section, requirements, probe, present: undefined, note: undefined }); } async function probeFeatures() { @@ -177,6 +178,11 @@ let networkFailures = 0; * a SKIP on a server the probe found does not implement it. */ function assert({ id, section, level, title, ok, expected, actual, skip, note, feature: featureKey }) { + // An identifier under a feature's prefix belongs to that feature whether or + // not the call site says so. Without this, one omitted `feature` charges a + // server for an option it never claimed. + const owner = [...features.values()].find((f) => f.requirements && id.startsWith(f.requirements)); + if (owner && featureKey !== owner.key) throw new Error(`${id} belongs to the feature "${owner.key}" and must say so`); if (featureKey !== undefined) { const f = features.get(featureKey); if (!f) throw new Error(`${id} names the unregistered feature "${featureKey}"`); @@ -647,7 +653,7 @@ async function main() { note: "KOReader sets PROGRESS_TIMEOUTS = {2, 5}; a slower answer is a failed push that gets queued for retry", }); assert({ - id: "K-PUT-202", section: "§5.4", level: "MUST", + id: "K-PUT-7", section: "§5.4", level: "MUST", title: "the server does not emit 202", ok: r.status !== 202, expected: "not 202 — api.json declares it but no client treats it as success", @@ -954,6 +960,10 @@ async function main() { }); } + // ===================================================== identifiers + section("§5.8 Optional identifier matching (PROPOSED — not merged upstream)"); + await identifierSuite(); + // ===================================================== optional endpoints section("§5.6/5.7 Optional server-only endpoints"); @@ -983,6 +993,530 @@ async function main() { return finish(); } +// -------------------------------------------------- identifiers (proposed) + +/** + * SPEC.md §5.8 — optional matching of one reading position by several document + * identifiers, as proposed in koreader/koreader-sync-server#55. THAT PULL + * REQUEST IS NOT MERGED. Every requirement here is MAY: a server that does not + * implement it skips the family, and one that does is held to all of it. + */ + +const ID_SECTION = "§5.8"; +const ID_XPOINTER = "/body/DocFragment[11]/body/div/p[7]/text().123"; + +// Fresh digests per run. An alias is created and never repointed, so a second +// run against reused ids would resolve through the first run's aliases and +// assert nothing. Every digest is 32 hex characters, which the reference +// server's read route can serve ([K-GET-1]). +const ID_RUN = `${Date.now().toString(36)}-${Math.floor(Math.random() * 1e9).toString(36)}`; +const dg = (tag) => md5(`${opts.document}:${ID_RUN}:${tag}`); + +const idList = (pairs) => pairs.map(([type, value]) => ({ type, value })); +const idQuery = (pairs) => pairs.map(([type, value]) => `${type}:${value}`).join(","); + +const idPut = (document, pairs, progress, percentage = 0.5) => + call("PUT", "/syncs/progress", { + body: { + document, + ...(pairs === null ? {} : { identifiers: Array.isArray(pairs) ? idList(pairs) : pairs }), + progress, + percentage, + device: "kosync-conformance", + device_id: "conformance-device-1", + }, + }); + +const idGet = (document, pairs) => + call("GET", `/syncs/progress/${document}${pairs === null ? "" : `?ids=${typeof pairs === "string" ? pairs : idQuery(pairs)}`}`); + +feature({ + key: "identifiers", + title: "multi-identifier document matching", + section: ID_SECTION, + requirements: "K-ID-", + // A push naming identifiers answers with `match` on a server that implements + // the proposal and with the pre-proposal body on one that does not. The + // proposal adds no endpoint, so there is nothing else to look at. + probe: async () => { + const id = dg("probe"); + const r = await idPut(id, [["content", id], ["structure", dg("probe-s")]], ID_XPOINTER, 0.11); + if (r.status === 200 && typeof r.json?.match === "string") return true; + return { + present: false, + note: `a push naming identifiers answered ${r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`}, with no \`match\` field`, + }; + }, +}); + +// Titles, in one place, because the skip path emits the whole family at once. +const ID_ASSERTIONS = [ + ["K-ID-1", "a push naming no identifiers returns today's body, with no match field"], + ["K-ID-1b", "a read naming no identifiers returns today's body and follows no alias"], + ["K-ID-2", "a push naming identifiers returns {document, match, timestamp}"], + ["K-ID-3", "match names the identifier type that resolved the lookup"], + ["K-ID-3b", "a read naming its own identifiers matches on the first of them"], + ["K-ID-4", "a renamed copy resolves through an identifier it shares, and gets the canonical digest"], + ["K-ID-5", "the reader's order of preference decides which identifier matches"], + ["K-ID-6", "progress_match differs from match when a weaker identifier let another copy write"], + ["K-ID-6b", "progress_match is none when the reader shares nothing with the writer"], + ["K-ID-6c", "a progress string written without identifiers is attributed to its own digest"], + ["K-ID-7", "an alias never shadows a document that exists in its own right"], + ["K-ID-12b", "a weak match does not register the identifiers ranked above it"], + ["K-ID-16", "an entry marked weak is accepted, and a strong match still adopts"], + ["K-ID-17", "a push resolving only through a weak entry does not adopt that record"], + ["K-ID-17b", "the weak entry is still registered, so a later read is seeded through it"], + ["K-ID-8", "an identifier list that does not name the document is rejected"], + ["K-ID-8b", "the document need not be first, and the record is created under it"], + ["K-ID-9", "more than 8 identifiers is rejected"], + ["K-ID-9d", "exactly 8 identifiers is accepted"], + ["K-ID-9b", "a malformed ids parameter is rejected"], + ["K-ID-9c", "a duplicate identifier type is rejected"], + ["K-ID-10", "one account's aliases do not resolve for another"], + ["K-ID-11", "a document unknown under every identifier still returns 200 with an empty body"], +]; + +const ID_TITLE = Object.fromEntries(ID_ASSERTIONS); + +async function identifierSuite() { + const XP = ID_XPOINTER; + const SEC = ID_SECTION; + const put = idPut; + const get = idGet; + const query = idQuery; + const rejected = (r) => r.status === 403 && r.json?.code === 2003; + + // The probe has already answered. A server that does not implement the + // feature gets the family as SKIP rather than thirty pointless requests. + if (!features.get("identifiers").present) { + for (const [id, title] of ID_ASSERTIONS) { + assert({ id, section: SEC, level: "MAY", feature: "identifiers", title }); + } + return; + } + + // --- [K-ID-1] a request that names none is answered exactly as before ----- + { + const plain = dg("plain"); + const r = await put(plain, null, XP, 0.32); + const keys = isObject(r.json) ? Object.keys(r.json).sort() : []; + assert({ + id: "K-ID-1", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-1"], + ok: r.status === 200 && keys.join(",") === "document,timestamp", + expected: '200 with exactly {"document":…,"timestamp":…}', + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: isObject(r.json) && (r.json.match !== undefined || r.json.progress_match !== undefined) + ? "the server volunteered a matching field to a request that asked for none; a client that did not opt in must see the response it has always seen" + : undefined, + }); + } + + { + const doc = dg("bare"); + const alias = dg("bare-s"); + await put(doc, [["content", doc], ["structure", alias]], XP, 0.32); + const r = await get(doc, null); + const keys = isObject(r.json) ? Object.keys(r.json).sort().join(",") : ""; + const viaAlias = await get(alias, null); + assert({ + id: "K-ID-1b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-1b"], + ok: r.status === 200 + && r.json?.match === undefined && r.json?.progress_match === undefined + && r.json?.document === doc + && viaAlias.status === 200 && isObject(viaAlias.json) && viaAlias.json.percentage === undefined, + expected: "the pre-proposal body for the document itself, and {} for a digest that is only an alias", + actual: r.error ?? viaAlias.error ?? `GET ${doc}: ${r.status} keys=[${keys}]; GET ${alias}: ${viaAlias.status} ${fmt(viaAlias.json ?? viaAlias.text)}`, + note: viaAlias.json?.percentage !== undefined + ? "a read that named no identifiers followed an alias. Aliases exist only for requests that opt in; following one silently changes what an existing client reads." + : undefined, + }); + } + + // --- the three-copy fixture ---------------------------------------------- + // original and repack share structure and metadata; edition shares only + // metadata. The same shape the reference implementation's own spec uses. + const c1 = dg("c1"), c2 = dg("c2"), c3 = dg("c3"); + const s1 = dg("s1"), s3 = dg("s3"), m = dg("m"); + const original = [["content", c1], ["structure", s1], ["metadata", m]]; + const repack = [["content", c2], ["structure", s1], ["metadata", m]]; + const edition = [["content", c3], ["structure", s3], ["metadata", m]]; + + { + const r = await put(c1, original, XP, 0.32); + const keys = isObject(r.json) ? Object.keys(r.json).sort().join(",") : ""; + assert({ + id: "K-ID-2", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-2"], + ok: r.status === 200 && keys === "document,match,timestamp" + && r.json.document === c1 && typeof r.json.timestamp === "number", + expected: `200 with exactly {"document":"${c1}","match":…,"timestamp":…}`, + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: keys.includes("progress_match") + ? "progress_match belongs to a read; a write has no writer to compare against" + : undefined, + }); + assert({ + id: "K-ID-3", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-3"], + ok: r.json?.match === "content", + expected: '"content" — the first identifier, under which the record was created', + actual: r.error ?? fmt(r.json?.match), + }); + } + + { + const r = await get(c1, original); + assert({ + id: "K-ID-3b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-3b"], + ok: r.status === 200 && r.json?.document === c1 + && r.json?.match === "content" && r.json?.progress_match === "content" + && r.json?.progress === XP, + expected: `200 with document ${c1}, match "content", progress_match "content", progress round-tripped`, + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + }); + } + + { + const r = await get(c2, repack); + assert({ + id: "K-ID-4", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-4"], + ok: r.status === 200 && r.json?.document === c1 && r.json?.match === "structure" + && r.json?.progress === XP, + expected: `200 with document ${c1} (the canonical digest), match "structure", the position written for the other copy`, + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: isObject(r.json) && r.json.percentage === undefined + ? "the renamed copy found nothing. Sharing a weaker identifier with a copy the account has already read is the whole point of the proposal." + : r.json?.document === c2 + ? "the canonical digest was not echoed; a client cannot then address the record directly on its next read" + : undefined, + }); + } + + { + // The same three identifiers, weakest first. A different one must win. + const weakest = await get(m, [["metadata", m], ["structure", s1], ["content", c1]]); + const strongest = await get(c1, original); + assert({ + id: "K-ID-5", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-5"], + ok: weakest.json?.match === "metadata" && strongest.json?.match === "content" + && weakest.json?.document === c1, + expected: 'match "metadata" when metadata is offered first, "content" when content is', + actual: weakest.error ?? strongest.error + ?? `metadata-first: ${fmt(weakest.json?.match)} (document ${fmt(weakest.json?.document)}); content-first: ${fmt(strongest.json?.match)}`, + note: "the list is a preference order, not a set; a server that resolves in its own order gives two clients different answers for the same book", + }); + } + + { + // A third edition shares only metadata, and writes the position. The + // reader still matches on its own content digest, so how the record was + // found and who wrote the progress string are different answers. + await put(c3, edition, "/body/DocFragment[3]/body/p[9]", 0.5); + const r = await get(c1, original); + assert({ + id: "K-ID-6", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-6"], + ok: r.json?.match === "content" && r.json?.progress_match === "metadata", + expected: 'match "content" (the record is the reader\'s own digest), progress_match "metadata" (all the writer shared)', + actual: r.error ?? `match ${fmt(r.json?.match)}, progress_match ${fmt(r.json?.progress_match)}`, + note: r.json?.progress_match === r.json?.match + ? "progress_match tracked match. They answer different questions: a reader can match a record on its own content digest and still be reading a different edition from the one that wrote the position, in which case the xpointer does not apply." + : undefined, + }); + } + + { + // d2 takes the position over through the structure digest, leaving a + // writer whose identifiers the metadata-only reader shares nothing with. + const d1 = dg("d1"), d2 = dg("d2"), ds = dg("ds"), dm = dg("dm"); + await put(d1, [["content", d1], ["structure", ds], ["metadata", dm]], XP, 0.2); + await put(d2, [["content", d2], ["structure", ds]], "/body/p[4]", 0.3); + const r = await get(dm, [["metadata", dm]]); + assert({ + id: "K-ID-6b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-6b"], + ok: r.json?.document === d1 && r.json?.match === "metadata" && r.json?.progress_match === "none", + expected: 'match "metadata", progress_match "none"', + actual: r.error ?? `document ${fmt(r.json?.document)}, match ${fmt(r.json?.match)}, progress_match ${fmt(r.json?.progress_match)}`, + note: "the reader has to be able to tell 'found your book, but a copy you know nothing about wrote this position' from 'this position is yours'", + }); + } + + { + const e1 = dg("e1"), es = dg("es"); + await put(e1, [["content", e1], ["structure", es]], XP, 0.2); + await put(e1, null, "/body/p[7]", 0.4); + const r = await get(e1, [["content", e1], ["structure", es]]); + assert({ + id: "K-ID-6c", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-6c"], + ok: r.json?.progress === "/body/p[7]" && r.json?.match === "content" + && r.json?.progress_match === "content", + expected: 'the later position, match "content", progress_match "content"', + actual: r.error ?? `progress ${fmt(r.json?.progress)}, match ${fmt(r.json?.match)}, progress_match ${fmt(r.json?.progress_match)}`, + note: "identifiers left on a record by an earlier client must stop being attributed once a client that named none overwrites the progress string, or the reader is told an xpointer is safe on the strength of a claim nobody made", + }); + } + + { + const f1 = dg("f1"), f2 = dg("f2"), fs = dg("fs"); + await put(f1, [["content", f1], ["structure", fs]], XP, 0.2); + // f2 is known separately first, so it keeps its own record. + await put(f2, [["content", f2]], "/body/p[2]", 0.1); + await put(f2, [["content", f2], ["structure", fs]], "/body/p[3]", 0.5); + const own = await get(f2, [["content", f2]]); + const other = await get(f1, [["content", f1]]); + assert({ + id: "K-ID-7", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-7"], + ok: own.json?.document === f2 && own.json?.progress === "/body/p[3]" + && other.json?.document === f1 && other.json?.progress === XP, + expected: `${f2} keeps its own record at /body/p[3], and ${f1} is untouched`, + actual: own.error ?? other.error + ?? `own: document ${fmt(own.json?.document)} progress ${fmt(own.json?.progress)}; other: document ${fmt(other.json?.document)} progress ${fmt(other.json?.progress)}`, + note: other.json?.progress !== XP + ? "a weak identifier moved one book's position onto another book's record. An alias must lose a race against a digest that is a document in its own right, not win it." + : undefined, + }); + } + + { + // Two unrelated books a library tagged alike: they share only the weakest + // identifier the client offers. + const shared = dg("shared"); + const b1 = dg("b1"), b1s = dg("b1s"); + const b2 = dg("b2"), b2s = dg("b2s"); + await put(b1, [["content", b1], ["structure", b1s], ["weak", shared]], XP, 0.8); + const merged = await put(b2, [["content", b2], ["structure", b2s], ["weak", shared]], "/body/p[1]", 0.01); + + // The tagging is corrected, so nothing is shared any more. The second book + // gets its own record back only if its content digest was never registered. + const corrected = [["content", b2], ["structure", b2s], ["weak", dg("shared2")]]; + const after = await put(b2, corrected, "/body/p[4]", 0.05); + const read = await get(b2, corrected); + assert({ + id: "K-ID-12b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-12b"], + ok: merged.json?.match === "weak" && merged.json?.document === b1 + && after.status === 200 && after.json?.document === b2 && after.json?.match === "content" + && read.json?.percentage === 0.05, + expected: `the weak match resolves to ${b1}; once it no longer matches, ${b2} is its own record again`, + actual: merged.error ?? after.error ?? read.error + ?? `weak match: ${fmt(merged.json)}; after correcting: ${fmt(after.json)}; read: ${fmt(read.json?.percentage)}`, + note: after.json?.document === b1 + ? "the caller's content digest was registered as an alias to a record found on its weakest identifier, so a wrong match is permanent: correcting what caused it does not free the copy. An alias created here is never repointed and nothing unlinks one." + : undefined, + }); + } + + { + // A weak entry may resolve a read, but it must not let a write claim a + // record that already exists. + const k1 = dg("wk1"), k1s = dg("wk1s"), shared = dg("wkm"); + const k2 = dg("wk2"), k2s = dg("wk2s"); + const weak = (v) => ({ type: "metadata", value: v, weak: true }); + const listFor = (c, st) => [["content", c], ["structure", st]]; + + const mk = (document, c, st, progress, pct) => + call("PUT", "/syncs/progress", { + body: { + document, + identifiers: [...listFor(c, st).map(([type, value]) => ({ type, value })), weak(shared)], + progress, percentage: pct, device: "kosync-conformance", device_id: "conformance-device-1", + }, + }); + + const first = await mk(k1, k1, k1s, XP, 0.8); + assert({ + id: "K-ID-16", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-16"], + ok: first.status === 200 && first.json?.document === k1 && first.json?.match === "content", + expected: `200 creating ${k1}, match "content"`, + actual: first.error ?? `${first.status} ${fmt(first.json ?? first.text)}`, + note: rejected(first) + ? "a `weak` flag on an identifier was rejected; it is an optional field on the entry, not a new type" + : undefined, + }); + + const second = await mk(k2, k2, k2s, "/body/p[1]", 0.01); + const original = await get(k1, listFor(k1, k1s)); + assert({ + id: "K-ID-17", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-17"], + ok: second.status === 200 && second.json?.document === k2 + && original.json?.progress === XP && original.json?.percentage === 0.8, + expected: `${k2} written under its own digest, and ${k1} left at ${XP}`, + actual: second.error ?? original.error + ?? `push answered ${fmt(second.json?.document)}; the first record now reads ${fmt(original.json?.progress)} at ${fmt(original.json?.percentage)}`, + note: second.json?.document === k1 + ? "the second work adopted the first's record through an identifier the caller marked weak, and overwrote the position stored there. A weak identifier seeds a reader; it does not claim a record." + : undefined, + }); + + // K-ID-17 on its own cannot tell "did not adopt" from "never registered the + // weak value at all": a server that skipped weak aliases would create k2, + // leave k1 alone, and pass it vacuously. Weakness governs adoption, not + // registration, so a third copy sharing only the weak value must still be + // seeded from the first record. + const k3 = dg("wk3"); + const seeded = await get(k3, [["content", k3], ["metadata", shared]]); + assert({ + id: "K-ID-17b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-17b"], + ok: seeded.status === 200 && seeded.json?.document === k1 + && seeded.json?.progress === XP && seeded.json?.progress_match === "metadata", + expected: `200 resolving to ${k1} through the weak value, progress_match "metadata"`, + actual: seeded.error ?? `${seeded.status} ${fmt(seeded.json ?? seeded.text)}`, + note: isObject(seeded.json) && seeded.json.percentage === undefined + ? "the weak identifier resolved nothing, so it was never registered as an alias. Weakness governs whether a write claims a record, not whether the value is indexed — without the alias there is nothing to seed a later copy from, which is the whole reason the type exists." + : undefined, + }); + } + + // --- validation ----------------------------------------------------------- + { + const g1 = dg("g1"), gs = dg("gs"); + const write = await put(g1, [["structure", gs], ["metadata", dg("gm")]], XP, 0.2); + const read = await get(g1, [["structure", gs]]); + assert({ + id: "K-ID-8", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-8"], + ok: rejected(write) && rejected(read), + expected: "403 with {code:2003} on both the push and the read", + actual: write.error ?? read.error + ?? `PUT ${write.status} ${fmt(write.json)}, GET ${read.status} ${fmt(read.json)}`, + note: write.status === 200 + ? "a list that names the document nowhere leaves the record unreachable by a client that sends no identifiers, which is every client today" + : undefined, + }); + } + + { + // The shape a filename-matching KOReader sends: the digest it is addressed + // by is the weakest thing it knows. + const w1 = dg("w1"), wc = dg("wc"), ws = dg("ws"); + const write = await put(w1, [["content", wc], ["structure", ws], ["filename", w1]], XP, 0.2); + const plain = await get(w1, null); + const matched = await get(w1, [["content", wc], ["structure", ws], ["filename", w1]]); + assert({ + id: "K-ID-8b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-8b"], + ok: write.status === 200 && write.json?.document === w1 && write.json?.match === "filename" + && plain.status === 200 && plain.json?.percentage !== undefined + && matched.json?.match === "content", + expected: `200 creating ${w1}, match "filename"; a read naming none finds it; a read naming all matches on "content"`, + actual: write.error ?? plain.error ?? matched.error + ?? `PUT ${write.status} ${fmt(write.json)}; plain GET ${plain.status} ${fmt(plain.json)}; matched GET match=${fmt(matched.json?.match)}`, + note: rejected(write) + ? "the document was rejected for not being first. Position is preference, not identity: a client whose document digest is its weakest identifier has to be able to rank the others above it." + : plain.json?.percentage === undefined + ? "the record was not created under `document`, so a client that names no identifiers cannot reach it" + : undefined, + }); + } + + { + const h1 = dg("h1"); + const many = [["content", h1]]; + for (let i = 1; i <= 8; i++) many.push([`t${i}`, dg(`h-${i}`)]); + const write = await put(h1, many, XP, 0.2); + const read = await get(h1, many); + assert({ + id: "K-ID-9", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-9"], + ok: rejected(write) && rejected(read), + expected: "403 with {code:2003} for a list of 9", + actual: write.error ?? read.error + ?? `PUT ${write.status} ${fmt(write.json)}, GET ${read.status} ${fmt(read.json)}`, + note: "each identifier is a redis lookup on the read path and an alias write on the write path; the cap is what keeps one request's cost bounded", + }); + const eight = many.slice(0, 8); + const ok8 = await put(h1, eight, XP, 0.2); + assert({ + id: "K-ID-9d", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-9d"], + ok: ok8.status === 200, + expected: "200 — the cap is 8, inclusive", + actual: ok8.error ?? `${ok8.status} ${fmt(ok8.json ?? ok8.text)}`, + }); + } + + { + const j1 = dg("j1"); + const read = await get(j1, j1); // no type, no colon + const empty = await get(j1, ""); // ids present, empty + assert({ + id: "K-ID-9b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-9b"], + ok: rejected(read) && (rejected(empty) || empty.status === 403), + expected: "403 with {code:2003}", + actual: read.error ?? `bare value: ${read.status} ${fmt(read.json)}; empty: ${empty.status} ${fmt(empty.json)}`, + note: "an unparseable list must not degrade into 'no identifiers named', which would silently answer a matching request with an unmatched body", + }); + } + + { + const k1 = dg("k1"); + const write = await put(k1, [["content", k1], ["content", dg("k2")]], XP, 0.2); + const read = await get(k1, [["content", k1], ["content", dg("k2")]]); + assert({ + id: "K-ID-9c", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-9c"], + ok: rejected(write) && rejected(read), + expected: "403 with {code:2003}", + actual: write.error ?? read.error + ?? `PUT ${write.status} ${fmt(write.json)}, GET ${read.status} ${fmt(read.json)}`, + note: "two values for one type is a client bug; resolving it in list order would make the answer depend on which the client happened to put first", + }); + } + + // --- isolation ------------------------------------------------------------ + if (USER2 && KEY2) { + const r = await call("GET", `/syncs/progress/${c2}?ids=${query(repack)}`, { + auth: false, + headers: { "x-auth-user": USER2, "x-auth-key": String(KEY2).toLowerCase() }, + }); + assert({ + id: "K-ID-10", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-10"], + ok: r.status === 200 && isObject(r.json) && r.json.percentage === undefined, + expected: "200 with no percentage — an alias is a per-account record, like the position it points at", + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: r.json?.percentage !== undefined + ? "the second account resolved the first account's alias. Identifiers are derived from file contents, so every account that owns the same book collides." + : undefined, + }); + } else { + assert({ + id: "K-ID-10", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-10"], + skip: true, + note: "pass --second-user and --second-password (or --second-key) to check it", + }); + } + + { + const unseen = dg(`unseen-${Date.now()}`); + const r = await get(unseen, [["content", unseen], ["structure", dg("unseen-s")]]); + assert({ + id: "K-ID-11", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-11"], + ok: r.status === 200 && isObject(r.json) && r.json.percentage === undefined + && r.json.match === undefined && r.json.progress_match === undefined, + expected: "200 with {} — no percentage, and no matching fields to report", + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: r.status !== 200 + ? "naming identifiers must not change [K-GET-3]: an unread book is still 200 with an empty body, not 404" + : undefined, + }); + } +} + // ------------------------------------------------------------------- report function finish() {