From 450403a844e9de48a06be0cf0226b2680d7218a3 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Tue, 22 Sep 2026 20:01:33 -0500 Subject: [PATCH 01/16] Describe and check optional multi-identifier matching MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SPEC.md gains §5.8: a reading position matched by several document identifiers, as proposed in koreader/koreader-sync-server#55. The section is pinned to the proposal's branch commit and marked as describing an unmerged pull request. Its eighteen requirements are MAY, under the optional feature `identifiers`. The verifier pushes one position naming identifiers and reads the answer: a `match` field means the feature is implemented and the family is checked, its absence means the family is recorded as SKIP. A server that does not implement §5.8 stays conformant; one that does is held to all of it. An assertion whose id falls under a feature's requirement prefix has to name that feature, and coverage.mjs fails if a proposed requirement belongs to no feature at all. Together those stop §5.8 reaching MUST and being charged to every server. Measured 2026-09-22: koreader/kosync:latest 49 passed, 0 MUST, 2 SHOULD, 18 skipped pid1/tsundoku main 47 passed, 0 MUST, 0 SHOULD, 20 skipped koreader-sync-server#55 at 0c0f5ad 69 passed, 0 MUST, 0 SHOULD, 0 skipped The results table also gains a Skipped column, and tsundoku's SHOULD-failure count is corrected from 2 to 0: that 2 was its skipped registration assertions. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 40 +++-- SPEC.md | 268 ++++++++++++++++++++++++++++++++ coverage.mjs | 39 ++++- verify.mjs | 424 ++++++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 753 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index feb0076..77488a4 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,14 @@ user's reading position quietly stops syncing. - **[`reference-server/`](reference-server/)** — how to run the reference implementation locally, which was itself undocumented. - **[`coverage.mjs`](coverage.mjs)** — fails if a requirement in `SPEC.md` has - neither a test nor a written reason why it cannot have one. Runs in CI, so - the spec and its test suite cannot drift apart quietly. + neither a test nor a written reason why it cannot have one, and if a proposed + requirement belongs to no optional feature. Runs in CI, so the spec and its + test suite cannot drift apart quietly. + +One section is not observed behaviour of a released server: **§5.8 describes an +open, unmerged pull request** and is pinned to a branch commit. It is an +optional feature, so a server that does not implement it skips those +requirements and stays conformant. This is **not** an official KOReader project and carries no endorsement from it. It is a description of observed behaviour. Where it and the reference @@ -68,16 +74,25 @@ writes to your server: see [Running the verifier](#running-the-verifier) below. ## Results -Measured 2026-09-21. `verify.mjs` against each server, with the profile flags +Measured 2026-09-22. `verify.mjs` against each server, with the profile flags each one's design calls for. -| Implementation | Version | Passed | MUST failed | SHOULD failed | Verdict | -|---|---|---:|---:|---:|---| -| `koreader/koreader-sync-server` (reference) | `koreader/kosync:latest`, OpenResty 1.29.2.3, gin 0.2.0 | 49 | **0** | 2 | conformant | -| `pid1/tsundoku` | branch `main` | 47 | **0** | 2 | conformant | +| Implementation | Version | Passed | MUST failed | SHOULD failed | Skipped | Verdict | +|---|---|---:|---:|---:|---:|---| +| `koreader/koreader-sync-server` (reference) | `koreader/kosync:latest`, OpenResty 1.29.2.3, gin 0.2.0 | 49 | **0** | 2 | 18 | conformant | +| `pid1/tsundoku` | branch `main` | 47 | **0** | 0 | 20 | conformant | + +Neither server implements §5.8, and nor does any server you can download, so both +runs skip its eighteen `[K-ID-…]` requirements. tsundoku's run skips two more, +the registration assertions, because it closes kosync self-registration by design +and the run declared `--registration-off`. -tsundoku's run skips the two registration assertions, because it closes kosync -self-registration by design and the run declared `--registration-off`. +Measured 2026-09-22 against `koreader/koreader-sync-server` PR #55 (branch +`multi-identifier-aliases` at `0c0f5ad`, built locally on OpenResty 1.29.2.3), +where the probe does find §5.8: **69 passed, 0 MUST failures, 0 SHOULD failures, +0 skipped**. The 51 outside §5.8 are what the published image is scored on, where +it passes 49 and fails two SHOULD — that branch also carries the unreleased fix +for the document-id defect below. The reference server's two SHOULD failures are a genuine defect in it, not a disagreement about the protocol: `PUT` accepts any document id, while `GET` @@ -166,6 +181,13 @@ of synthetic document ids derived from `--document`. Those ids are constant across runs by design, so a hundred runs leave five rows rather than five hundred. Point it at a test account. +The `identifiers` feature of §5.8 is the exception: it allocates **fresh digests +on every run**, because an alias, once created, is never repointed, so a second +run over the same digests would resolve through the first run's aliases and +assert nothing. Against a server that implements it, each run leaves roughly +twenty document and alias keys behind; against one that does not, it leaves the +single record its probe writes. + It **never probes `DELETE /users/me`**, because a conformant server would delete the account under test. diff --git a/SPEC.md b/SPEC.md index 0b9c45e..87debcc 100644 --- a/SPEC.md +++ b/SPEC.md @@ -31,6 +31,11 @@ This document describes the behaviour of those two programs. It is a KOReader project.** Where this document and the reference implementations disagree, the implementations are right and this document has a bug. +**§5.8 is the one exception, and it says so at its head.** It describes an +open, unmerged pull request rather than released behaviour, it is pinned to a +branch commit rather than to the table above, and nothing in it is part of +conformance. Every other section describes code that ships. + The reference server was **built and run** for this document: `docker.io/koreader/kosync:latest` (image `db2a16684d0b`, OpenResty 1.29.2.3, `X-Framework: gin/0.2.0`), under Podman on arm64, on 2026-09-21. Claims marked @@ -496,6 +501,9 @@ Push a reading position. | `device_id` | string | no | opaque per-device id | | `metadata` | object | no | see §7.6 | +An unmerged proposal adds one more optional field, `identifiers`; it is +described in §5.8 and is not part of conformance. + `api.json` lists `document`, `progress`, `percentage`, `device` and `device_id` as `required_params` and `metadata` as an `optional_param` (`koreader plugins/kosync.koplugin/api.json:23-44`) — but `required_params` is @@ -662,6 +670,10 @@ cannot serve on `GET`, or accept a wider character set on both. | `device_id` | string | stored (omitted if the pushing client never sent one) | | `timestamp` | number | stored | +An unmerged proposal adds an optional `ids` query parameter and two more +response fields to this endpoint; it is described in §5.8 and is not part of +conformance. A request that does not send `ids` is answered as below. + `syncs_controller.lua:193-226`. Each field is emitted only if the Redis hash returned a non-`null` value for it, and `document` is added **only if at least one other field is present** (`:221-224`): @@ -787,6 +799,245 @@ succeed (`syncs_controller.lua:64-72`). A retry with the now-stale key returns **[K-PWD-2]** Stored progress is preserved. Document keys are not touched. +### 5.8 Optional identifier matching — **PROPOSED, NOT MERGED** + +> **Status: proposed.** Everything in this section describes +> `koreader/koreader-sync-server` **pull request #55**, which is **open and not +> merged**. No released server implements it. If #55 is merged in a different +> shape, this section is wrong and not the server. +> +> Every `[K-ID-…]` requirement is `MAY`, under the optional feature +> `identifiers` (§12.5). Implementing the section is optional and a server that +> ignores the fields below stays conformant; a server that implements it is +> held to all of them, so `MUST` below means "if you implement this". + +| | | +|---|---| +| Upstream PR | [`koreader/koreader-sync-server#55`](https://github.com/koreader/koreader-sync-server/pull/55) | +| Wire shape described here | branch `multi-identifier-aliases` at `0c0f5ad4eedf31668a6ca7646ed8ae9a9c4ed853` (`pid1/koreader-sync-server`, 2026-09-22) | +| Built and run for this section | that branch under OpenResty 1.29.2.3 with Redis 8.10.2, `GIN_ENV=test`, on 2026-09-22 | + +Citations in this section are to that branch, not to the pinned reference +commit of §1.1, and are written `identifiers.lua:7` for +`lib/identifiers.lua`. + +**The problem.** `document` is one digest of one file (§8). A reader who +recompresses an EPUB, re-downloads it from another shop, or converts it gets a +different digest and loses the position, even though it is the same book. +There is no way to say "this file is also known as" in v1. + +**The shape.** Both progress endpoints take an optional, ordered list of +identifiers. A request that names none is answered exactly as it is today. + +#### Request + +`PUT /syncs/progress` takes an optional `identifiers` array beside the existing +fields: + +```json +{"document": "C1", + "identifiers": [{"type": "content", "value": "C1"}, + {"type": "structure", "value": "S1"}, + {"type": "metadata", "value": "M"}], + "percentage": 0.32, "progress": "/body/DocFragment[20]/body/p[22]", + "device": "my kpw"} +``` + +`GET /syncs/progress/:document` takes the same list flattened into one `ids` +query parameter, because a GET has no body and repeated query parameters are +not reliably ordered (`identifiers.lua:92-94`): + +``` +GET /syncs/progress/C1?ids=content:C1,structure:S1,metadata:M +``` + +An entry is a `{type, value}` pair. `type` is an **opaque label chosen by the +client**: the server stores and echoes it without interpreting it, so a new +kind of identifier needs no server change (`identifiers.lua:1-3`). + +| Rule | Value | Citation | +|---|---|---| +| Maximum entries per request | **8** | `identifiers.lua:7` | +| `type` pattern | `^[a-z][a-z0-9-]*$`, at most 32 characters | `identifiers.lua:8,14` | +| `value` pattern | `^[A-Za-z0-9][A-Za-z0-9-_.]*$`, at most 128 characters | `identifiers.lua:9,15` | +| Duplicate `type` in one list | rejected | `identifiers.lua:82-85`, `:119-122` | +| Empty list | rejected | `identifiers.lua:64-66` | +| Order | the client's **order of preference**, strongest first | `identifiers.lua:49-50` | + +**[K-ID-8]** The **first entry's `value` MUST equal `document`** +(`syncs_controller.lua:184-186`). A list that does not is rejected with **403** +code **2003**, on both the write and the read. This is what keeps `document` +meaning *"the identifier I would send if you only took one"*, so an old client +and a new one addressing the same file address the same record. + +**[K-ID-9]** A list of more than **8** entries is rejected with **403** code +**2003** (`identifiers.lua:67-69`, `:103-105`). Exactly 8 is accepted. Each +identifier costs a lookup on the read path and a possible alias write on the +write path, so the cap is what bounds one request's work. + +#### Response + +**[K-ID-2]** A `PUT` that names identifiers returns **200** with exactly three +fields (`syncs_controller.lua:500-504`): + +```json +{"document": "C1", "match": "content", "timestamp": 1790124995} +``` + +`document` is the **canonical** digest the record is stored under, which is not +necessarily the one the request sent. There is no `progress_match` on a write: +the writer is the request itself. + +**[K-ID-3]** `match` is the `type` of the identifier that **resolved the +lookup**, reported with the *caller's* own label. For a record being created it +is the first entry's type, since that is what the record is created under +(`syncs_controller.lua:98-100`). + +A `GET` that names identifiers returns today's fields plus two: + +| Field | Type | Meaning | +|---|---|---| +| `match` | string | the identifier type that **found** the record | +| `progress_match` | string | the strongest identifier the caller shares with whoever **wrote the current `progress` string**, or `"none"` | + +**[K-ID-6]** `match` and `progress_match` answer different questions and +routinely differ. A reader can match a record on its own content digest — +`match: "content"`, the record is literally its file — while the position +stored there was written by a different edition that only shared the metadata +digest: `progress_match: "metadata"`. The first says *we found your book*, the +second says *how much you should trust this xpointer*. Only `progress_match` +bears on whether the position can be followed +(`syncs_controller.lua:350-361`, `identifiers.lua:180-195`). + +`progress_match` is computed against the identifiers stored **beside the +progress string they were written with** (`identifiers_for`), not against +whatever the record last saw, so identifiers are never attributed to a string +their owner did not write (`syncs_controller.lua:426-431`, `:356`). Two +consequences: + +- **[K-ID-6b]** `"none"` when the caller and the writer share no value at all. +- **[K-ID-6c]** When the current `progress` was written by a request that named + **no** identifiers, the record is treated as written by the digest it is + stored under, so `progress_match` equals `match` (`syncs_controller.lua:360`). + +**[K-ID-1]** A request that names **no** identifiers gets **exactly today's +response**: `{document, timestamp}` on the write, the §5.5 body on the read, +**with neither `match` nor `progress_match`**, and the read **follows no +alias** — it looks up the literal `document` and nothing else +(`syncs_controller.lua:378-386`). An existing client sees no change of any +kind. + +**[K-ID-11]** A document that matches nothing, under any identifier offered, +is still **200 with an empty body** — `[K-GET-3]` is unaffected, and there are +no matching fields to report on a miss (`syncs_controller.lua:318-320`, +`:346-348`). + +#### Resolution and aliases + +Identifiers other than the record's own become **aliases**: per account, +`user::alias:` holding `:`. + +Resolution walks the caller's list **in order**, and for each entry tries the +value as a document first and as an alias second, stopping at the first hit +(`syncs_controller.lua:84-97` on the write, `:131-141` on the read). + +**[K-ID-5]** Because the walk is in the caller's order, the same three +identifiers offered strongest-first and weakest-first resolve through different +entries and report different `match` values. The list is a preference order, +not a set; a server that resolves in its own order gives two clients different +answers about the same book. + +**[K-ID-4]** A copy that shares any identifier with a record the account +already holds resolves to that record, and the response carries the +**canonical** digest so the next request can address it directly. + +**[K-ID-7]** An alias **MUST NOT shadow a document that exists in its own +right.** The walk tests `EXISTS document:` before consulting the alias +table, and an alias is created only for a value that is not already a document +(`syncs_controller.lua:87-88`, `:104`). A weak identifier can therefore fail to +match, but it cannot move one book's position onto another book's record. + +**[K-ID-12]** An alias is **created, never repointed.** The write path writes +an alias only when none exists, or when the one that exists points at a +document that is gone (`syncs_controller.lua:105-110`). A digest that has +resolved to a record keeps resolving to it. + +**[K-ID-10]** Aliases are namespaced per account, like positions +(`[K-ISO-1]`). **[K-ID-13]** They live under the account's `user::` +prefix, so `DELETE /users/me` removes them with everything else +(`[K-DEL-1]`). + +#### [measured] + +Against the branch above on 2026-09-22, `GIN_ENV=test`, plaintext listener, +account `reader`: + +``` +$ curl -X PUT .../syncs/progress -d '{"document":"C1", + "identifiers":[{"type":"content","value":"C1"}, + {"type":"structure","value":"S1"}, + {"type":"metadata","value":"M"}], + "percentage":0.32,"progress":"/body/DocFragment[20]/body/p[22]", + "device":"my kpw"}' +{"document":"C1","timestamp":1790124995,"match":"content"} + +# a recompressed copy: different content digest, same structure and metadata +$ curl '.../syncs/progress/C2?ids=content:C2,structure:S1,metadata:M' +{"percentage":0.32,"progress_match":"structure","device":"my kpw", + "match":"structure","document":"C1", + "progress":"\/body\/DocFragment[20]\/body\/p[22]","timestamp":1790124995} + +# a different edition writes, sharing only the metadata digest +$ curl -X PUT .../syncs/progress -d '{"document":"C3", + "identifiers":[{"type":"content","value":"C3"}, + {"type":"structure","value":"S3"}, + {"type":"metadata","value":"M"}], + "percentage":0.5,"progress":"/body/DocFragment[3]/body/p[9]","device":"pb"}' +{"document":"C1","timestamp":1790124995,"match":"metadata"} + +# the original copy reads again: found by its own digest, written by the other +$ curl '.../syncs/progress/C1?ids=content:C1,structure:S1,metadata:M' +{"percentage":0.5,"progress_match":"metadata","device":"pb","match":"content", + "document":"C1","progress":"\/body\/DocFragment[3]\/body\/p[9]", + "timestamp":1790124995} + +# the same read naming no identifiers: today's body, no new fields +$ curl '.../syncs/progress/C1' +{"percentage":0.5,"device":"pb","document":"C1", + "progress":"\/body\/DocFragment[3]\/body\/p[9]","timestamp":1790124995} + +# first identifier is not the document; nine identifiers +$ curl '.../syncs/progress/C1?ids=metadata:M' +403 {"message":"Invalid request","code":2003} +$ curl -X PUT ... nine entries ... +403 {"message":"Invalid request","code":2003} +``` + +Redis after that sequence — one document, four aliases, all under the account: + +``` +user:reader:alias:C3 user:reader:alias:M +user:reader:alias:S1 user:reader:alias:S3 +user:reader:document:C1 user:reader:key +``` + +#### Running the checks + +```bash +node verify.mjs --base-url http://127.0.0.1:8080 \ + --user alice --password hunter2 +``` + +There is no flag. The verifier pushes one position naming identifiers and reads +the answer: a `match` field means the feature is implemented and the family is +checked, and its absence means the family is recorded as `SKIP`. + +The identifier checks allocate **fresh digests on every run**, unlike the rest +of the suite, which reuses stable ids by design. They have to: `[K-ID-12]` says +an alias is never repointed, so a second run over the same digests would +resolve through the first run's aliases and assert nothing. + + --- ## 6. Error model @@ -1723,6 +1974,11 @@ Full options, profile flags and the recipe for standing up the reference server are in [`README.md`](README.md) and [`reference-server/`](reference-server/). +The `[K-ID-…]` requirements of §5.8 do not decide that verdict. They describe an +unmerged proposal and are `MAY`, under the optional feature `identifiers` +(§12.5): a server that does not implement them skips them and stays conformant, +and a server that does is held to all of them. + ### 12.2 Checklist for a new server Work down this list; each row names the requirement it satisfies. @@ -1791,6 +2047,12 @@ It fails if a requirement is defined here with neither an assertion nor a §12.4 row, and it fails if `verify.mjs` asserts an identifier this document does not define. It runs in CI. +Requirements defined in a section whose heading says **proposed** are counted +separately, and `coverage.mjs` additionally fails if one of them belongs to no +optional feature. `verify.mjs` refuses at run time to record a requirement under +a feature's prefix without naming that feature, so a proposed requirement cannot +reach `MUST` and be charged to every server. + ### 12.4 Requirements the verifier cannot check | Requirement | Why not | @@ -1812,6 +2074,8 @@ define. It runs in CI. | `[K-FLD-12]` `timestamp` is epoch seconds | Asserted as `[K-FLD-14]`, which checks the value is within a day of now and so would catch milliseconds. | | `[K-FLD-8]` device self-detection | Client-side. The server precondition — that `device_id` round-trips — is `[K-FLD-7]`. | | `[K-SYNC-1]` … `[K-SYNC-6]` conflict resolution | Entirely client-side; the server never compares anything. The verifier asserts the server-side preconditions instead: `timestamp` present (`[K-FLD-13]`), in seconds (`[K-FLD-14]`), server-generated (`[K-PUT-4]`), and last-write-wins (`[K-PUT-5]`). | +| `[K-ID-12]` an alias is created, never repointed | Not observable over HTTP: the alias table is private, and every wire consequence of it is already asserted — resolution is stable (`[K-ID-4]`) and a document that exists in its own right is never shadowed (`[K-ID-7]`). Proposed; see §5.8. | +| `[K-ID-13]` aliases are removed with the account | Same reason as `[K-DEL-1]`: probing `DELETE /users/me` would delete the account under test. Proposed; see §5.8. | | `[K-DOC-1]` … `[K-DOC-10]` document identity | Not a server behaviour: `document` is opaque to a server, and four of the eight surveyed servers never compute it (§11.1). Checked by `vectors/check.mjs` against the golden vectors of §8.6, not over HTTP. | @@ -2002,6 +2266,10 @@ questioning the code's existence (`syncs_controller.lua:17`): | Is there any client that sends a `timestamp` on PUT? | Not among those surveyed. The reference server would ignore it. | | Is `202` reachable on any server? | Not among those surveyed. | | Is error code `100` reachable? | **No**, in practice — §3.3. Every HTTP client sends a default `Accept`, so the reference server answers `101`. | +| Will `identifiers` (§5.8) be merged, and in this shape? | **Open.** `koreader/koreader-sync-server#55` is unmerged as of 2026-09-22. §5.8 is pinned to a branch commit for that reason. | +| What identifier **types** should a client send? | **Undecided, and deliberately not specified.** The server treats `type` as an opaque label and never interprets it (`identifiers.lua:1-3`), so the set can grow without a server change. `content`, `structure` and `metadata` are the branch's own examples; none is computed by any released KOReader build. | +| How should a client pick the `document` when it holds several identifiers? | **Open.** §5.8 requires only that the first entry equal `document`. Which identifier that ought to be is a client decision no implementation has yet made. | +| Does an alias ever expire? | **No, in the proposal.** Aliases are removed only with the account (`[K-ID-13]`). A long-lived account accumulates one alias per distinct identifier it has ever offered, and nothing in the branch bounds that. | | Does the official deployment carry the `[K-GET-1a]` router defect? | **[unverified]** — the published image does (measured). Whether `sync.koreader.rocks` runs that image is unknown. | --- diff --git a/coverage.mjs b/coverage.mjs index 51f4dac..96dc1bf 100644 --- a/coverage.mjs +++ b/coverage.mjs @@ -6,8 +6,14 @@ * being prose. This fails if either: * * - a requirement is defined in SPEC.md but is neither asserted by verify.mjs - * nor listed in SPEC.md §12.4 as untestable, with a reason; or - * - verify.mjs asserts an identifier that SPEC.md does not define. + * nor listed in SPEC.md §12.4 as untestable, with a reason; + * - verify.mjs asserts an identifier that SPEC.md does not define; or + * - a PROPOSED requirement belongs to no optional feature, which would hold + * every server to an unmerged proposal. + * + * A requirement is proposed if it is defined under a heading whose text says + * so. Those are counted separately, because "SPEC.md defines N requirements" + * should not quietly grow by things no released server implements. * * node coverage.mjs # report and exit non-zero on a gap * node coverage.mjs --list # also print the full mapping @@ -21,7 +27,27 @@ const list = process.argv.includes("--list"); // A requirement is DEFINED by a bold identifier at the head of a statement: // **[K-GET-3]** ... or **[K-GET-3] An unknown document ...** -const defined = new Set([...spec.matchAll(/\*\*\[(K-[A-Z0-9-]+)\]/g)].map((m) => m[1])); +const DEFINITION = /\*\*\[(K-[A-Z0-9-]+)\]/g; +const defined = new Set([...spec.matchAll(DEFINITION)].map((m) => m[1])); + +// PROPOSED requirements: defined under a heading that says "proposed". The +// heading owns everything up to the next heading of the same or higher level. +const proposed = new Set(); +for (const m of spec.matchAll(/^(#{2,4}) .*$/gm)) { + if (!/proposed/i.test(m[0])) continue; + const rest = spec.slice(m.index + m[0].length); + const next = rest.search(new RegExp(`^#{1,${m[1].length}} `, "m")); + for (const d of (next === -1 ? rest : rest.slice(0, next)).matchAll(DEFINITION)) proposed.add(d[1]); +} + +// verify.mjs registers each optional feature with the identifier prefix its +// requirements share, and refuses at run time to record one of them without +// naming the feature. What that cannot catch is a proposed requirement attached +// to no feature at all, which would be checked as MUST against every server. +const optional = [...verifier.matchAll(/requirements:\s*"(K-[A-Z0-9-]+)"/g)].map((m) => m[1]); +const unattached = [...proposed] + .filter((id) => !optional.some((prefix) => id.startsWith(prefix))) + .sort(); // ASSERTED by the verifier. const asserted = new Set([...verifier.matchAll(/id:\s*"(K-[A-Za-z0-9-]+)"/g)].map((m) => m[1])); @@ -91,6 +117,7 @@ const counts = rows.reduce((a, [, s]) => ((a[s] = (a[s] ?? 0) + 1), a), {}); process.stdout.write( `SPEC.md defines ${defined.size} requirements: ` + `${counts.asserted ?? 0} asserted, ${counts.untestable ?? 0} declared untestable, ${gaps.length} unaccounted for.\n` + + `${proposed.size} of them are PROPOSED (unmerged upstream), and belong to an optional feature that a server is probed for.\n` + `verify.mjs makes ${asserted.size} distinct assertions.\n`, ); @@ -101,6 +128,12 @@ if (gaps.length) { for (const id of gaps) process.stdout.write(` ${id}\n`); process.stdout.write(`\nEither assert them in verify.mjs, or add a row to SPEC.md §12.4 saying why not.\n`); } +if (unattached.length) { + bad = true; + process.stdout.write(`\nProposed requirements that belong to no optional feature:\n`); + for (const id of unattached) process.stdout.write(` ${id}\n`); + process.stdout.write(`\nRegister a feature in verify.mjs whose \`requirements\` prefix covers them, or\nevery server is held to an unmerged proposal.\n`); +} if (undefinedIds.length) { bad = true; process.stdout.write(`\nAsserted by verify.mjs but not defined in SPEC.md:\n`); diff --git a/verify.mjs b/verify.mjs index a845ea4..8cd1a54 100644 --- a/verify.mjs +++ b/verify.mjs @@ -123,15 +123,16 @@ const KEY2 = USER2 ? (opts.secondKey ?? (opts.secondPassword && md5(opts.secondP * to the verifier to be scored correctly will be described wrongly, and the * resulting page of failures says nothing about the server. * - * key short identifier, used in the report and in `--json` - * title what the feature is, in a noun phrase - * section the section of SPEC.md that defines it - * probe async, returns a boolean or { present, note } + * key short identifier, used in the report and in `--json` + * title what the feature is, in a noun phrase + * section the section of SPEC.md that defines it + * requirements the identifier prefix its requirements share + * probe async, returns a boolean or { present, note } */ const features = new Map(); -function feature({ key, title, section, probe }) { - features.set(key, { key, title, section, probe, present: undefined, note: undefined }); +function feature({ key, title, section, requirements, probe }) { + features.set(key, { key, title, section, requirements, probe, present: undefined, note: undefined }); } async function probeFeatures() { @@ -177,6 +178,11 @@ let networkFailures = 0; * a SKIP on a server the probe found does not implement it. */ function assert({ id, section, level, title, ok, expected, actual, skip, note, feature: featureKey }) { + // An identifier under a feature's prefix belongs to that feature whether or + // not the call site says so. Without this, one omitted `feature` charges a + // server for an option it never claimed. + const owner = [...features.values()].find((f) => f.requirements && id.startsWith(f.requirements)); + if (owner && featureKey !== owner.key) throw new Error(`${id} belongs to the feature "${owner.key}" and must say so`); if (featureKey !== undefined) { const f = features.get(featureKey); if (!f) throw new Error(`${id} names the unregistered feature "${featureKey}"`); @@ -954,6 +960,10 @@ async function main() { }); } + // ===================================================== identifiers + section("§5.8 Optional identifier matching (PROPOSED — not merged upstream)"); + await identifierSuite(); + // ===================================================== optional endpoints section("§5.6/5.7 Optional server-only endpoints"); @@ -983,6 +993,408 @@ async function main() { return finish(); } +// -------------------------------------------------- identifiers (proposed) + +/** + * SPEC.md §5.8 — optional matching of one reading position by several document + * identifiers, as proposed in koreader/koreader-sync-server#55. THAT PULL + * REQUEST IS NOT MERGED. Every requirement here is MAY: a server that does not + * implement it skips the family, and one that does is held to all of it. + */ + +const ID_SECTION = "§5.8"; +const ID_XPOINTER = "/body/DocFragment[11]/body/div/p[7]/text().123"; + +// Fresh digests per run. An alias is created and never repointed, so a second +// run against reused ids would resolve through the first run's aliases and +// assert nothing. Every digest is 32 hex characters, which the reference +// server's read route can serve ([K-GET-1]). +const ID_RUN = `${Date.now().toString(36)}-${Math.floor(Math.random() * 1e9).toString(36)}`; +const dg = (tag) => md5(`${opts.document}:${ID_RUN}:${tag}`); + +const idList = (pairs) => pairs.map(([type, value]) => ({ type, value })); +const idQuery = (pairs) => pairs.map(([type, value]) => `${type}:${value}`).join(","); + +const idPut = (document, pairs, progress, percentage = 0.5) => + call("PUT", "/syncs/progress", { + body: { + document, + ...(pairs === null ? {} : { identifiers: Array.isArray(pairs) ? idList(pairs) : pairs }), + progress, + percentage, + device: "kosync-conformance", + device_id: "conformance-device-1", + }, + }); + +const idGet = (document, pairs) => + call("GET", `/syncs/progress/${document}${pairs === null ? "" : `?ids=${typeof pairs === "string" ? pairs : idQuery(pairs)}`}`); + +feature({ + key: "identifiers", + title: "multi-identifier document matching", + section: ID_SECTION, + requirements: "K-ID-", + // A push naming identifiers answers with `match` on a server that implements + // the proposal and with the pre-proposal body on one that does not. The + // proposal adds no endpoint, so there is nothing else to look at. + probe: async () => { + const id = dg("probe"); + const r = await idPut(id, [["content", id], ["structure", dg("probe-s")]], ID_XPOINTER, 0.11); + if (r.status === 200 && typeof r.json?.match === "string") return true; + return { + present: false, + note: `a push naming identifiers answered ${r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`}, with no \`match\` field`, + }; + }, +}); + +// Titles, in one place, because the skip path emits the whole family at once. +const ID_ASSERTIONS = [ + ["K-ID-1", "a push naming no identifiers returns today's body, with no match field"], + ["K-ID-1b", "a read naming no identifiers returns today's body and follows no alias"], + ["K-ID-2", "a push naming identifiers returns {document, match, timestamp}"], + ["K-ID-3", "match names the identifier type that resolved the lookup"], + ["K-ID-3b", "a read naming its own identifiers matches on the first of them"], + ["K-ID-4", "a renamed copy resolves through an identifier it shares, and gets the canonical digest"], + ["K-ID-5", "the reader's order of preference decides which identifier matches"], + ["K-ID-6", "progress_match differs from match when a weaker identifier let another copy write"], + ["K-ID-6b", "progress_match is none when the reader shares nothing with the writer"], + ["K-ID-6c", "a progress string written without identifiers is attributed to its own digest"], + ["K-ID-7", "an alias never shadows a document that exists in its own right"], + ["K-ID-8", "an identifier list whose first entry is not the document is rejected"], + ["K-ID-9", "more than 8 identifiers is rejected"], + ["K-ID-9d", "exactly 8 identifiers is accepted"], + ["K-ID-9b", "a malformed ids parameter is rejected"], + ["K-ID-9c", "a duplicate identifier type is rejected"], + ["K-ID-10", "one account's aliases do not resolve for another"], + ["K-ID-11", "a document unknown under every identifier still returns 200 with an empty body"], +]; + +const ID_TITLE = Object.fromEntries(ID_ASSERTIONS); + +async function identifierSuite() { + const XP = ID_XPOINTER; + const SEC = ID_SECTION; + const put = idPut; + const get = idGet; + const query = idQuery; + const rejected = (r) => r.status === 403 && r.json?.code === 2003; + + // The probe has already answered. A server that does not implement the + // feature gets the family as SKIP rather than thirty pointless requests. + if (!features.get("identifiers").present) { + for (const [id, title] of ID_ASSERTIONS) { + assert({ id, section: SEC, level: "MAY", feature: "identifiers", title }); + } + return; + } + + // --- [K-ID-1] a request that names none is answered exactly as before ----- + { + const plain = dg("plain"); + const r = await put(plain, null, XP, 0.32); + const keys = isObject(r.json) ? Object.keys(r.json).sort() : []; + assert({ + id: "K-ID-1", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-1"], + ok: r.status === 200 && keys.join(",") === "document,timestamp", + expected: '200 with exactly {"document":…,"timestamp":…}', + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: isObject(r.json) && (r.json.match !== undefined || r.json.progress_match !== undefined) + ? "the server volunteered a matching field to a request that asked for none; a client that did not opt in must see the response it has always seen" + : undefined, + }); + } + + { + const doc = dg("bare"); + const alias = dg("bare-s"); + await put(doc, [["content", doc], ["structure", alias]], XP, 0.32); + const r = await get(doc, null); + const keys = isObject(r.json) ? Object.keys(r.json).sort().join(",") : ""; + const viaAlias = await get(alias, null); + assert({ + id: "K-ID-1b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-1b"], + ok: r.status === 200 + && r.json?.match === undefined && r.json?.progress_match === undefined + && r.json?.document === doc + && viaAlias.status === 200 && isObject(viaAlias.json) && viaAlias.json.percentage === undefined, + expected: "the pre-proposal body for the document itself, and {} for a digest that is only an alias", + actual: r.error ?? viaAlias.error ?? `GET ${doc}: ${r.status} keys=[${keys}]; GET ${alias}: ${viaAlias.status} ${fmt(viaAlias.json ?? viaAlias.text)}`, + note: viaAlias.json?.percentage !== undefined + ? "a read that named no identifiers followed an alias. Aliases exist only for requests that opt in; following one silently changes what an existing client reads." + : undefined, + }); + } + + // --- the three-copy fixture ---------------------------------------------- + // original and repack share structure and metadata; edition shares only + // metadata. The same shape the reference implementation's own spec uses. + const c1 = dg("c1"), c2 = dg("c2"), c3 = dg("c3"); + const s1 = dg("s1"), s3 = dg("s3"), m = dg("m"); + const original = [["content", c1], ["structure", s1], ["metadata", m]]; + const repack = [["content", c2], ["structure", s1], ["metadata", m]]; + const edition = [["content", c3], ["structure", s3], ["metadata", m]]; + + { + const r = await put(c1, original, XP, 0.32); + const keys = isObject(r.json) ? Object.keys(r.json).sort().join(",") : ""; + assert({ + id: "K-ID-2", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-2"], + ok: r.status === 200 && keys === "document,match,timestamp" + && r.json.document === c1 && typeof r.json.timestamp === "number", + expected: `200 with exactly {"document":"${c1}","match":…,"timestamp":…}`, + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: keys.includes("progress_match") + ? "progress_match belongs to a read; a write has no writer to compare against" + : undefined, + }); + assert({ + id: "K-ID-3", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-3"], + ok: r.json?.match === "content", + expected: '"content" — the first identifier, under which the record was created', + actual: r.error ?? fmt(r.json?.match), + }); + } + + { + const r = await get(c1, original); + assert({ + id: "K-ID-3b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-3b"], + ok: r.status === 200 && r.json?.document === c1 + && r.json?.match === "content" && r.json?.progress_match === "content" + && r.json?.progress === XP, + expected: `200 with document ${c1}, match "content", progress_match "content", progress round-tripped`, + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + }); + } + + { + const r = await get(c2, repack); + assert({ + id: "K-ID-4", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-4"], + ok: r.status === 200 && r.json?.document === c1 && r.json?.match === "structure" + && r.json?.progress === XP, + expected: `200 with document ${c1} (the canonical digest), match "structure", the position written for the other copy`, + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: isObject(r.json) && r.json.percentage === undefined + ? "the renamed copy found nothing. Sharing a weaker identifier with a copy the account has already read is the whole point of the proposal." + : r.json?.document === c2 + ? "the canonical digest was not echoed; a client cannot then address the record directly on its next read" + : undefined, + }); + } + + { + // The same three identifiers, weakest first. A different one must win. + const weakest = await get(m, [["metadata", m], ["structure", s1], ["content", c1]]); + const strongest = await get(c1, original); + assert({ + id: "K-ID-5", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-5"], + ok: weakest.json?.match === "metadata" && strongest.json?.match === "content" + && weakest.json?.document === c1, + expected: 'match "metadata" when metadata is offered first, "content" when content is', + actual: weakest.error ?? strongest.error + ?? `metadata-first: ${fmt(weakest.json?.match)} (document ${fmt(weakest.json?.document)}); content-first: ${fmt(strongest.json?.match)}`, + note: "the list is a preference order, not a set; a server that resolves in its own order gives two clients different answers for the same book", + }); + } + + { + // A third edition shares only metadata, and writes the position. The + // reader still matches on its own content digest, so how the record was + // found and who wrote the progress string are different answers. + await put(c3, edition, "/body/DocFragment[3]/body/p[9]", 0.5); + const r = await get(c1, original); + assert({ + id: "K-ID-6", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-6"], + ok: r.json?.match === "content" && r.json?.progress_match === "metadata", + expected: 'match "content" (the record is the reader\'s own digest), progress_match "metadata" (all the writer shared)', + actual: r.error ?? `match ${fmt(r.json?.match)}, progress_match ${fmt(r.json?.progress_match)}`, + note: r.json?.progress_match === r.json?.match + ? "progress_match tracked match. They answer different questions: a reader can match a record on its own content digest and still be reading a different edition from the one that wrote the position, in which case the xpointer does not apply." + : undefined, + }); + } + + { + // d2 takes the position over through the structure digest, leaving a + // writer whose identifiers the metadata-only reader shares nothing with. + const d1 = dg("d1"), d2 = dg("d2"), ds = dg("ds"), dm = dg("dm"); + await put(d1, [["content", d1], ["structure", ds], ["metadata", dm]], XP, 0.2); + await put(d2, [["content", d2], ["structure", ds]], "/body/p[4]", 0.3); + const r = await get(dm, [["metadata", dm]]); + assert({ + id: "K-ID-6b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-6b"], + ok: r.json?.document === d1 && r.json?.match === "metadata" && r.json?.progress_match === "none", + expected: 'match "metadata", progress_match "none"', + actual: r.error ?? `document ${fmt(r.json?.document)}, match ${fmt(r.json?.match)}, progress_match ${fmt(r.json?.progress_match)}`, + note: "the reader has to be able to tell 'found your book, but a copy you know nothing about wrote this position' from 'this position is yours'", + }); + } + + { + const e1 = dg("e1"), es = dg("es"); + await put(e1, [["content", e1], ["structure", es]], XP, 0.2); + await put(e1, null, "/body/p[7]", 0.4); + const r = await get(e1, [["content", e1], ["structure", es]]); + assert({ + id: "K-ID-6c", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-6c"], + ok: r.json?.progress === "/body/p[7]" && r.json?.match === "content" + && r.json?.progress_match === "content", + expected: 'the later position, match "content", progress_match "content"', + actual: r.error ?? `progress ${fmt(r.json?.progress)}, match ${fmt(r.json?.match)}, progress_match ${fmt(r.json?.progress_match)}`, + note: "identifiers left on a record by an earlier client must stop being attributed once a client that named none overwrites the progress string, or the reader is told an xpointer is safe on the strength of a claim nobody made", + }); + } + + { + const f1 = dg("f1"), f2 = dg("f2"), fs = dg("fs"); + await put(f1, [["content", f1], ["structure", fs]], XP, 0.2); + // f2 is known separately first, so it keeps its own record. + await put(f2, [["content", f2]], "/body/p[2]", 0.1); + await put(f2, [["content", f2], ["structure", fs]], "/body/p[3]", 0.5); + const own = await get(f2, [["content", f2]]); + const other = await get(f1, [["content", f1]]); + assert({ + id: "K-ID-7", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-7"], + ok: own.json?.document === f2 && own.json?.progress === "/body/p[3]" + && other.json?.document === f1 && other.json?.progress === XP, + expected: `${f2} keeps its own record at /body/p[3], and ${f1} is untouched`, + actual: own.error ?? other.error + ?? `own: document ${fmt(own.json?.document)} progress ${fmt(own.json?.progress)}; other: document ${fmt(other.json?.document)} progress ${fmt(other.json?.progress)}`, + note: other.json?.progress !== XP + ? "a weak identifier moved one book's position onto another book's record. An alias must lose a race against a digest that is a document in its own right, not win it." + : undefined, + }); + } + + // --- validation ----------------------------------------------------------- + { + const g1 = dg("g1"), gs = dg("gs"); + const write = await put(g1, [["structure", gs], ["content", g1]], XP, 0.2); + const read = await get(g1, [["structure", gs]]); + assert({ + id: "K-ID-8", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-8"], + ok: rejected(write) && rejected(read), + expected: "403 with {code:2003} on both the push and the read", + actual: write.error ?? read.error + ?? `PUT ${write.status} ${fmt(write.json)}, GET ${read.status} ${fmt(read.json)}`, + note: write.status === 200 + ? "`document` and the first identifier must be the same string, or `document` stops meaning 'the identifier I would send if you only took one' and an old client and a new one address different records" + : undefined, + }); + } + + { + const h1 = dg("h1"); + const many = [["content", h1]]; + for (let i = 1; i <= 8; i++) many.push([`t${i}`, dg(`h-${i}`)]); + const write = await put(h1, many, XP, 0.2); + const read = await get(h1, many); + assert({ + id: "K-ID-9", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-9"], + ok: rejected(write) && rejected(read), + expected: "403 with {code:2003} for a list of 9", + actual: write.error ?? read.error + ?? `PUT ${write.status} ${fmt(write.json)}, GET ${read.status} ${fmt(read.json)}`, + note: "each identifier is a redis lookup on the read path and an alias write on the write path; the cap is what keeps one request's cost bounded", + }); + const eight = many.slice(0, 8); + const ok8 = await put(h1, eight, XP, 0.2); + assert({ + id: "K-ID-9d", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-9d"], + ok: ok8.status === 200, + expected: "200 — the cap is 8, inclusive", + actual: ok8.error ?? `${ok8.status} ${fmt(ok8.json ?? ok8.text)}`, + }); + } + + { + const j1 = dg("j1"); + const read = await get(j1, j1); // no type, no colon + const empty = await get(j1, ""); // ids present, empty + assert({ + id: "K-ID-9b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-9b"], + ok: rejected(read) && (rejected(empty) || empty.status === 403), + expected: "403 with {code:2003}", + actual: read.error ?? `bare value: ${read.status} ${fmt(read.json)}; empty: ${empty.status} ${fmt(empty.json)}`, + note: "an unparseable list must not degrade into 'no identifiers named', which would silently answer a matching request with an unmatched body", + }); + } + + { + const k1 = dg("k1"); + const write = await put(k1, [["content", k1], ["content", dg("k2")]], XP, 0.2); + const read = await get(k1, [["content", k1], ["content", dg("k2")]]); + assert({ + id: "K-ID-9c", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-9c"], + ok: rejected(write) && rejected(read), + expected: "403 with {code:2003}", + actual: write.error ?? read.error + ?? `PUT ${write.status} ${fmt(write.json)}, GET ${read.status} ${fmt(read.json)}`, + note: "two values for one type is a client bug; resolving it in list order would make the answer depend on which the client happened to put first", + }); + } + + // --- isolation ------------------------------------------------------------ + if (USER2 && KEY2) { + const r = await call("GET", `/syncs/progress/${c2}?ids=${query(repack)}`, { + auth: false, + headers: { "x-auth-user": USER2, "x-auth-key": String(KEY2).toLowerCase() }, + }); + assert({ + id: "K-ID-10", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-10"], + ok: r.status === 200 && isObject(r.json) && r.json.percentage === undefined, + expected: "200 with no percentage — an alias is a per-account record, like the position it points at", + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: r.json?.percentage !== undefined + ? "the second account resolved the first account's alias. Identifiers are derived from file contents, so every account that owns the same book collides." + : undefined, + }); + } else { + assert({ + id: "K-ID-10", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-10"], + skip: true, + note: "pass --second-user and --second-password (or --second-key) to check it", + }); + } + + { + const unseen = dg(`unseen-${Date.now()}`); + const r = await get(unseen, [["content", unseen], ["structure", dg("unseen-s")]]); + assert({ + id: "K-ID-11", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-11"], + ok: r.status === 200 && isObject(r.json) && r.json.percentage === undefined + && r.json.match === undefined && r.json.progress_match === undefined, + expected: "200 with {} — no percentage, and no matching fields to report", + actual: r.error ?? `${r.status} ${fmt(r.json ?? r.text)}`, + note: r.status !== 200 + ? "naming identifiers must not change [K-GET-3]: an unread book is still 200 with an empty body, not 404" + : undefined, + }); + } +} + // ------------------------------------------------------------------- report function finish() { From 2e393cb97094fa95b44df3adc563576d1237296b Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 09:15:11 -0500 Subject: [PATCH 02/16] Require the document among the identifiers, not first The list is the client's order of preference, so pinning `document` to the first position makes preference and identity the same thing. KOReader set to match documents by filename is addressed by the weakest digest it holds, and would report `progress_match: "filename"` for a copy whose content is byte-identical. K-ID-8 now asserts the list contains an entry equal to `document`. K-ID-8b asserts that entry need not be first, that the record is created under `document`, and that a read naming no identifiers still finds it. Measured against koreader/koreader-sync-server#55: the new assertion fails on the branch before the corresponding fix and passes after it. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 18 ++++++++++++++---- verify.mjs | 31 ++++++++++++++++++++++++++++--- 2 files changed, 42 insertions(+), 7 deletions(-) diff --git a/SPEC.md b/SPEC.md index 87debcc..263c264 100644 --- a/SPEC.md +++ b/SPEC.md @@ -864,11 +864,21 @@ kind of identifier needs no server change (`identifiers.lua:1-3`). | Empty list | rejected | `identifiers.lua:64-66` | | Order | the client's **order of preference**, strongest first | `identifiers.lua:49-50` | -**[K-ID-8]** The **first entry's `value` MUST equal `document`** -(`syncs_controller.lua:184-186`). A list that does not is rejected with **403** +**[K-ID-8]** The list **MUST contain an entry whose `value` equals `document`** +(`syncs_controller.lua:197-202`). A list that does not is rejected with **403** code **2003**, on both the write and the read. This is what keeps `document` -meaning *"the identifier I would send if you only took one"*, so an old client -and a new one addressing the same file address the same record. +meaning *"the identifier I would send if you only took one"*: the record stays +addressable by the digest a client naming no identifiers sends, and a read that +names none follows no alias to reach it (`[K-ID-1b]`). + +**[K-ID-8b]** That entry **need not be first.** Position in the list carries +preference, not identity. When nothing resolves, the record is created under +`document` rather than under the first entry, and `match` is that entry's type +(`syncs_controller.lua:104-109`). Pinning it to the first position would force a +client whose `document` digest is its weakest identifier to offer that one first +and be matched on it — which is what KOReader sends when its document matching +is set to the filename, and it would report `progress_match: "filename"` for a +copy whose content is byte-identical. **[K-ID-9]** A list of more than **8** entries is rejected with **403** code **2003** (`identifiers.lua:67-69`, `:103-105`). Exactly 8 is accepted. Each diff --git a/verify.mjs b/verify.mjs index 8cd1a54..0ef1dda 100644 --- a/verify.mjs +++ b/verify.mjs @@ -1062,7 +1062,8 @@ const ID_ASSERTIONS = [ ["K-ID-6b", "progress_match is none when the reader shares nothing with the writer"], ["K-ID-6c", "a progress string written without identifiers is attributed to its own digest"], ["K-ID-7", "an alias never shadows a document that exists in its own right"], - ["K-ID-8", "an identifier list whose first entry is not the document is rejected"], + ["K-ID-8", "an identifier list that does not name the document is rejected"], + ["K-ID-8b", "the document need not be first, and the record is created under it"], ["K-ID-9", "more than 8 identifiers is rejected"], ["K-ID-9d", "exactly 8 identifiers is accepted"], ["K-ID-9b", "a malformed ids parameter is rejected"], @@ -1283,7 +1284,7 @@ async function identifierSuite() { // --- validation ----------------------------------------------------------- { const g1 = dg("g1"), gs = dg("gs"); - const write = await put(g1, [["structure", gs], ["content", g1]], XP, 0.2); + const write = await put(g1, [["structure", gs], ["metadata", dg("gm")]], XP, 0.2); const read = await get(g1, [["structure", gs]]); assert({ id: "K-ID-8", section: SEC, level: "MAY", feature: "identifiers", @@ -1293,11 +1294,35 @@ async function identifierSuite() { actual: write.error ?? read.error ?? `PUT ${write.status} ${fmt(write.json)}, GET ${read.status} ${fmt(read.json)}`, note: write.status === 200 - ? "`document` and the first identifier must be the same string, or `document` stops meaning 'the identifier I would send if you only took one' and an old client and a new one address different records" + ? "a list that names the document nowhere leaves the record unreachable by a client that sends no identifiers, which is every client today" : undefined, }); } + { + // The shape a filename-matching KOReader sends: the digest it is addressed + // by is the weakest thing it knows. + const w1 = dg("w1"), wc = dg("wc"), ws = dg("ws"); + const write = await put(w1, [["content", wc], ["structure", ws], ["filename", w1]], XP, 0.2); + const plain = await get(w1, null); + const matched = await get(w1, [["content", wc], ["structure", ws], ["filename", w1]]); + assert({ + id: "K-ID-8b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-8b"], + ok: write.status === 200 && write.json?.document === w1 && write.json?.match === "filename" + && plain.status === 200 && plain.json?.percentage !== undefined + && matched.json?.match === "content", + expected: `200 creating ${w1}, match "filename"; a read naming none finds it; a read naming all matches on "content"`, + actual: write.error ?? plain.error ?? matched.error + ?? `PUT ${write.status} ${fmt(write.json)}; plain GET ${plain.status} ${fmt(plain.json)}; matched GET match=${fmt(matched.json?.match)}`, + note: rejected(write) + ? "the document was rejected for not being first. Position is preference, not identity: a client whose document digest is its weakest identifier has to be able to rank the others above it." + : plain.json?.percentage === undefined + ? "the record was not created under `document`, so a client that names no identifiers cannot reach it" + : undefined, + }); + } + { const h1 = dg("h1"); const many = [["content", h1]]; From 1b5858ca9ea1fb5d26f6e08ec657609bb1b66dc7 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 09:52:28 -0500 Subject: [PATCH 03/16] Add the type registry, and the rule that a weak match glues nothing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `type` is opaque to the server and not to clients: `progress_match` decides whether one device follows another's position, so a label without a recipe is not interoperable. The registry gives `content`, `structure` and `filename` theirs, and K-ID-14 says an unrecognised label is never sufficient to follow a position. `structure` covers no file contents. CrossPoint's EPUB optimizer re-encodes images, injects a stylesheet into every chapter and rewrites `img src`, so every entry's bytes change while the spine does not; a digest over entry CRCs fails against the tool this exists for. The spine href list is also what an xpointer counts. There is no `metadata` type. It is the only identifier that can match two different files, and an alias is never repointed. K-ID-12b, asserted: an identifier ranked above the one that matched is not registered. Measured against koreader/koreader-sync-server#55 — the assertion fails on the branch before that fix and passes after it. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++- verify.mjs | 30 ++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 263c264..d1e2456 100644 --- a/SPEC.md +++ b/SPEC.md @@ -855,6 +855,43 @@ An entry is a `{type, value}` pair. `type` is an **opaque label chosen by the client**: the server stores and echoes it without interpreting it, so a new kind of identifier needs no server change (`identifiers.lua:1-3`). +##### Type registry + +`type` is opaque **to the server**, which stores and echoes it without +interpreting it. It is not opaque to clients: `progress_match` is a trust +signal, and a client decides whether to follow another device's `progress` +string on the strength of a shared label. Two clients that compute the same +label differently therefore disagree about what they have agreed on, so a label +without a recipe is not interoperable. + +| `type` | value | changes when | +|---|---|---| +| `content` | the document's own digest over the file bytes, as §8 derives it | any byte of the file changes | +| `structure` | md5 over the OPF `dc:identifier` when the container has one, followed by every spine item's `href` in reading order, `\n`-separated | the edition, or the chapter list or its order, changes | +| `filename` | md5 of the file name, as §8.5 derives it | the file is renamed | + +**[K-ID-14]** A client **MUST NOT** treat a `progress_match` type it does not +recognise as sufficient to follow a `progress` string. An unrecognised label +carries no guarantee about the file the position was written against, so it is +`percentage` that applies, not the stored position. + +`structure` deliberately covers **no file contents.** The tools that motivate +this section rewrite them: CrossPoint's EPUB optimizer re-encodes images to +JPEG through a canvas, injects a stylesheet into every chapter's `` and +rewrites `` when it splits an image, so every entry's bytes change +while the spine does not. A digest over entry CRCs survives recompression and +fails against the very tool this exists for. The spine href list is also exactly +what an xpointer counts — `/body/DocFragment[N]` is the Nth spine entry — so a +`structure` match says precisely that the position's chapter index means the +same thing here. + +**There is no `metadata` type.** A digest over title and author is the only +identifier that can match two genuinely different files, and `[K-ID-12]` never +repoints an alias. Two books a library has tagged alike merge on it, and until +`[K-ID-12b]` the merge outlived the tagging being corrected. The recompression +case is covered by `structure`, so the type bought cross-edition matching at the +cost of the only unrecoverable failure in the section. + | Rule | Value | Citation | |---|---|---| | Maximum entries per request | **8** | `identifiers.lua:7` | @@ -969,9 +1006,21 @@ match, but it cannot move one book's position onto another book's record. **[K-ID-12]** An alias is **created, never repointed.** The write path writes an alias only when none exists, or when the one that exists points at a -document that is gone (`syncs_controller.lua:105-110`). A digest that has +document that is gone (`syncs_controller.lua:115-120`). A digest that has resolved to a record keeps resolving to it. +**[K-ID-12b]** An identifier the caller ranks **above** the one that matched is +**not registered at all** (`syncs_controller.lua:110`). Matching on a weak +identifier is a guess; registering the caller's stronger digests as aliases to +that guess would make a wrong one permanent, because `[K-ID-12]` never repoints +and no endpoint unlinks. Two books a library has tagged alike share only their +weakest identifier, and without this rule the second one pushed takes the first +one's record, overwrites the position stored there, and cannot be separated +again by correcting the tagging — its own content digest has been glued to the +other record on the way through. Confining a wrong guess to the identifier that +made it is what makes it recoverable. On a **create** the record is the caller's +own, so every identifier it offers describes it and all of them are registered. + **[K-ID-10]** Aliases are namespaced per account, like positions (`[K-ISO-1]`). **[K-ID-13]** They live under the account's `user::` prefix, so `DELETE /users/me` removes them with everything else @@ -2085,6 +2134,7 @@ reach `MUST` and be charged to every server. | `[K-FLD-8]` device self-detection | Client-side. The server precondition — that `device_id` round-trips — is `[K-FLD-7]`. | | `[K-SYNC-1]` … `[K-SYNC-6]` conflict resolution | Entirely client-side; the server never compares anything. The verifier asserts the server-side preconditions instead: `timestamp` present (`[K-FLD-13]`), in seconds (`[K-FLD-14]`), server-generated (`[K-PUT-4]`), and last-write-wins (`[K-PUT-5]`). | | `[K-ID-12]` an alias is created, never repointed | Not observable over HTTP: the alias table is private, and every wire consequence of it is already asserted — resolution is stable (`[K-ID-4]`) and a document that exists in its own right is never shadowed (`[K-ID-7]`). Proposed; see §5.8. | +| `[K-ID-14]` a client must not follow a position on an unrecognised `progress_match` | Client-side. The server reports the type and acts on it in no way, so nothing over the wire distinguishes a client that honours this from one that ignores it. Checkable only by inspecting a client. Proposed; see §5.8. | | `[K-ID-13]` aliases are removed with the account | Same reason as `[K-DEL-1]`: probing `DELETE /users/me` would delete the account under test. Proposed; see §5.8. | | `[K-DOC-1]` … `[K-DOC-10]` document identity | Not a server behaviour: `document` is opaque to a server, and four of the eight surveyed servers never compute it (§11.1). Checked by `vectors/check.mjs` against the golden vectors of §8.6, not over HTTP. | diff --git a/verify.mjs b/verify.mjs index 0ef1dda..d1119b6 100644 --- a/verify.mjs +++ b/verify.mjs @@ -1062,6 +1062,7 @@ const ID_ASSERTIONS = [ ["K-ID-6b", "progress_match is none when the reader shares nothing with the writer"], ["K-ID-6c", "a progress string written without identifiers is attributed to its own digest"], ["K-ID-7", "an alias never shadows a document that exists in its own right"], + ["K-ID-12b", "a weak match does not register the identifiers ranked above it"], ["K-ID-8", "an identifier list that does not name the document is rejected"], ["K-ID-8b", "the document need not be first, and the record is created under it"], ["K-ID-9", "more than 8 identifiers is rejected"], @@ -1281,6 +1282,35 @@ async function identifierSuite() { }); } + { + // Two unrelated books a library tagged alike: they share only the weakest + // identifier the client offers. + const shared = dg("shared"); + const b1 = dg("b1"), b1s = dg("b1s"); + const b2 = dg("b2"), b2s = dg("b2s"); + await put(b1, [["content", b1], ["structure", b1s], ["weak", shared]], XP, 0.8); + const merged = await put(b2, [["content", b2], ["structure", b2s], ["weak", shared]], "/body/p[1]", 0.01); + + // The tagging is corrected, so nothing is shared any more. The second book + // gets its own record back only if its content digest was never registered. + const corrected = [["content", b2], ["structure", b2s], ["weak", dg("shared2")]]; + const after = await put(b2, corrected, "/body/p[4]", 0.05); + const read = await get(b2, corrected); + assert({ + id: "K-ID-12b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-12b"], + ok: merged.json?.match === "weak" && merged.json?.document === b1 + && after.status === 200 && after.json?.document === b2 && after.json?.match === "content" + && read.json?.percentage === 0.05, + expected: `the weak match resolves to ${b1}; once it no longer matches, ${b2} is its own record again`, + actual: merged.error ?? after.error ?? read.error + ?? `weak match: ${fmt(merged.json)}; after correcting: ${fmt(after.json)}; read: ${fmt(read.json?.percentage)}`, + note: after.json?.document === b1 + ? "the caller's content digest was registered as an alias to a record found on its weakest identifier, so a wrong match is permanent: correcting what caused it does not free the copy. An alias created here is never repointed and nothing unlinks one." + : undefined, + }); + } + // --- validation ----------------------------------------------------------- { const g1 = dg("g1"), gs = dg("gs"); From 8249f5747c7fb55e5d997c6669655da3b01b27de Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 09:55:31 -0500 Subject: [PATCH 04/16] Pin the structure recipe so two clients compute it identically Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index d1e2456..1bb4b0b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -867,7 +867,7 @@ without a recipe is not interoperable. | `type` | value | changes when | |---|---|---| | `content` | the document's own digest over the file bytes, as §8 derives it | any byte of the file changes | -| `structure` | md5 over the OPF `dc:identifier` when the container has one, followed by every spine item's `href` in reading order, `\n`-separated | the edition, or the chapter list or its order, changes | +| `structure` | md5 over the spine, as defined below | the edition, or the chapter list or its order, changes | | `filename` | md5 of the file name, as §8.5 derives it | the file is renamed | **[K-ID-14]** A client **MUST NOT** treat a `progress_match` type it does not @@ -875,6 +875,23 @@ recognise as sufficient to follow a `progress` string. An unrecognised label carries no guarantee about the file the position was written against, so it is `percentage` that applies, not the stored position. +`structure` is computed from the OPF, and two clients have to compute it +identically or the label means nothing. The recipe, exactly: + +1. Take the `` element's `unique-identifier` attribute and find the + `` whose `id` matches it. Failing that, take the first + ``. Trim it. If the result is non-empty, it is the first line. +2. Walk `` in document order. For each ``, resolve its `idref` + against `` and take that item's `href` **exactly as written** in + the attribute: not percent-decoded, not resolved against the OPF directory, + not reduced to a basename. Strip a `#fragment` if one is present. Each is a + line, in spine order. +3. Join the lines with `\n`, with no trailing newline, and take the md5 of the + UTF-8 bytes. + +A container with no spine, or one that is not an OPF-bearing archive, has no +`structure` digest and the identifier is omitted rather than guessed. + `structure` deliberately covers **no file contents.** The tools that motivate this section rewrite them: CrossPoint's EPUB optimizer re-encodes images to JPEG through a canvas, injects a stylesheet into every chapter's `` and From a859ddb4a4823905ccfa33b892642644b4202894 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 09:58:07 -0500 Subject: [PATCH 05/16] Record today's results The K-ID family is twenty assertions, so a server without the feature skips twenty. Re-measured against the reference image, tsundoku main, and #55 at 49dbd38. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 77488a4..52875bd 100644 --- a/README.md +++ b/README.md @@ -74,22 +74,22 @@ writes to your server: see [Running the verifier](#running-the-verifier) below. ## Results -Measured 2026-09-22. `verify.mjs` against each server, with the profile flags +Measured 2026-09-23. `verify.mjs` against each server, with the profile flags each one's design calls for. | Implementation | Version | Passed | MUST failed | SHOULD failed | Skipped | Verdict | |---|---|---:|---:|---:|---:|---| -| `koreader/koreader-sync-server` (reference) | `koreader/kosync:latest`, OpenResty 1.29.2.3, gin 0.2.0 | 49 | **0** | 2 | 18 | conformant | -| `pid1/tsundoku` | branch `main` | 47 | **0** | 0 | 20 | conformant | +| `koreader/koreader-sync-server` (reference) | `koreader/kosync:latest`, OpenResty 1.29.2.3, gin 0.2.0 | 49 | **0** | 2 | 20 | conformant | +| `pid1/tsundoku` | branch `main` | 47 | **0** | 0 | 22 | conformant | Neither server implements §5.8, and nor does any server you can download, so both -runs skip its eighteen `[K-ID-…]` requirements. tsundoku's run skips two more, -the registration assertions, because it closes kosync self-registration by design -and the run declared `--registration-off`. +runs skip its twenty `[K-ID-…]` requirements. tsundoku's run skips two more, the +registration assertions, because it closes kosync self-registration by design and +the run declared `--registration-off`. -Measured 2026-09-22 against `koreader/koreader-sync-server` PR #55 (branch -`multi-identifier-aliases` at `0c0f5ad`, built locally on OpenResty 1.29.2.3), -where the probe does find §5.8: **69 passed, 0 MUST failures, 0 SHOULD failures, +Measured 2026-09-23 against `koreader/koreader-sync-server` PR #55 (branch +`multi-identifier-aliases` at `49dbd38`, built locally on OpenResty 1.29.2.3), +where the probe does find §5.8: **71 passed, 0 MUST failures, 0 SHOULD failures, 0 skipped**. The 51 outside §5.8 are what the published image is scored on, where it passes 49 and fails two SHOULD — that branch also carries the unreleased fix for the document-id defect below. From 8fbc958c6e8f822532db292b43a8c8d2ba6cf508 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 10:10:11 -0500 Subject: [PATCH 06/16] Pin the parts of the structure recipe two parsers would read differently A hand-rolled parser keeps XML entities in an href and a DOM parser decodes them; one matches element names literally and the other on the local name. Neither notices it made a choice. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index 1bb4b0b..a4a2625 100644 --- a/SPEC.md +++ b/SPEC.md @@ -884,14 +884,31 @@ identically or the label means nothing. The recipe, exactly: 2. Walk `` in document order. For each ``, resolve its `idref` against `` and take that item's `href` **exactly as written** in the attribute: not percent-decoded, not resolved against the OPF directory, - not reduced to a basename. Strip a `#fragment` if one is present. Each is a - line, in spine order. + not reduced to a basename, and **not XML-entity-decoded** — an `href` written + `a&b/ch2.xhtml` contributes those eighteen characters, not sixteen. Strip + a `#fragment` if one is present. Each is a line, in spine order. 3. Join the lines with `\n`, with no trailing newline, and take the md5 of the UTF-8 bytes. A container with no spine, or one that is not an OPF-bearing archive, has no `structure` digest and the identifier is omitted rather than guessed. +The rest of the recipe, because two implementations will otherwise each make a +defensible choice and disagree: + +- **Element names are matched on the local name**, so `` and + `` are the same element and `` under a default Dublin Core + namespace counts as ``. Attribute values are not namespaced. +- **An `` whose `idref` resolves to no manifest item contributes no + line**, and does not invalidate the digest. +- **`linear="no"` items are included.** They are spine entries, and the + renderer's `DocFragment` numbering counts them. +- `` is looked up within ``, `` within ``, and + `` within ``. +- **The OPF is the first ``** in `META-INF/container.xml`, whatever + its `media-type`. Its `full-path` is used as an archive member name as + written; if no member matches, there is no digest rather than a guessed one. + `structure` deliberately covers **no file contents.** The tools that motivate this section rewrite them: CrossPoint's EPUB optimizer re-encodes images to JPEG through a canvas, injects a stylesheet into every chapter's `` and From a7add4b6d0540dbf949a9b980bdfec1a276fce9a Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 10:18:14 -0500 Subject: [PATCH 07/16] Require strongest-first ordering, and regenerate the worked example K-ID-12b is expressed in terms of list position, so a list that opens with its weakest identifier matches at the first position and registers everything behind it. Measured: two books sharing only a weak identifier, offered weakest-first, merge and stay merged after that identifier is corrected. K-ID-5b makes the ordering a client MUST; a server cannot check it. The section's worked example was dated before both of today's rules and taught the behaviour they forbid. Re-run against 49dbd38: two aliases, not four. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 95 ++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 61 insertions(+), 34 deletions(-) diff --git a/SPEC.md b/SPEC.md index a4a2625..830bbc3 100644 --- a/SPEC.md +++ b/SPEC.md @@ -838,7 +838,7 @@ fields: {"document": "C1", "identifiers": [{"type": "content", "value": "C1"}, {"type": "structure", "value": "S1"}, - {"type": "metadata", "value": "M"}], + {"type": "filename", "value": "F"}], "percentage": 0.32, "progress": "/body/DocFragment[20]/body/p[22]", "device": "my kpw"} ``` @@ -848,7 +848,7 @@ query parameter, because a GET has no body and repeated query parameters are not reliably ordered (`identifiers.lua:92-94`): ``` -GET /syncs/progress/C1?ids=content:C1,structure:S1,metadata:M +GET /syncs/progress/C1?ids=content:C1,structure:S1,filename:F ``` An entry is a `{type, value}` pair. `type` is an **opaque label chosen by the @@ -933,7 +933,7 @@ cost of the only unrecoverable failure in the section. | `value` pattern | `^[A-Za-z0-9][A-Za-z0-9-_.]*$`, at most 128 characters | `identifiers.lua:9,15` | | Duplicate `type` in one list | rejected | `identifiers.lua:82-85`, `:119-122` | | Empty list | rejected | `identifiers.lua:64-66` | -| Order | the client's **order of preference**, strongest first | `identifiers.lua:49-50` | +| Order | the client's **order of preference**, strongest first — required, see `[K-ID-5b]` | `identifiers.lua:49-50` | **[K-ID-8]** The list **MUST contain an entry whose `value` equals `document`** (`syncs_controller.lua:197-202`). A list that does not is rejected with **403** @@ -984,8 +984,8 @@ A `GET` that names identifiers returns today's fields plus two: **[K-ID-6]** `match` and `progress_match` answer different questions and routinely differ. A reader can match a record on its own content digest — `match: "content"`, the record is literally its file — while the position -stored there was written by a different edition that only shared the metadata -digest: `progress_match: "metadata"`. The first says *we found your book*, the +stored there was written by a different book that only shared the file name: +`progress_match: "filename"`. The first says *we found your book*, the second says *how much you should trust this xpointer*. Only `progress_match` bears on whether the position can be followed (`syncs_controller.lua:350-361`, `identifiers.lua:180-195`). @@ -1028,6 +1028,18 @@ entries and report different `match` values. The list is a preference order, not a set; a server that resolves in its own order gives two clients different answers about the same book. +**[K-ID-5b]** A client **MUST** order its identifiers strongest first — most +specific to the file in hand, least specific last. The registry table above is +in descending strength order for the types it names. This is not a stylistic +preference: `[K-ID-12b]` is the protection against a wrong match becoming +permanent, it is expressed in terms of list position, and a list that opens with +its weakest identifier matches at the first position and so registers every +identifier behind it. **Measured 2026-09-23** against the branch above with the +`[K-ID-12b]` rule in place: two books sharing only a weak identifier, offered +weakest-first, merge and stay merged after the weak identifier is corrected — the +same failure `[K-ID-12b]` exists to prevent, reproduced with nothing but a +different list order. A server cannot check this (§12.4). + **[K-ID-4]** A copy that shares any identifier with a record the account already holds resolves to that record, and the response carries the **canonical** digest so the next request can address it directly. @@ -1055,6 +1067,14 @@ other record on the way through. Confining a wrong guess to the identifier that made it is what makes it recoverable. On a **create** the record is the caller's own, so every identifier it offers describes it and all of them are registered. +What this rule recovers is the **copy**, not the position: the intruding push +still overwrote what was stored on the other record, and nothing restores that. +It also has a cost on the benign path. A recompressed copy that matched on +`structure` never has its own `content` digest registered, on that push or any +later one, so it depends on the weaker identifier continuing to match and is +orphaned if the spine later changes. That trade is deliberate, and it is a trade. +The rule is only as good as `[K-ID-5b]`, which a server cannot enforce. + **[K-ID-10]** Aliases are namespaced per account, like positions (`[K-ISO-1]`). **[K-ID-13]** They live under the account's `user::` prefix, so `DELETE /users/me` removes them with everything else @@ -1062,58 +1082,64 @@ prefix, so `DELETE /users/me` removes them with everything else #### [measured] -Against the branch above on 2026-09-22, `GIN_ENV=test`, plaintext listener, -account `reader`: +Against the branch above at `49dbd38` on 2026-09-23, `GIN_ENV=production`, +plaintext listener, account `reader`. `C1` is the original, `C2` a recompressed +copy of it, and `C3` a different book that happens to carry the same file name. ``` $ curl -X PUT .../syncs/progress -d '{"document":"C1", "identifiers":[{"type":"content","value":"C1"}, {"type":"structure","value":"S1"}, - {"type":"metadata","value":"M"}], + {"type":"filename","value":"F"}], "percentage":0.32,"progress":"/body/DocFragment[20]/body/p[22]", "device":"my kpw"}' -{"document":"C1","timestamp":1790124995,"match":"content"} +{"document":"C1","timestamp":1790176644,"match":"content"} -# a recompressed copy: different content digest, same structure and metadata -$ curl '.../syncs/progress/C2?ids=content:C2,structure:S1,metadata:M' -{"percentage":0.32,"progress_match":"structure","device":"my kpw", - "match":"structure","document":"C1", - "progress":"\/body\/DocFragment[20]\/body\/p[22]","timestamp":1790124995} +# the recompressed copy: different content digest, same spine +$ curl '.../syncs/progress/C2?ids=content:C2,structure:S1,filename:F' +{"percentage":0.32,"device":"my kpw","progress_match":"structure", + "progress":"\/body\/DocFragment[20]\/body\/p[22]","document":"C1", + "timestamp":1790176644,"match":"structure"} -# a different edition writes, sharing only the metadata digest +# a different book sharing only the file name writes, and takes the record $ curl -X PUT .../syncs/progress -d '{"document":"C3", "identifiers":[{"type":"content","value":"C3"}, {"type":"structure","value":"S3"}, - {"type":"metadata","value":"M"}], + {"type":"filename","value":"F"}], "percentage":0.5,"progress":"/body/DocFragment[3]/body/p[9]","device":"pb"}' -{"document":"C1","timestamp":1790124995,"match":"metadata"} +{"document":"C1","timestamp":1790176644,"match":"filename"} -# the original copy reads again: found by its own digest, written by the other -$ curl '.../syncs/progress/C1?ids=content:C1,structure:S1,metadata:M' -{"percentage":0.5,"progress_match":"metadata","device":"pb","match":"content", - "document":"C1","progress":"\/body\/DocFragment[3]\/body\/p[9]", - "timestamp":1790124995} +# the original reads again: found by its own digest, written by the other +$ curl '.../syncs/progress/C1?ids=content:C1,structure:S1,filename:F' +{"percentage":0.5,"device":"pb","document":"C1", + "progress":"\/body\/DocFragment[3]\/body\/p[9]","progress_match":"filename", + "timestamp":1790176644,"match":"content"} # the same read naming no identifiers: today's body, no new fields $ curl '.../syncs/progress/C1' -{"percentage":0.5,"device":"pb","document":"C1", - "progress":"\/body\/DocFragment[3]\/body\/p[9]","timestamp":1790124995} +{"percentage":0.5,"device":"pb","document":"C1","timestamp":1790176644, + "progress":"\/body\/DocFragment[3]\/body\/p[9]"} -# first identifier is not the document; nine identifiers -$ curl '.../syncs/progress/C1?ids=metadata:M' -403 {"message":"Invalid request","code":2003} -$ curl -X PUT ... nine entries ... -403 {"message":"Invalid request","code":2003} +# a list that names the document nowhere +$ curl '.../syncs/progress/C1?ids=filename:F' +403 {"code":2003,"message":"Invalid request"} ``` -Redis after that sequence — one document, four aliases, all under the account: +Redis after that sequence — one document and **two** aliases, all under the +account: ``` -user:reader:alias:C3 user:reader:alias:M -user:reader:alias:S1 user:reader:alias:S3 +user:reader:alias:F user:reader:alias:S1 user:reader:document:C1 user:reader:key ``` +`C3` and `S3` are absent, and that is `[K-ID-12b]`. The third push matched on +`filename`, the weakest identifier it offered, so the two digests it ranked above +that one were not registered. Had they been, the second book would be reachable +only through the first book's record for good, and correcting the file name would +not undo it. What the rule does not undo is the overwrite: `C1` now holds the +other book's position, and the `my kpw` reading at 0.32 is gone. + #### Running the checks ```bash @@ -2167,7 +2193,8 @@ reach `MUST` and be charged to every server. | `[K-FLD-12]` `timestamp` is epoch seconds | Asserted as `[K-FLD-14]`, which checks the value is within a day of now and so would catch milliseconds. | | `[K-FLD-8]` device self-detection | Client-side. The server precondition — that `device_id` round-trips — is `[K-FLD-7]`. | | `[K-SYNC-1]` … `[K-SYNC-6]` conflict resolution | Entirely client-side; the server never compares anything. The verifier asserts the server-side preconditions instead: `timestamp` present (`[K-FLD-13]`), in seconds (`[K-FLD-14]`), server-generated (`[K-PUT-4]`), and last-write-wins (`[K-PUT-5]`). | -| `[K-ID-12]` an alias is created, never repointed | Not observable over HTTP: the alias table is private, and every wire consequence of it is already asserted — resolution is stable (`[K-ID-4]`) and a document that exists in its own right is never shadowed (`[K-ID-7]`). Proposed; see §5.8. | +| `[K-ID-12]` an alias is created, never repointed | Not observable over HTTP: the alias table is private, and every wire consequence of it is already asserted — resolution is stable (`[K-ID-4]`), a document that exists in its own right is never shadowed (`[K-ID-7]`), and an identifier above the match is never registered (`[K-ID-12b]`). Proposed; see §5.8. | +| `[K-ID-5b]` identifiers are ordered strongest first | Client-side, and not inferable from the wire: the server has no idea what any type means, so a weakest-first list is indistinguishable from a strongest-first one whose strongest identifier happens to be the one that matched. Checkable only by inspecting a client. Proposed; see §5.8. | | `[K-ID-14]` a client must not follow a position on an unrecognised `progress_match` | Client-side. The server reports the type and acts on it in no way, so nothing over the wire distinguishes a client that honours this from one that ignores it. Checkable only by inspecting a client. Proposed; see §5.8. | | `[K-ID-13]` aliases are removed with the account | Same reason as `[K-DEL-1]`: probing `DELETE /users/me` would delete the account under test. Proposed; see §5.8. | | `[K-DOC-1]` … `[K-DOC-10]` document identity | Not a server behaviour: `document` is opaque to a server, and four of the eight surveyed servers never compute it (§11.1). Checked by `vectors/check.mjs` against the golden vectors of §8.6, not over HTTP. | @@ -2361,7 +2388,7 @@ questioning the code's existence (`syncs_controller.lua:17`): | Is `202` reachable on any server? | Not among those surveyed. | | Is error code `100` reachable? | **No**, in practice — §3.3. Every HTTP client sends a default `Accept`, so the reference server answers `101`. | | Will `identifiers` (§5.8) be merged, and in this shape? | **Open.** `koreader/koreader-sync-server#55` is unmerged as of 2026-09-22. §5.8 is pinned to a branch commit for that reason. | -| What identifier **types** should a client send? | **Undecided, and deliberately not specified.** The server treats `type` as an opaque label and never interprets it (`identifiers.lua:1-3`), so the set can grow without a server change. `content`, `structure` and `metadata` are the branch's own examples; none is computed by any released KOReader build. | +| What identifier **types** should a client send? | `content`, `structure` and `filename`, with the recipes in the §5.8 registry. The server still treats `type` as an opaque label and never interprets it (`identifiers.lua:1-3`), so the set can grow without a server change — but `progress_match` makes the label a trust signal between clients, so a label without a recipe is not interoperable (`[K-ID-14]`). None of the three is computed by any released KOReader build. | | How should a client pick the `document` when it holds several identifiers? | **Open.** §5.8 requires only that the first entry equal `document`. Which identifier that ought to be is a client decision no implementation has yet made. | | Does an alias ever expire? | **No, in the proposal.** Aliases are removed only with the account (`[K-ID-13]`). A long-lived account accumulates one alias per distinct identifier it has ever offered, and nothing in the branch bounds that. | | Does the official deployment carry the `[K-GET-1a]` router defect? | **[unverified]** — the published image does (measured). Whether `sync.koreader.rocks` runs that image is unknown. | From fb8aa6c3fea224712b55bb53824a9306ba914b72 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 10:29:43 -0500 Subject: [PATCH 08/16] Correct the entity example's arithmetic, and pin four more edge cases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The href example counted seventeen characters as eighteen and thirteen as sixteen. A reimplementer checking their work against those numbers chases a phantom. Also: an empty named identifier falls back, trimming is XML whitespace only, a manifest item with no href contributes no line, and the rootfile is the first declared package document rather than the first of any type — which keeps the digest agreeing with whichever OPF a reader already opens. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/SPEC.md b/SPEC.md index 830bbc3..acb9f74 100644 --- a/SPEC.md +++ b/SPEC.md @@ -879,13 +879,17 @@ carries no guarantee about the file the position was written against, so it is identically or the label means nothing. The recipe, exactly: 1. Take the `` element's `unique-identifier` attribute and find the - `` whose `id` matches it. Failing that, take the first - ``. Trim it. If the result is non-empty, it is the first line. + `` whose `id` matches it — the first, if several do. Trim it. + If the result is empty, or no element matches, fall back to the first + `` whose trimmed text is non-empty. That value is the first + line; if there is none, there is no first line. Trimming removes the XML + whitespace characters (space, tab, CR, LF) and nothing else. 2. Walk `` in document order. For each ``, resolve its `idref` against `` and take that item's `href` **exactly as written** in the attribute: not percent-decoded, not resolved against the OPF directory, not reduced to a basename, and **not XML-entity-decoded** — an `href` written - `a&b/ch2.xhtml` contributes those eighteen characters, not sixteen. Strip + `a&b/ch2.xhtml` contributes those seventeen characters, not the thirteen + it decodes to. Strip a `#fragment` if one is present. Each is a line, in spine order. 3. Join the lines with `\n`, with no trailing newline, and take the md5 of the UTF-8 bytes. @@ -900,14 +904,18 @@ defensible choice and disagree: `` are the same element and `` under a default Dublin Core namespace counts as ``. Attribute values are not namespaced. - **An `` whose `idref` resolves to no manifest item contributes no - line**, and does not invalidate the digest. + line**, and does not invalidate the digest. So does one that resolves to an + `` carrying no `href`: no href, no line, rather than an empty one. - **`linear="no"` items are included.** They are spine entries, and the renderer's `DocFragment` numbering counts them. - `` is looked up within ``, `` within ``, and `` within ``. -- **The OPF is the first ``** in `META-INF/container.xml`, whatever - its `media-type`. Its `full-path` is used as an archive member name as - written; if no member matches, there is no digest rather than a guessed one. +- **The OPF is the first `` whose `media-type` is + `application/oebps-package+xml`**, or the first `` of any type when + none declares it. Its `full-path` is used as an archive member name as written; + if no member matches, there is no digest rather than a guessed one. Preferring + the declared package document keeps this from disagreeing with whatever a + reader already opens for a multi-rendition container. `structure` deliberately covers **no file contents.** The tools that motivate this section rewrite them: CrossPoint's EPUB optimizer re-encodes images to From 05c1bec6d2b4b5762c869db174f5cee0ef2ef406 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 10:31:22 -0500 Subject: [PATCH 09/16] Take every value as the XML parser yields it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The href rule said raw source bytes. Complying with that took expat's XML_DefaultCurrent and a hand-rolled attribute scan in one implementation, while the identifier and full-path arrived decoded through the same parser — so the recipe asked for the harder thing in one place and the natural thing in two, for no reason. Entity expansion is now uniform. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/SPEC.md b/SPEC.md index acb9f74..e428a7f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -885,12 +885,9 @@ identically or the label means nothing. The recipe, exactly: line; if there is none, there is no first line. Trimming removes the XML whitespace characters (space, tab, CR, LF) and nothing else. 2. Walk `` in document order. For each ``, resolve its `idref` - against `` and take that item's `href` **exactly as written** in - the attribute: not percent-decoded, not resolved against the OPF directory, - not reduced to a basename, and **not XML-entity-decoded** — an `href` written - `a&b/ch2.xhtml` contributes those seventeen characters, not the thirteen - it decodes to. Strip - a `#fragment` if one is present. Each is a line, in spine order. + against `` and take that item's `href`: **not** percent-decoded, + **not** resolved against the OPF directory, **not** reduced to a basename. + Strip a `#fragment` if one is present. Each is a line, in spine order. 3. Join the lines with `\n`, with no trailing newline, and take the md5 of the UTF-8 bytes. @@ -900,6 +897,12 @@ A container with no spine, or one that is not an OPF-bearing archive, has no The rest of the recipe, because two implementations will otherwise each make a defensible choice and disagree: +- **Every value here is the one an XML parser yields, with entity references + expanded** — the identifier, the hrefs and `full-path` alike. An `href` written + `a&b/ch2.xhtml` contributes the thirteen characters `a&b/ch2.xhtml`. Raw + source bytes are used nowhere. Expanding is what a parser does without being + asked; keeping a value raw means going out of its way, and a rule that is + harder to obey is one two implementations will obey differently. - **Element names are matched on the local name**, so `` and `` are the same element and `` under a default Dublin Core namespace counts as ``. Attribute values are not namespaced. @@ -912,8 +915,9 @@ defensible choice and disagree: `` within ``. - **The OPF is the first `` whose `media-type` is `application/oebps-package+xml`**, or the first `` of any type when - none declares it. Its `full-path` is used as an archive member name as written; - if no member matches, there is no digest rather than a guessed one. Preferring + none declares it. Its `full-path`, as the parser yields it, is the archive + member name; if no member matches, there is no digest rather than a guessed + one. Preferring the declared package document keeps this from disagreeing with whatever a reader already opens for a multi-rendition container. From 1551b89811223bbc13585f58ca1da9d8283276e6 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 10:41:54 -0500 Subject: [PATCH 10/16] Pin the last four places two parsers could still disagree Attribute names are literal and unprefixed, because an unprefixed attribute is in no namespace and opf:href would be a different attribute, not the same one. An element's value is all of its character data, since a parser may split a run at an entity reference. Only the XML-defined entities are expanded, so a lenient parser that knows HTML names has no licence to use them. Items outside the manifest yield no digest. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index e428a7f..da1ead9 100644 --- a/SPEC.md +++ b/SPEC.md @@ -905,7 +905,22 @@ defensible choice and disagree: harder to obey is one two implementations will obey differently. - **Element names are matched on the local name**, so `` and `` are the same element and `` under a default Dublin Core - namespace counts as ``. Attribute values are not namespaced. + namespace counts as ``. **Attribute names are matched + literally and unprefixed** — `href`, `id`, `idref`, `full-path`, `media-type`, + `unique-identifier`. An unprefixed attribute is in no namespace, so `opf:href` + is a different attribute rather than the same one, and the package format + writes these unprefixed. +- **An element's value is its complete character data, concatenated**, not the + first text node. A parser may split a run at an entity reference or a buffer + boundary, and two that split differently must still agree. +- **Only the five predefined XML entities and numeric character references are + expanded**: `& < > " '`, `&#nnn;` and `&#xHH;`. Any other + undeclared reference makes the document not well-formed, and a digest taken + from one is undefined rather than wrong — a conforming parser rejects the + document where a lenient one may expand an HTML name it happens to know. +- A package that puts `` outside `` is malformed, and resolves + no spine entry, so it has **no** `structure` digest rather than one computed + from whatever was found elsewhere. - **An `` whose `idref` resolves to no manifest item contributes no line**, and does not invalidate the digest. So does one that resolves to an `` carrying no `href`: no href, no line, rather than an empty one. From 8655d7d8a0cddd406e3cc9e81caecd6c9b2cd5a4 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 12:31:28 -0500 Subject: [PATCH 11/16] Say what a client does with a type it cannot compute, and widen K-ID-14 A fourth implementation cannot derive a filename at all: Readest stores books by hash and the only name it can reconstruct is its OPF title, so a digest labelled filename would be the metadata type this section removed, wearing a different label. K-ID-15 says a client omits what it cannot compute honestly rather than approximating it, and sends no list when only the document is left. K-ID-14 gated following a position but not resolving one. A client that resolves an untrusted position to compare it with the local one gets a number that looks authoritative and can suppress its own conflict prompt, after which the position is overwritten with nobody asked. Also: one spine line is required, duplicate manifest ids take the first, and an href is not trimmed. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/SPEC.md b/SPEC.md index da1ead9..085dcd4 100644 --- a/SPEC.md +++ b/SPEC.md @@ -870,11 +870,29 @@ without a recipe is not interoperable. | `structure` | md5 over the spine, as defined below | the edition, or the chapter list or its order, changes | | `filename` | md5 of the file name, as §8.5 derives it | the file is renamed | +**[K-ID-15]** A client **offers only the types it can compute honestly.** The +registry is not a set a client has to fill: a type whose recipe it cannot follow +for the book in hand is omitted, not approximated. A client that can derive no +identifier but `document` itself has nothing to say and sends no list at all, +which is the request in §5.4 exactly as it was. Substituting a value that merely +resembles a registered type is worse than omitting it — a digest over a title +dressed up as `filename` is the `metadata` type this section removed, and +`[K-ID-12]` will make one bad match permanent. + **[K-ID-14]** A client **MUST NOT** treat a `progress_match` type it does not recognise as sufficient to follow a `progress` string. An unrecognised label carries no guarantee about the file the position was written against, so it is `percentage` that applies, not the stored position. +The same applies to anything **derived** from that string, not only to following +it. A client that resolves the stored position against the local book to compare +it with the local one — for conflict detection, for a preview, for deciding +whether to prompt — is resolving a position from a file it may not share, and the +number it gets back looks authoritative. Gating only the jump leaves that number +free to suppress a conflict prompt, at which point the position is overwritten +with no one asked. Where the match is not one the client may follow, `percentage` +is the only comparable quantity. + `structure` is computed from the OPF, and two clients have to compute it identically or the label means nothing. The recipe, exactly: @@ -921,6 +939,13 @@ defensible choice and disagree: - A package that puts `` outside `` is malformed, and resolves no spine entry, so it has **no** `structure` digest rather than one computed from whatever was found elsewhere. +- **At least one spine line is required.** A digest of the identifier alone is + not a `structure` digest, so a package whose ``s all dangle has none, + the same as one with no `` at all. +- Two `` elements carrying the same `id`: the **first**, as for + ``. +- **An `href` is not trimmed.** Step 1 trims the identifier and nothing else + does; leading or trailing space inside the attribute is part of the value. - **An `` whose `idref` resolves to no manifest item contributes no line**, and does not invalidate the digest. So does one that resolves to an `` carrying no `href`: no href, no line, rather than an empty one. @@ -2222,6 +2247,7 @@ reach `MUST` and be charged to every server. | `[K-SYNC-1]` … `[K-SYNC-6]` conflict resolution | Entirely client-side; the server never compares anything. The verifier asserts the server-side preconditions instead: `timestamp` present (`[K-FLD-13]`), in seconds (`[K-FLD-14]`), server-generated (`[K-PUT-4]`), and last-write-wins (`[K-PUT-5]`). | | `[K-ID-12]` an alias is created, never repointed | Not observable over HTTP: the alias table is private, and every wire consequence of it is already asserted — resolution is stable (`[K-ID-4]`), a document that exists in its own right is never shadowed (`[K-ID-7]`), and an identifier above the match is never registered (`[K-ID-12b]`). Proposed; see §5.8. | | `[K-ID-5b]` identifiers are ordered strongest first | Client-side, and not inferable from the wire: the server has no idea what any type means, so a weakest-first list is indistinguishable from a strongest-first one whose strongest identifier happens to be the one that matched. Checkable only by inspecting a client. Proposed; see §5.8. | +| `[K-ID-15]` a client offers only the types it can compute honestly | Client-side, and invisible from the server: a list of two is indistinguishable from a client that could only manage two, and a value that merely resembles a registered type is a correct-looking string the server has no way to judge. Checkable only by inspecting a client. Proposed; see §5.8. | | `[K-ID-14]` a client must not follow a position on an unrecognised `progress_match` | Client-side. The server reports the type and acts on it in no way, so nothing over the wire distinguishes a client that honours this from one that ignores it. Checkable only by inspecting a client. Proposed; see §5.8. | | `[K-ID-13]` aliases are removed with the account | Same reason as `[K-DEL-1]`: probing `DELETE /users/me` would delete the account under test. Proposed; see §5.8. | | `[K-DOC-1]` … `[K-DOC-10]` document identity | Not a server behaviour: `document` is opaque to a server, and four of the eight surveyed servers never compute it (§11.1). Checked by `vectors/check.mjs` against the golden vectors of §8.6, not over HTTP. | From b69d304b473a9f7ddae196e6055f000b4de98126 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 14:51:02 -0500 Subject: [PATCH 12/16] Restore metadata as a weak identifier, and fix the drift detector metadata is the only identifier that survives a spine change, so it is the only one that reaches a different edition or a re-chunked conversion. It is also the only one that can match two different works. K-ID-16 lets a PUT entry carry weak: true, and K-ID-17 says a write resolving only through a weak entry does not adopt that record. A weak identifier seeds a reader; it does not claim a record. progress_match told a reader not to trust a weak match while nothing told a writer not to clobber on one. The flag is body-only, so the ids grammar and anything parsing it are untouched. coverage.mjs treated any id with a matching prefix as a refinement, so K-ID-16 counted as covered by an assertion for K-ID-1. A refinement suffix is never a digit. Fixing it exposed two pre-existing gaps: K-PUT-202 was asserted and never defined, and K-FLD-1 had neither an assertion nor a reason. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 44 +++++++++++++++++++++++++++++++++++++------- coverage.mjs | 9 +++++++-- verify.mjs | 49 ++++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 92 insertions(+), 10 deletions(-) diff --git a/SPEC.md b/SPEC.md index 085dcd4..59a7aa6 100644 --- a/SPEC.md +++ b/SPEC.md @@ -540,6 +540,10 @@ code **2004**, message `Field 'document' not provided.` {"document": "", "timestamp": 1790005119} ``` +**[K-PUT-7]** The server **MUST NOT** answer **202**. `api.json` declares it as +an accepted status, and no client treats it as success, so a 202 is a push the +client queues for retry while the server believes it stored the position. + **[K-PUT-4]** `timestamp` is **generated by the server** as Unix epoch **seconds** (`syncs_controller.lua:246`: `local timestamp = os.time()`). Any client-supplied `timestamp` in the body is ignored — the reference client never @@ -869,6 +873,32 @@ without a recipe is not interoperable. | `content` | the document's own digest over the file bytes, as §8 derives it | any byte of the file changes | | `structure` | md5 over the spine, as defined below | the edition, or the chapter list or its order, changes | | `filename` | md5 of the file name, as §8.5 derives it | the file is renamed | +| `metadata` | md5 of `"title:" + title + "\n" + "authors:" + authors`, the title lowercased with whitespace runs collapsed and trimmed, the authors normalised the same way, sorted, and joined with `;`. A title with no author yields nothing. **Always weak** (`[K-ID-17]`) | the work's title or authors change | + +#### Weak identifiers + +**[K-ID-16]** An entry in the `PUT` body **MAY** carry `"weak": true`. It means +the client does not consider that identifier sufficient to claim a record that +already exists. The flag appears **only in the request body**: adoption is a +property of a write, so the `ids` parameter of a read has no equivalent and its +grammar is unchanged. + +**[K-ID-17]** A `PUT` whose walk resolves **only** through a weak entry **does +not adopt that record.** It writes under `document`, creating it if absent, +exactly as though nothing had matched, and answers with that digest. A record +found through a strong entry is adopted as before. Reads are unaffected: a weak +identifier resolves on a `GET` like any other, and `progress_match` says what the +position is worth. + +This is what separates seeding from binding. A copy that shares only a weak +identifier is shown the other copy's position and can seek to its percentage, +and it still keeps a record of its own; it cannot overwrite the position of the +record it read from, and it cannot merge two works that happen to be tagged +alike. Without it, `progress_match` tells a reader not to trust a weak match +while nothing tells a writer not to clobber on one. + +A client **MUST** mark as weak any identifier that can match a different work. +`metadata` is always weak. `content` and `structure` never are. **[K-ID-15]** A client **offers only the types it can compute honestly.** The registry is not a set a client has to fill: a type whose recipe it cannot follow @@ -971,12 +1001,11 @@ what an xpointer counts — `/body/DocFragment[N]` is the Nth spine entry — so `structure` match says precisely that the position's chapter index means the same thing here. -**There is no `metadata` type.** A digest over title and author is the only -identifier that can match two genuinely different files, and `[K-ID-12]` never -repoints an alias. Two books a library has tagged alike merge on it, and until -`[K-ID-12b]` the merge outlived the tagging being corrected. The recompression -case is covered by `structure`, so the type bought cross-edition matching at the -cost of the only unrecoverable failure in the section. +`metadata` is the only identifier that survives a **spine change**, so it is the +only one that reaches a different edition or a re-chunked conversion. It is also +the only one that can match two genuinely different works, which is why it is +never a binding identifier: it seeds a reader, it does not claim a record. See +`[K-ID-16]` and `[K-ID-17]`. | Rule | Value | Citation | |---|---|---| @@ -2191,7 +2220,7 @@ Work down this list; each row names the requirement it satisfies. - [ ] `GET` always includes `timestamp`. `[K-FLD-13]` - [ ] Accept `metadata` without failing; do not clear it when omitted. `[K-FLD-10]`, `[K-FLD-11]` - [ ] Answer within 5 seconds. `[K-SYNC-7]` -- [ ] Never emit `202`. §5.4 +- [ ] Never emit `202`. `[K-PUT-7]` **Errors** @@ -2232,6 +2261,7 @@ reach `MUST` and be charged to every server. | `[K-URL-1]` unversioned paths | True by construction: every other assertion is issued against exactly these paths, so the suite could not pass if they were versioned. | | `[K-AUTH-1]` credentials in `x-auth-user`/`x-auth-key` | Same: every authenticated assertion sends only these two headers. A server reading credentials elsewhere fails all of them. | | `[K-AUTH-2]` the key is `md5(password)` | Client-side. Checkable only by inspecting a client. The verifier computes the MD5 itself and so exercises the server's half. | +| `[K-FLD-1]` `document` is an opaque non-empty string without a colon | Asserted in consequence rather than directly: every authenticated assertion addresses a document, so an empty one fails them all, and the character set is probed as `[K-DOC-ID-1]` and `[K-DOC-ID-2]`. The colon is refused by the reference's own read route (§14.3) before the field rule can be observed. | | `[K-CT-1]` clients send valid JSON | Client-side. | | `[K-CT-4]` clients must not dispatch on response `Content-Type` | Client-side. The server half is `[K-CT-3]`, which is asserted. | | `[K-REG-1]` `password` is pre-hashed | Not directly observable: a correct server and a double-hashing server both accept the registration. The verifier asserts the consequence instead — register with key `K`, then authenticate with key `K` — which a double-hashing server fails. | diff --git a/coverage.mjs b/coverage.mjs index 96dc1bf..91a62b9 100644 --- a/coverage.mjs +++ b/coverage.mjs @@ -83,7 +83,12 @@ for (const line of table.split("\n")) { // An assertion may refine a requirement with a suffix (K-CT-3 -> K-CT-3b), and // §12.4 may cover a family with a trailing dash (K-SYNC- covers K-SYNC-4). -const assertedFor = (id) => asserted.has(id) || [...asserted].some((a) => a.startsWith(id)); +// +// A refinement suffix is never a digit: without that, K-ID-16 counts as covered +// by an assertion for K-ID-1, and every requirement numbered past 9 is checked +// against the wrong one. +const refines = (a, id) => a.startsWith(id) && !/^\d/.test(a.slice(id.length)); +const assertedFor = (id) => asserted.has(id) || [...asserted].some((a) => refines(a, id)); const untestableFor = (id) => { if (untestable.has(id)) return untestable.get(id); for (const [key, reason] of untestable) { @@ -103,7 +108,7 @@ for (const id of [...defined].sort()) { } } -const undefinedIds = [...asserted].filter((a) => !defined.has(a) && ![...defined].some((d) => a.startsWith(d))).sort(); +const undefinedIds = [...asserted].filter((a) => !defined.has(a) && ![...defined].some((d) => refines(a, d))).sort(); if (list) { const w = Math.max(...rows.map((r) => r[0].length)); diff --git a/verify.mjs b/verify.mjs index d1119b6..4f9a948 100644 --- a/verify.mjs +++ b/verify.mjs @@ -653,7 +653,7 @@ async function main() { note: "KOReader sets PROGRESS_TIMEOUTS = {2, 5}; a slower answer is a failed push that gets queued for retry", }); assert({ - id: "K-PUT-202", section: "§5.4", level: "MUST", + id: "K-PUT-7", section: "§5.4", level: "MUST", title: "the server does not emit 202", ok: r.status !== 202, expected: "not 202 — api.json declares it but no client treats it as success", @@ -1063,6 +1063,8 @@ const ID_ASSERTIONS = [ ["K-ID-6c", "a progress string written without identifiers is attributed to its own digest"], ["K-ID-7", "an alias never shadows a document that exists in its own right"], ["K-ID-12b", "a weak match does not register the identifiers ranked above it"], + ["K-ID-16", "an entry marked weak is accepted, and a strong match still adopts"], + ["K-ID-17", "a push resolving only through a weak entry does not adopt that record"], ["K-ID-8", "an identifier list that does not name the document is rejected"], ["K-ID-8b", "the document need not be first, and the record is created under it"], ["K-ID-9", "more than 8 identifiers is rejected"], @@ -1311,6 +1313,51 @@ async function identifierSuite() { }); } + { + // A weak entry may resolve a read, but it must not let a write claim a + // record that already exists. + const k1 = dg("wk1"), k1s = dg("wk1s"), shared = dg("wkm"); + const k2 = dg("wk2"), k2s = dg("wk2s"); + const weak = (v) => ({ type: "metadata", value: v, weak: true }); + const listFor = (c, st) => [["content", c], ["structure", st]]; + + const mk = (document, c, st, progress, pct) => + call("PUT", "/syncs/progress", { + body: { + document, + identifiers: [...listFor(c, st).map(([type, value]) => ({ type, value })), weak(shared)], + progress, percentage: pct, device: "kosync-conformance", device_id: "conformance-device-1", + }, + }); + + const first = await mk(k1, k1, k1s, XP, 0.8); + assert({ + id: "K-ID-16", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-16"], + ok: first.status === 200 && first.json?.document === k1 && first.json?.match === "content", + expected: `200 creating ${k1}, match "content"`, + actual: first.error ?? `${first.status} ${fmt(first.json ?? first.text)}`, + note: rejected(first) + ? "a `weak` flag on an identifier was rejected; it is an optional field on the entry, not a new type" + : undefined, + }); + + const second = await mk(k2, k2, k2s, "/body/p[1]", 0.01); + const original = await get(k1, listFor(k1, k1s)); + assert({ + id: "K-ID-17", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-17"], + ok: second.status === 200 && second.json?.document === k2 + && original.json?.progress === XP && original.json?.percentage === 0.8, + expected: `${k2} written under its own digest, and ${k1} left at ${XP}`, + actual: second.error ?? original.error + ?? `push answered ${fmt(second.json?.document)}; the first record now reads ${fmt(original.json?.progress)} at ${fmt(original.json?.percentage)}`, + note: second.json?.document === k1 + ? "the second work adopted the first's record through an identifier the caller marked weak, and overwrote the position stored there. A weak identifier seeds a reader; it does not claim a record." + : undefined, + }); + } + // --- validation ----------------------------------------------------------- { const g1 = dg("g1"), gs = dg("gs"); From 535474d659153d9f5e4d312417b32fdb128530ab Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 14:59:54 -0500 Subject: [PATCH 13/16] Pin the five things the weak rule left to each implementation Written against the first implementation of K-ID-16 and K-ID-17, which found all five. Two more servers are being written against the same text. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/SPEC.md b/SPEC.md index 59a7aa6..6197fb8 100644 --- a/SPEC.md +++ b/SPEC.md @@ -900,6 +900,26 @@ while nothing tells a writer not to clobber on one. A client **MUST** mark as weak any identifier that can match a different work. `metadata` is always weak. `content` and `structure` never are. +Five things the rule would otherwise leave to each implementation: + +- **"Only through a weak entry" means the entry the walk stopped on.** The walk + stops at the first hit (`[K-ID-5]`) and does not continue past a weak one + looking for a strong hit further down. Under `[K-ID-5b]` the two readings + coincide, since weak entries sort last; they differ only for a client that + breaks the ordering rule. +- **`weak` is a boolean.** Anything else is rejected with **403** code **2003**, + like any other malformed entry. Reading a non-boolean as strong is exactly the + clobber the flag exists to prevent. +- **`match` on a non-adopting write is the create rule** (`[K-ID-8b]`): the type + of the entry whose value equals `document`, not the weak type that resolved + the walk. +- **Weakness governs adoption, not registration.** A weak entry at or below the + match is still registered as an alias, whether or not the write adopted. That + is what lets a later read be seeded at all. +- **The stored writer list does not record the flag.** `progress_match` reports + the shared type and leaves the judgement to the reader, so weakness is not + recoverable after the write. + **[K-ID-15]** A client **offers only the types it can compute honestly.** The registry is not a set a client has to fill: a type whose recipe it cannot follow for the book in hand is omitted, not approximated. A client that can derive no From 828bd2df27eee000d9b99078d29d72c23c2d25d4 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 15:08:16 -0500 Subject: [PATCH 14/16] Assert the weak value is registered, not only that it did not adopt K-ID-17 could not tell 'did not adopt' from 'never registered the weak value'. Its second push finds the shared value only because the first push registered it, so a server that skipped weak aliases would create the second record, leave the first alone, and pass vacuously. K-ID-17b reads a third copy that shares only the weak value and requires it to be seeded from the first record. Found by the second and third implementations, which both register weak aliases and both carry their own test for it. Co-Authored-By: Claude Opus 5 (1M context) --- verify.mjs | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/verify.mjs b/verify.mjs index 4f9a948..2d7e625 100644 --- a/verify.mjs +++ b/verify.mjs @@ -1065,6 +1065,7 @@ const ID_ASSERTIONS = [ ["K-ID-12b", "a weak match does not register the identifiers ranked above it"], ["K-ID-16", "an entry marked weak is accepted, and a strong match still adopts"], ["K-ID-17", "a push resolving only through a weak entry does not adopt that record"], + ["K-ID-17b", "the weak entry is still registered, so a later read is seeded through it"], ["K-ID-8", "an identifier list that does not name the document is rejected"], ["K-ID-8b", "the document need not be first, and the record is created under it"], ["K-ID-9", "more than 8 identifiers is rejected"], @@ -1356,6 +1357,25 @@ async function identifierSuite() { ? "the second work adopted the first's record through an identifier the caller marked weak, and overwrote the position stored there. A weak identifier seeds a reader; it does not claim a record." : undefined, }); + + // K-ID-17 on its own cannot tell "did not adopt" from "never registered the + // weak value at all": a server that skipped weak aliases would create k2, + // leave k1 alone, and pass it vacuously. Weakness governs adoption, not + // registration, so a third copy sharing only the weak value must still be + // seeded from the first record. + const k3 = dg("wk3"); + const seeded = await get(k3, [["content", k3], ["metadata", shared]]); + assert({ + id: "K-ID-17b", section: SEC, level: "MAY", feature: "identifiers", + title: ID_TITLE["K-ID-17b"], + ok: seeded.status === 200 && seeded.json?.document === k1 + && seeded.json?.progress === XP && seeded.json?.progress_match === "metadata", + expected: `200 resolving to ${k1} through the weak value, progress_match "metadata"`, + actual: seeded.error ?? `${seeded.status} ${fmt(seeded.json ?? seeded.text)}`, + note: isObject(seeded.json) && seeded.json.percentage === undefined + ? "the weak identifier resolved nothing, so it was never registered as an alias. Weakness governs whether a write claims a record, not whether the value is indexed — without the alias there is nothing to seed a later copy from, which is the whole reason the type exists." + : undefined, + }); } // --- validation ----------------------------------------------------------- From cbcfff24ea79a1d90b6d971d83d3873f623c4ac8 Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Wed, 23 Sep 2026 15:22:43 -0500 Subject: [PATCH 15/16] Pin the metadata normalisation and the weak-document case Three clients implemented the recipe and folded case three ways: full Unicode twice, ASCII once, because a microcontroller has no case table. An identifier two clients fold differently is worse than one that folds less, so the fold is ASCII only. Where the authors come from was also unstated: dc:creator element text, never opf:file-as, contributors excluded. A weak entry carrying the document value adopts, because writing to the record you are addressed by is not claiming another's. Co-Authored-By: Claude Opus 5 (1M context) --- SPEC.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 6197fb8..83ebe16 100644 --- a/SPEC.md +++ b/SPEC.md @@ -873,7 +873,7 @@ without a recipe is not interoperable. | `content` | the document's own digest over the file bytes, as §8 derives it | any byte of the file changes | | `structure` | md5 over the spine, as defined below | the edition, or the chapter list or its order, changes | | `filename` | md5 of the file name, as §8.5 derives it | the file is renamed | -| `metadata` | md5 of `"title:" + title + "\n" + "authors:" + authors`, the title lowercased with whitespace runs collapsed and trimmed, the authors normalised the same way, sorted, and joined with `;`. A title with no author yields nothing. **Always weak** (`[K-ID-17]`) | the work's title or authors change | +| `metadata` | md5 of `"title:" + title + "\n" + "authors:" + authors`, normalised as below. A title with no author yields nothing. **Always weak** (`[K-ID-17]`) | the work's title or authors change | #### Weak identifiers @@ -919,6 +919,12 @@ Five things the rule would otherwise leave to each implementation: - **The stored writer list does not record the flag.** `progress_match` reports the shared type and leaves the judgement to the reader, so weakness is not recoverable after the write. +- **A weak entry that is itself the `document` still adopts.** `[K-ID-8]` + requires the list to contain an entry equal to `document`, and nothing stops + that entry being the weak one — a client whose configured document digest is + its weakest identifier is in exactly that position. Writing to the record you + are addressed by is not claiming another's, so `[K-ID-17]` does not fire when + the entry that resolved the walk carries the `document` value. **[K-ID-15]** A client **offers only the types it can compute honestly.** The registry is not a set a client has to fill: a type whose recipe it cannot follow @@ -1021,6 +1027,24 @@ what an xpointer counts — `/body/DocFragment[N]` is the Nth spine entry — so `structure` match says precisely that the position's chapter index means the same thing here. +The `metadata` normalisation, because three implementations disagreed on it: + +- **The authors are the `` elements of the OPF**, in the order the + document gives them, taken from each element's **text** and never from an + `opf:file-as` attribute. `` is not an author. Each is + normalised on its own, then they are **sorted** and joined with `;`. +- **Case folding is ASCII only.** `A`–`Z` fold to `a`–`z` and nothing else does. + A full Unicode fold is not available on every client — a microcontroller has no + case table and is not going to carry one — and an identifier two clients fold + differently is worse than one that folds less. The cost is a miss on a title + whose case varies outside ASCII, which for a weak identifier is the safe + direction. +- **Whitespace is the XML set**, space, tab, CR and LF, as in the `structure` + recipe. Runs collapse to one space and the ends are trimmed. No other + character is whitespace. +- A value that normalises to nothing is absent, and an absent title or an empty + author list yields no `metadata` digest at all. + `metadata` is the only identifier that survives a **spine change**, so it is the only one that reaches a different edition or a re-chunked conversion. It is also the only one that can match two genuinely different works, which is why it is From 84fa8cda6de29eb75277a7c349d6b511c60402ae Mon Sep 17 00:00:00 2001 From: Jonathan Roemer Date: Sun, 27 Sep 2026 13:41:54 -0500 Subject: [PATCH 16/16] Answer the identifier-type and document questions from the registry and K-ID-8b Co-Authored-By: Claude Opus 5.5 --- SPEC.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SPEC.md b/SPEC.md index 83ebe16..40d6207 100644 --- a/SPEC.md +++ b/SPEC.md @@ -2514,9 +2514,9 @@ questioning the code's existence (`syncs_controller.lua:17`): | Is there any client that sends a `timestamp` on PUT? | Not among those surveyed. The reference server would ignore it. | | Is `202` reachable on any server? | Not among those surveyed. | | Is error code `100` reachable? | **No**, in practice — §3.3. Every HTTP client sends a default `Accept`, so the reference server answers `101`. | -| Will `identifiers` (§5.8) be merged, and in this shape? | **Open.** `koreader/koreader-sync-server#55` is unmerged as of 2026-09-22. §5.8 is pinned to a branch commit for that reason. | -| What identifier **types** should a client send? | `content`, `structure` and `filename`, with the recipes in the §5.8 registry. The server still treats `type` as an opaque label and never interprets it (`identifiers.lua:1-3`), so the set can grow without a server change — but `progress_match` makes the label a trust signal between clients, so a label without a recipe is not interoperable (`[K-ID-14]`). None of the three is computed by any released KOReader build. | -| How should a client pick the `document` when it holds several identifiers? | **Open.** §5.8 requires only that the first entry equal `document`. Which identifier that ought to be is a client decision no implementation has yet made. | +| Will `identifiers` (§5.8) be merged, and in this shape? | **Open.** `koreader/koreader-sync-server#55` is unmerged as of 2026-09-27. §5.8 is pinned to a branch commit for that reason. | +| What identifier **types** should a client send? | `content`, `structure`, `metadata` and `filename`, strongest first, with the recipes in the §5.8 registry. `metadata` is always weak (`[K-ID-17]`). The server treats `type` as an opaque label and never interprets it (`identifiers.lua:1-3`), so the set can grow without a server change — but `progress_match` makes the label a trust signal between clients, so a label without a recipe is not interoperable (`[K-ID-14]`). None of the four is computed by any released KOReader build. | +| How should a client pick the `document` when it holds several identifiers? | The one it would send if the server took only one. §5.8 requires one entry to equal `document` and not that it be first (`[K-ID-8b]`), so a client addressed by its filename digest ranks that entry last. | | Does an alias ever expire? | **No, in the proposal.** Aliases are removed only with the account (`[K-ID-13]`). A long-lived account accumulates one alias per distinct identifier it has ever offered, and nothing in the branch bounds that. | | Does the official deployment carry the `[K-GET-1a]` router defect? | **[unverified]** — the published image does (measured). Whether `sync.koreader.rocks` runs that image is unknown. |