Resolve #221 against main's validated-statistics fixes

.github/workflows/ci.yml

@@ -400,11 +400,26 @@ jobs:
         shell: bash
         run: |
           set +e
-          julia --project=. -t2 --compiled-modules=no test/runtests.jl validated_statistics numeric_policy exact_summaries high_precision derived_analyses 2>&1 | tee /tmp/julia-validated-tests.log
+          julia --project=. -t2 --compiled-modules=no test/runtests.jl numeric_policy exact_summaries high_precision derived_analyses validated_statistics 2>&1 | tee /tmp/validated-statistics-tests.log
           status="${PIPESTATUS[0]}"
           if [ "$status" -ne 0 ]; then
-            enc() { head -c 3500 | sed -e 's/%/%25/g' -e 's/\r/%0D/g' | awk '{printf "%s%%0A", $0}'; }
-            echo "::error title=Validated statistics failed (tail)::$(tail -c 3500 /tmp/julia-validated-tests.log | enc)"
+            enc() { head -c 3000 | sed -e 's/%/%25/g' -e 's/\r/%0D/g' | awk '{printf "%s%%0A", $0}'; }
+            # One annotation cannot hold a failing run, and a tail hides every
+            # failure but the last one. Publish a digest of all failing
+            # assertions first - the testset, the expression and the evaluated
+            # values are what identify the bug - then the end of the log.
+            grep -E "Test Failed at |Error During Test at |^  Expression: |^   Evaluated: |^  Test threw exception" \
+              /tmp/validated-statistics-tests.log > /tmp/validated-statistics-failures.log || true
+            rm -f /tmp/validated-statistics-digest-*
+            split -b 2500 /tmp/validated-statistics-failures.log /tmp/validated-statistics-digest- || true
+            index=0
+            for chunk in /tmp/validated-statistics-digest-*; do
+              [ -e "$chunk" ] || continue
+              index=$((index + 1))
+              [ "$index" -le 8 ] || break
+              echo "::error title=Validated-statistics failure digest $index::$(enc < "$chunk")"
+            done
+            echo "::error title=Validated-statistics log tail::$(tail -c 3000 /tmp/validated-statistics-tests.log | enc)"
           fi
           exit "$status"
 

README.md

@@ -84,7 +84,7 @@ Once a run completes, analysis is performed on request through the web UI, both
 
 Counts may be normalised before analysis (none, rarefaction to a fixed or auto-resolved depth, or scaling with ranked subsampling (SRS), which reaches the depth by scaling counts and keeps more of the community structure), and contamination-flagged taxa may be included or excluded. Ordinations can also apply a Hellinger transform before the Bray-Curtis dissimilarity. Composition charts can be drawn as a grid faceted on any two of run, group and sub-group. All analysis charts are returned as Plotly JSON and rendered interactively in the browser.
 
-Opt-in exact-count and high-precision descriptive summaries are available through a separate validated API without changing these ordinary workflows. Their numeric, zero-depth, DuckDB, persistence, and reproducibility contracts are documented in [Validated descriptive statistics](VALIDATED_STATISTICS.md).
+Opt-in exact-count and high-precision descriptive summaries are available through a separate validated API without changing these ordinary workflows. Their numeric, zero-depth, DuckDB, persistence, and reproducibility contracts are documented in [Validated descriptive statistics](VALIDATED_STATISTICS.md). The staged first-party JSON migration toward the [I-JSON/JCS estate standard](docs/ijson-jcs.md) records its scope and compatibility exclusions separately.
 
 ## Prerequisites
 

VALIDATED_STATISTICS.md

@@ -101,6 +101,8 @@ input matrix as strings, making the direct derived record self-contained.
 - Stored approximate `BigFloat` values use a decimal string that can be parsed
   directly at the recorded precision without an intermediate `Float64`.
 
+Validated-statistics requests are also checked against the strict I-JSON profile on the raw request bytes (duplicate keys, unsafe integer tokens and invalid Unicode are rejected with HTTP 400). Float-form count tokens still fail with HTTP 422. Responses and persisted derived records use JCS bytes; see [the migration notes](docs/ijson-jcs.md).
+
 The request body is limited to 16 MiB and a validated matrix to 100,000 cells.
 Other reviewed limits are defined in `src/analysis/numeric_policy.jl`; exceeding
 one returns an explicit resource error rather than a float fallback.

docs/ijson-jcs.md

@@ -0,0 +1,17 @@
+# I-JSON/JCS migration (D306)
+
+The reference is [hyperpolymath/ijson-jcs](https://github.com/hyperpolymath/ijson-jcs) (reviewed at `3a9a4ee`). I-JSON (RFC 7493) is an **input/value profile**, not a different JSON object schema; JCS (RFC 8785) is its deterministic **byte encoding**. This migration is incremental: do not relabel ordinary JSON as canonical or silently recompute old IDs.
+
+## First-party boundary implemented here
+
+- Julia `MetaManifold.Canonical.ijson_read(raw)` validates UTF-8, repeated object keys (including escaped aliases at every depth), lone surrogates/noncharacters, non-finite numbers and integer tokens outside ±(2^53−1) **on the raw text**. It returns ordinary JSON3 values. `jcs_bytes` validates emitted values (including noncharacters) and sorts keys by UTF-16 code units, formats ECMAScript numbers and emits no newline. The existing `jcs_sha256` hashes those bytes.
+- Validated-statistics request bodies and their responses, error responses, derived-analysis records, figure documents, tree **view-state sidecars**, and report basket indices use those helpers. Stored derived analyses must be canonical on read; existing correctly persisted JCS records are unchanged. Invalid first-party requests get 400 (the exact-count endpoint still rejects floating *count tokens* with 422).
+- The frontend's first-party API POST/PATCH/PUT payloads use `ijsonJcsStringify`, which sorts object keys by UTF-16, preserves array order and refuses unsafe JS numbers, non-finite values, lone surrogates, noncharacters and sparse arrays. Undefined optional object properties are omitted as before. Exact large counts remain decimal **strings**. `frontend/public/config.json` is canonical (plus the optional one-file newline).
+
+Use the upstream `ijson-jcs check frontend/public/config.json` to verify the checked-in config if the Rust CLI is installed. The Julia writer has its own JCS fixture tests, but cross-language equivalence with the Rust binary has **not** been independently verified here.
+
+## Boundaries not yet migrated
+
+Other Oxygen `json(...)` route responses, legacy JSON3 inputs/outputs, pipeline job/QC JSON, DOI receipts and epistemic receipt hashes still use their old serialization. The latter two are part of existing checksums/IDs: switching their byte encoding without an explicit versioned migration would make previously stored receipts unverifiable. External JSON (notably jplace, Plotly, renv metadata and imported report `.json` files) must not be rewritten just because its name ends in `.json`; the owning format and compatibility contract take precedence. Existing figure/view/index documents with merely different whitespace or key order still read; they are written canonically on their next save. Documents with duplicate keys or unsafe values need explicit migration.
+
+Follow-up: audit all remaining first-party route JSON with representative large result sets and tests; decide whether oversized legacy numeric fields become strings and how existing persisted records migrate, then replace their serializers and add an upstream CLI interoperability check to CI. Do **not** automatically JCS-encode opaque upstream/third-party formats or change an established content hash without versioning.

frontend/public/config.json

@@ -1,3 +1 @@
-{
-  "apiBase": ""
-}
+{"apiBase":""}

frontend/src/api/client.ts

@@ -28,6 +28,7 @@ import type {
 
 import type { FigureDoc } from '../figure/types'
 import { saveBlob } from '../utils/download'
+import { ijsonJcsStringify } from './ijsonJcs'
 
 // Base URL for the backend API. Empty string means same-origin.
 // Set "apiBase" in config.json (e.g. "https://bioserver:8080") for split deployments.
@@ -55,7 +56,7 @@ async function rawPost(path: string, body: unknown): Promise<Response> {
   const res = await fetch(apiUrl(path), {
     method: 'POST',
     headers: { 'Content-Type': 'application/json' },
-    body: JSON.stringify(body),
+    body: ijsonJcsStringify(body),
   })
   if (!res.ok) {
     const err: ApiError = await res.json().catch(() => ({
@@ -83,9 +84,9 @@ async function request<T>(path: string, init?: RequestInit): Promise<T> {
 }
 
 const get  = <T>(path: string)                       => request<T>(path)
-const post = <T>(path: string, body?: unknown)       => request<T>(path, { method: 'POST',  body: body !== undefined ? JSON.stringify(body) : undefined })
-const patch = <T>(path: string, body?: unknown)      => request<T>(path, { method: 'PATCH', body: body !== undefined ? JSON.stringify(body) : undefined })
-const put  = <T>(path: string, body?: unknown)       => request<T>(path, { method: 'PUT',  body: body !== undefined ? JSON.stringify(body) : undefined })
+const post = <T>(path: string, body?: unknown)       => request<T>(path, { method: 'POST',  body: body !== undefined ? ijsonJcsStringify(body) : undefined })
+const patch = <T>(path: string, body?: unknown)      => request<T>(path, { method: 'PATCH', body: body !== undefined ? ijsonJcsStringify(body) : undefined })
+const put  = <T>(path: string, body?: unknown)       => request<T>(path, { method: 'PUT',  body: body !== undefined ? ijsonJcsStringify(body) : undefined })
 const del  = <T>(path: string)                       => request<T>(path, { method: 'DELETE' })
 
 /** PUT a text body; the response is JSON. */

frontend/src/api/ijsonJcs.test.ts

@@ -0,0 +1,30 @@
+import { expect, test } from 'bun:test'
+import { ijsonJcsStringify as jcs } from './ijsonJcs'
+
+test('JCS sorts by UTF-16 code units recursively and preserves array order', () => {
+  expect(jcs({ b: 1, a: { z: 1, x: [3, 2] }, 'Ａ': 4, '😀': 5 }))
+    .toBe('{"a":{"x":[3,2],"z":1},"b":1,"😀":5,"Ａ":4}')
+  // Numeric-like keys are sorted too, unlike JSON.stringify's object enumeration.
+  expect(jcs({ '10': 1, '2': 2 })).toBe('{"10":1,"2":2}')
+  expect(jcs({ optional: undefined, exact: '9007199254740992' })).toBe('{"exact":"9007199254740992"}')
+})
+
+test('ECMAScript number formatting and escaped controls match JCS', () => {
+  expect(jcs({ a: -0, b: 1e-7, c: 1e-6, d: '\n' }))
+    .toBe('{"a":0,"b":1e-7,"c":0.000001,"d":"\\n"}')
+})
+
+test('I-JSON rejects unsafe and non-finite numbers, noncharacters and lone surrogates', () => {
+  for (const value of [Number.MAX_SAFE_INTEGER + 1, NaN, Infinity, -Infinity]) {
+    expect(() => jcs({ count: value })).toThrow()
+  }
+  for (const text of ['\ud800', '\udc00', '\ufdd0', '\uffff', '\u{1ffff}']) {
+    expect(() => jcs({ text })).toThrow()
+    expect(() => jcs({ [text]: 1 })).toThrow()
+  }
+  expect(() => jcs({ a: [undefined] })).toThrow()
+  expect(() => jcs(new Array(1))).toThrow()
+  const cycle: { self?: unknown } = {}
+  cycle.self = cycle
+  expect(() => jcs(cycle)).toThrow()
+})

frontend/src/api/ijsonJcs.ts

@@ -0,0 +1,58 @@
+// SPDX-License-Identifier: AGPL-3.0-only
+// First-party API JSON: I-JSON values in RFC 8785 canonical form. JS string
+// ordering is UTF-16 code-unit ordering, as required by JCS (unlike UTF-8).
+// This deliberately doesn't canonicalise external jplace/Plotly documents.
+
+function validString(value: string): void {
+  for (let i = 0; i < value.length; i++) {
+    const code = value.codePointAt(i)!
+    if (code > 0xffff) i++ // a valid UTF-16 surrogate pair
+    else if (code >= 0xd800 && code <= 0xdfff) throw new TypeError('I-JSON refuses unpaired surrogates')
+    if ((code >= 0xfdd0 && code <= 0xfdef) || (code & 0xfffe) === 0xfffe) {
+      throw new TypeError('I-JSON refuses Unicode noncharacters')
+    }
+  }
+}
+
+// JCS sorts UTF-16 code units, NOT human-language collation (localeCompare).
+const compareUtf16 = (a: string, b: string): number => a < b ? -1 : a > b ? 1 : 0
+
+/** Canonicalise a JSON-shaped API payload; reject values that cannot round-trip. */
+export function ijsonJcsStringify(value: unknown): string {
+  const active = new Set<object>()
+  function encode(input: unknown): string {
+    if (input === null) return 'null'
+    if (typeof input === 'string') { validString(input); return JSON.stringify(input) }
+    if (typeof input === 'boolean') return String(input)
+    if (typeof input === 'number') {
+      if (!Number.isFinite(input) || (Number.isInteger(input) && !Number.isSafeInteger(input))) {
+        throw new TypeError('I-JSON refuses non-finite or unsafe numbers; use a decimal string for exact counts')
+      }
+      return JSON.stringify(input)
+    }
+    if (typeof input !== 'object') throw new TypeError(`Not an I-JSON value: ${typeof input}`)
+    if (active.has(input)) throw new TypeError('Cyclic JSON value')
+    if (!Array.isArray(input) && Object.getPrototypeOf(input) !== Object.prototype && Object.getPrototypeOf(input) !== null) {
+      throw new TypeError('I-JSON objects must be plain objects')
+    }
+    active.add(input)
+    try {
+      if (Array.isArray(input)) {
+        return `[${Array.from({ length: input.length }, (_, i) => {
+          if (!(i in input)) throw new TypeError('I-JSON refuses sparse arrays')
+          return encode(input[i])
+        }).join(',')}]`
+      }
+      const object = input as Record<string, unknown>
+      // Omit undefined optional properties just as JSON.stringify does, but
+      // never convert an undefined array element into null silently.
+      return `{${Object.keys(object).sort(compareUtf16).filter(key => object[key] !== undefined).map(key => {
+        validString(key)
+        return `${JSON.stringify(key)}:${encode(object[key])}`
+      }).join(',')}}`
+    } finally {
+      active.delete(input)
+    }
+  }
+  return encode(value)
+}

src/analysis/derived_analyses.jl

@@ -7,7 +7,7 @@
 module DerivedAnalyses
 
 using JSON3
-using ..Canonical: jcs_bytes, jcs_sha256
+using ..Canonical: jcs_bytes, jcs_sha256, ijson_read
 
 export DerivedAnalysisError, derived_analysis_id, persist_derived_analysis,
        load_derived_analysis, MAX_DERIVED_BYTES
@@ -43,13 +43,15 @@ function _validate_bytes(bytes::Vector{UInt8})
     length(bytes) <= MAX_DERIVED_BYTES || throw(DerivedAnalysisError(
         "invalid_record", "derived analysis exceeds the $MAX_DERIVED_BYTES-byte read limit"))
     record = try
-        JSON3.read(bytes)
+        ijson_read(bytes)
     catch err
         throw(DerivedAnalysisError("invalid_record",
             "derived analysis is not valid JSON: $(sprint(showerror, err))"))
     end
     (record isa JSON3.Object || record isa AbstractDict) || throw(DerivedAnalysisError(
         "invalid_record", "derived analysis root must be a JSON object"))
+    jcs_bytes(record) == bytes || throw(DerivedAnalysisError(
+        "invalid_record", "derived analysis is not canonical I-JSON/JCS"))
     _get(record, "schema") == SCHEMA || throw(DerivedAnalysisError(
         "invalid_record", "derived analysis has an unsupported schema"))
     id = _get(record, "analysis_id")