[SamGu-NRX]

Zephyrus task Z10: audit of the server pipeline on alpha at 7e6b2ef3. Step 1 ran the backend, verification and recon suites. Step 2 sent hostile input through the real FastAPI apps and the placement solver. This branch fixes the two most severe new problems; the rest are in the table below. There is no parent PR, so there were no review threads to read.

What changed

Finalize refuses NaN and infinite numbers before accepting a packet

Python's json.loads reads NaN, Infinity and -Infinity, and turns an overflowing literal such as 1e999 into inf. None of these is JSON, and the packet models take any float (backend/server/contracts/_base.py:24). On alpha, accept_packet (backend/server/api/finalize.py:210) stored such a packet and answered 202. Two outcomes followed, both shown by the probes:

• NaN or 1e999 in stills[].gravity, stills[].timestamp or session.startedAtUptime went on to processing and reached the pipeline.
• NaN or inf in a keyframe pose, intrinsics, timestamp or tap ray was caught by the background manifest check, but only after the 202 and the full upload. The capture then ended failed with next: none.

accept_packet now walks the parsed packet and answers 422 schema with a JSON pointer to the first non-finite number (for example /stills/0/gravity/1), before it stores anything. The capture stays in uploading, so a corrected packet still finalizes. The check sits at the API boundary rather than in _base.py, because the shared contract types are also used for output that this audit did not trace.

A second API container can no longer accept a different packet

The per-capture finalize lock (backend/server/api/routes/finalize.py:36, backend/server/api/locks.py) works within one process. modal_entry.py:35 allows up to four API containers. accept_packet decides whether a packet was already accepted from a record it read before the request body (finalize.py:192). So two containers that both load the capture before either writes can both accept.

I reproduced this with two app instances that share one state store and one volume, two threads, and a 300 ms delay on the packet write to stand in for a GCS PUT. Container A answered 202 with run_2c51…. Container B answered 202 with a different packet and run_337e…, and B's bytes replaced packet.json. Only B's run started, so A's client was polling for a run that never existed. Identical packets were already deduplicated downstream by the run claim in orchestrator/api.py:55.

Before writing the packet, accept_packet now claims accepted_packet:<capture>:<amendment or -> with state.put_if_absent, which is atomic on both state backends (Modal Dict uses skip_if_exists). A different digest gets the 409 packet_sha_conflict that a single container already returns. The same digest, already claimed, writes nothing and schedules nothing: only the claimant writes packet.json and starts the finish, and the retry replays the claimant's 202 (processing, or awaiting_files with the same missing list; the run id comes from the bytes, so it matches). An earlier version let the retry write packet.json again. The volume backend's write truncates before it writes (storage_local.py:32), so the first worker could read an empty file and fail with JSONDecodeError. Review of 3c24e69a caught it, and 3864c436 fixes it. If the packet write fails, the claim is released, so a corrected packet can still finalize. With the fix, the same two-thread probe returns 202 and 409, and the stored packet is A's.

Step 1: suites on alpha

Suite	Command	Result	Failures and cause
verification	uvx --from "uv>=0.11.15,<0.12" uv sync --locked && … uv run pytest -q	596 passed	None.
recon	same as above	168 passed	None.
backend (as documented)	uv sync --locked && OMP_NUM_THREADS=1 uv run pytest -q	0 run, collection aborts	server/api/tests/test_events_orchestrator_wake.py:18 imports uvicorn, which is only in the api and ios-kit extras, not the dev group. No CI workflow runs backend/, so nothing caught this.
backend with --extra api	uv sync --locked --extra api && OMP_NUM_THREADS=1 uv run --extra api pytest -q	1515 passed, 5 failed, 14 errors, 16 skipped (42 min)	14 errors: tests/test_capture_cold_restart.py, finding 1. 4 failures in server/eval/tests/test_gates.py, finding 3. 1 failure in server/packet/tests/test_round2.py::test_many_missing_files_stay_linear, finding 6.
backend ruff	uv run ruff check . && uv run ruff format --check .	check passes, format fails	server/orchestrator/tests/test_greptile_followup.py:163 is not formatted.

verification/pyproject.toml:21 and recon/pyproject.toml:17 pin required-version = ">=0.11.15,<0.12". The WSL machine has uv 0.13.5, so plain uv sync --locked refuses to run there. I ran both through uvx with a uv inside the pin.

pytest-xdist is not installed in any of the three environments, so every run was single-process.

Step 2: other findings

#	Where	Failing input	Severity	Minimal fix
1	backend/scripts/capture_cold_restart.py:82-88	Any run of tests/test_capture_cold_restart.py. All 14 tests error with ImportError: cannot import name load_receipt.	Medium. The client cold-restart recovery tests had not run on alpha since 596ed9a9 (Oct 16) replaced load_receipt with receipt_problem in scripts/replay_capture.py.	Fixed in 3c24e69a: restored the load_receipt JSON wrapper in replay_capture.py; collection now succeeds and the touched suite passes 30 tests.
2	backend/pyproject.toml dev group; test_events_orchestrator_wake.py:18	The documented uv sync --locked && uv run pytest.	Medium. The whole backend suite stops at collection.	Add uvicorn to the dev group and relock. I left this alone because lockfiles are off limits for this task. A backend CI job would also need .github/, which is off limits too.
3	backend/server/eval/gates.yaml:42	Any run after 2026-10-04. known_failure.until: "2026-10-04" on the gt-laser-gravel footprint3in row has expired, so the row reports FAIL rather than KNOWN_FAIL. Four tests read that row: test_committed_fixtures_are_scored_against_their_labels, test_offline_table_skips_the_engine_and_passes_the_unpatched_reference, test_route_over_the_bar_is_known_until_expiry and test_refresh_rewrites_pins_after_a_committed_edit.	Medium. The tests fail by date, not because of a code change.	The owner (WS-09) decides whether to extend until or fix the SD-96 miss. Eval thresholds are off limits for this task. The tests could also pin today's date so that expiry is tested on purpose.
4	backend/server/api/routes/files.py:231-241	A keyframe meta whose pose holds NaN, sent at file registration. It answers 200 and the record keeps nan.	Low. The finalize packet is authoritative and is now refused; only the per-file hook sees the bad meta.	Fixed in c7094b1c: registration rejects non-finite metadata with a schema pointer before persistence. New test failed on the test-only commit 7007328c and passes after the fix.
5	backend/server/placement/scene_bridge.py:328	A one-wall scene whose baseline_m repeats its first point.	Low. The refusal points at /walls/0/baseline, but the field is baseline_m.	Fixed in c7094b1c: pointer now names /walls/0/baseline_m. New test failed on the test-only commit 7007328c and passes after the fix.
6	backend/server/packet/tests/test_round2.py:302	200,000 missing files on a machine at load average ~30. It took 61.4 s against a 60 s budget.	Low. This is a wall-clock budget on a shared machine; the scan itself stays linear.	Measure CPU time (time.process_time) instead of wall time, or compare against a smaller run so the check tests growth rather than speed.
7	backend/server/orchestrator/tests/test_greptile_followup.py:163	ruff format --check .	Low.	Run ruff format on the file.

Inputs that the code already handles correctly, each checked through the real app:

• Placement API (server/api.py), zip bundles: zip64 entries, .., backslash, absolute and drive-letter entry names, a declared size of 4 GiB, a scene.json whose declared size is smaller than its real size, an 11 MiB scene.json, LZMA and stored compression, and a truncated zip. Each one gets the documented 4xx. The bound on the central directory's entry count arrives with Server hardening: camera-evidence coverage, solver budgets, bounded bundles, provenance #221, which Z8 is porting.
• Placement API, numbers: NaN, Infinity, 1e999, -1e999, 100,000-deep nesting and a 5,000-digit integer. All get a 422.
• Placement solver, degenerate walls: zero-length walls, single points, walls under 1e-6 ft, repeated first or last points, collinear middle points, collinear walls that join, reversed walls, overlapping collinear walls, the same wall twice, a self-crossing baseline and coordinates near 1e5 ft. Every case ends in a 422 or in pass, reject or manual_review. None returned a 500. Through the backend bridge (scene_bridge.py), the same walls plus NaN and inf are refused with placement_scene_invalid.
• Capture API (backend/server/api): formatVersion values 0.5, 0.4.1, 1.0, " 0.4", 0.40, empty, null and the number 0.4, both at create and at finalize. File paths with .., ./, //, a leading /, %2e%2e, a backslash, a dotless ı, a NUL byte, a leading dot and packet.json. Packet img and id values with ... A gzip bomb (49 KB inflating to 49 MiB), truncated gzip, deflate, GZIP and gzip, identity. Bodies over 1 MiB and over 48 MiB, including chunked bodies with no length.
• Idempotency: a repeated create replays the 201 body with 200. A different body under the same key, another principal's packetId, and a key that is not the packetId all get 409. A create after delete gets 410. A repeated finalize replays its 202, and a different packet gets 409.
• Concurrent finalize inside one container: two, and then five, simultaneous finalizes of one capture start exactly one run. A different body gets 409. This holds with all files committed and with files still missing.

Verification

Failing first. The tests are in backend/server/api/tests/test_finalize_hostile.py, and the test-only commit c6586261 sits on top of alpha.

cd backend && uv run --extra api pytest -q server/api/tests/test_finalize_hostile.py
c6586261 (tests only, alpha code): 5 failed, 1 passed
7f073804 (fix):                    6 passed
7d3a996e (+ release guard):        7 passed
3c24e69a (review base):            1 failed, 7 passed (new retry test: 2 packet writes)
3864c436 (retry writes nothing):   8 passed, 20 of 20 runs
3864c436 + takeover tests:         2 failed, 1 passed ((0, 0) == (1, 1) writes and jobs)
49904399 (abandoned-claim takeover): 11 passed, 20 of 20 runs

The four non-finite cases fail on alpha with 202 == 422. The two-container case fails with DID NOT RAISE ApiFailure. The identical-packet replay test guards against the claim turning a retry into a conflict. At 3c24e69a it failed about one run in three, because the retry rewrote packet.json while the worker read it. test_a_same_packet_retry_on_another_container_writes_nothing makes that race deterministic by counting packet writes and finish jobs. test_a_failed_packet_write_releases_the_claim cannot fail on alpha, because alpha has no claim to release. To check that the test works, I made delete_kv a no-op, and it then fails with 409.

• OMP_NUM_THREADS=1 uv run --extra api pytest -q server/api/tests: 223 passed at 49904399.
• test_a_retry_leaves_a_fresh_claim_alone cannot fail on 3864c436, because that commit never schedules on a retry. With the abandonment bound replaced by -1, it fails with (1, 1) == (0, 0).
• Full backend suite (uv run --extra api pytest -q, single process), run at 7d3a996e and not rerun since. On the branch, 1523 passed, 4 failed, 14 errors and 16 skipped. On alpha, 1515 passed, 5 failed, 14 errors and 16 skipped. The difference is the 7 new tests, plus test_many_missing_files_stay_linear, which passed this time under lighter load. The 4 failures and 14 errors left are findings 3 and 1, and they are the same on both.
• uv run ruff check . passes. uv run ruff format --check . reports only the alpha file in finding 7.

The probes are throwaway scripts that drive create_app() through TestClient, httpx.ASGITransport and two threads, against the volume storage and memory state backends. Nothing ran against Modal, GCS or a real Modal Dict. The cross-container race was reproduced with threads and a delayed write, not with real containers.

Risk

The claim adds one put_if_absent, plus one get_kv when another request already holds it. A capture accepted before this deploy already has packet_sha256 set, so it never reaches the claim. Only a retry holding a stale record reaches it, so later retries skip the extra packet read.

A claimant can die after it claims and before it writes packet.json. A same-packet retry then takes the claim over once the claim is older than web_function_timeout_s() (common/deploy_switch.py:28), which is 600 s, or 1560 s with real runs. That is the timeout modal_entry.py:121 gives the API function, so no live request can be older. No new constant was added. Until the bound passes, retries answer processing and schedule nothing. The store has no compare-and-swap. The takeover inserts <claim>:takeover:<old token> with put_if_absent instead, so exactly one of the retries that read the same abandoned claim wins. The tests cover this on the memory backend only. The Modal Dict path relies on skip_if_exists being atomic, as the first claim already does.

One gap is left. If the claimant writes a matching packet.json and dies before it saves the acceptance and schedules the finish, the packet check sees a valid packet. Retries then keep answering processing without scheduling, until the capture's finalize window expires. Closing it would mean checking for the published acceptance as well as the packet. I didn't do that here.

Claim and takeover keys are not deleted with the capture. That matches the other per-capture keys, and a deleted capture answers 410 anyway.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[SamGu-NRX]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-22T18:21:54.991325Z	7d3a996	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 7d3a996efd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".