Repository navigation
Conversation
Give one round a plan-supplied attempt budget and a fail-closed execution boundary so lanes can submit synthetic candidates without sharing a device lease. Refs InferMatrixLab#249 Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Dong1017
left a comment
There was a problem hiding this comment.
Independent source review of head 56a40049148ced7bb80e60249a8f5df40d5c1587, against API base / observed main f93df0548a26714f6514337a8525c7c14108acad. Requesting changes for three current execution-contract defects, detailed inline: protected read/environment exposure, claim persistence after execution, and incomplete absolute-deadline enforcement.
I traced prepare/submit/serve, lane-budget transitions, confinement, leases, the CLI and all added tests against the existing architecture and #249 discussion. The attempt-scoped digest and permanently retained reservation charge are useful; they do not clear the inline execution-boundary findings. The Draft/partial-delivery description is appropriately explicit. Existing M0–M2 assignment/scope acceptance remains a maintainer decision, separate from this code review.
Verification was source and existing hosted-status inspection only. I did not execute this branch, tests, namespaces, models, devices or a paid workflow. Current hosted evidence contains the declaration check; no Ready full-check/installed-package result was observed for this head. Fixture results and author-reported tests do not qualify OS confinement or a campaign outcome. Next for the existing author @NumberWan: fix the three boundaries and supply the corresponding targeted evidence, preserving unknown/failed execution states and the Draft scope. No pilot, device use, incumbent advance, merge or Issue closure is authorized by this review.
| del readable | ||
| if not writable: | ||
| raise ConfinementUnavailable("refusing to run with an empty writable set") | ||
| config = writable[0] / ".inferrsi-confine.json" | ||
| payload = { | ||
| "argv": list(argv), | ||
| "writable": [str(path) for path in writable], | ||
| "env": dict(os.environ), |
There was a problem hiding this comment.
P1 — Enforce the declared read/credential boundary before candidate execution. readable is discarded, the existing host root remains visible in enter_namespace(), and the complete coordinator environment is copied into a JSON file under a candidate-writable directory and passed to execvpe. On a host where namespace setup succeeds, a build/trial can still read coordinator/grader files and other lanes that the host user can read; any coordinator-only environment credentials are also supplied to it. A read-only remount does not make these inputs invisible. This contradicts the explicit invisible-set contract in #249. Construct the allowed filesystem view and a minimal environment without exporting the full host environment, and fail closed if that boundary cannot be established. Verify build and trial stages with harmless private canaries; the argv-only FixtureBackend tests do not establish this property.
| incumbent_body = json.loads(incumbent_result.read_text(encoding="utf-8")) | ||
| candidate_metric = candidate_body["metrics"]["latency_ms"] | ||
| incumbent_metric = incumbent_body["metrics"]["latency_ms"] | ||
| finished = advance(advance(ledger, binding, "claimed"), binding, "finished", actual_cost=0) |
There was a problem hiding this comment.
P2 — Persist the execution claim before running any build or trial. The only durable state saved before the three commands is reserved; claimed is introduced here after commands have run and result files have been read. A crash/timeout after starting work therefore leaves an executed attempt looking unstarted. A later ConfinementUnavailable (for example after a completed build) even takes the reserved → cancelled path. This is the execution-state ambiguity already distinguished in #249's follow-up. Persist a non-idempotent claim before the first candidate command, distinguish pre-launch refusal from failure after execution starts, and retain the claimed/charged ambiguity until trusted settlement. Add a case that interrupts after launch and verifies restart cannot treat that execution as an unclaimed cancellation. The existing permanent reservation charge should remain.
| candidate, | ||
| (candidate, build_dir), | ||
| (), | ||
| time.monotonic() + 30, |
There was a problem hiding this comment.
P2 — Carry the absolute lane deadline through submission and every launch. admit_attempt(..., now) checks expiry once before materialization, but build, candidate and incumbent each receive a fresh time.monotonic() + 30. An attempt reserved just before expiry can start later commands after the frozen deadline; submit() also accepts new manifests after expiry without checking it. A restart rereading the plan does not close either path. Recheck the trusted clock at submission/claim/launch and bound running work by the same absolute cutoff, rather than resetting a per-command allowance. Cover both “reserve before expiry, launch after expiry” and “start before expiry, submit new bytes after expiry,” including restart, as requested in #249.
There was a problem hiding this comment.
Current-head follow-up (9b4b08b9, deadline finding still partially open): The new _launch.run() calls seconds_remaining(lane['deadline_utc'], current()) for every command, but in the production CLI tools/parallel_round.py calls serve_ticket(..., now=datetime.now(timezone.utc)) without a clock. serve_ticket.current() then returns that same entry-time now for every call (parallel_round.py:380–381). Thus a lane with 5s remaining can still run build, candidate and incumbent for up to 5s each (or three independent 30s caps when more remains), crossing the actual absolute deadline despite the new checks. The advancing-clock test only exercises the injected test callback, not this default. Please have the default current() read a fresh trusted clock for each gate, or derive one monotonically decreasing deadline once and share it across all stages; cover an expiry between build and candidate in a model-free fixture. Source inspection only, no command execution. The earlier thread remains the right place for this issue.
Dong1017
left a comment
There was a problem hiding this comment.
Incremental review of the existing Draft head 56a40049: one previously unraised file-deletion handling gap in the offline submit path (inline). Existing confinement, execution-claim, and deadline findings remain in their original threads; this review does not repeat or resolve them. Scope: static inspection of parallel_round.py, its caller and fixture tests against #249; no tests, namespace checks, model calls, or device experiments were run. This is not approval or M0–M2 acceptance.
| outside = [path for path in changed if not _match(path, lane["globs"])] | ||
| if outside or not changed: | ||
| return {"submitted": False, "outside": outside, "ticket": None} | ||
| manifest = {path: sha256(work / path) for path in changed} |
There was a problem hiding this comment.
P2 — An in-scope deletion crashes submission instead of producing a ticket or explicit rejection. _changed() includes paths present only in incumbent (set(before) | set(after)), so removing an allowed file, e.g. knobs/a.txt under lane glob knobs/a.txt, passes the outside check. This comprehension then calls sha256(work / path) on the deleted path and raises FileNotFoundError; the subsequent copy2 and _materialize likewise lack a deletion representation. Thus a normal scoped change cannot be submitted or deterministically measured. Either encode deletion/tombstone entries in the ticket manifest and apply removals when materializing, or explicitly reject deletions with a structured status before hashing. Add a model-free deletion-in-scope fixture covering submit and serve. This is a static code-path finding, not an executed test.
Dong1017
left a comment
There was a problem hiding this comment.
One additional blocking file-boundary finding at 56a40049: submission can copy a coordinator-readable external file through a lane-controlled symlink. This intake path is distinct from the existing candidate-execution confinement finding and must be closed before accepting untrusted lane submissions.
Verified with a disposable synthetic external canary and the exact pinned function bodies; no real private file or candidate command was accessed/executed. Existing findings remain in their threads; no duplicate findings or broader feature requirements are added.
Verification boundary
Function-level reproduction used AST-extracted prepare/submit and necessary helpers/artifact functions, standard library only, with a sanitized environment. Allowed path knobs/a.txt was replaced by a symlink to a synthetic sibling file outside work. Result: submitted=true, outside=[], source_is_symlink=true, outbox_is_regular_copy=true, copied_external_canary=true. This is not a full CLI or OS-confinement qualification; no model/hardware/network execution or repository modification occurred.
| for path in changed: | ||
| target = destination / path | ||
| target.parent.mkdir(parents=True, exist_ok=True) | ||
| shutil.copy2(work / path, target) |
There was a problem hiding this comment.
P1 — Reject lane-controlled symlinks before hashing or copying submissions. _files() admits symlinked files through is_file(), and _changed() reads their targets before this relative-path scope check. Replacing permitted knobs/a.txt with a symlink to a coordinator-readable file outside work still yields outside=[]; submission succeeds and copy2 writes the external contents into the outbox as a regular file. A synthetic sibling-canary reproduction confirmed this without running candidate commands. Fixing candidate namespace confinement alone does not protect this earlier intake path.
Reject symlinks/nonregular entries before any content read, and use root-confined, race-resistant no-follow reads for snapshot/hash/copy, including parent components. Checking resolve() and then reopening normally leaves a replacement race. Add a synthetic external-canary rejection test; no real sensitive files are needed.
Refuse symlinks and in-scope deletions before any content read, persist a claim before the first candidate command, and bound every launch by the frozen deadline. Confinement now builds a fresh root and a minimal environment instead of remounting the host root. Refs InferMatrixLab#249 Signed-off-by: NumberWan <wantszkin2003@gmail.com>
|
Addressed the five findings on 56a4004 in 9b4b08b.
Still out of scope: OS confinement on this host (user namespaces are unavailable, so the default backend fails closed), M3, incumbent advance, and shared drivers. |
Dong1017
left a comment
There was a problem hiding this comment.
Incremental static review of head 9b4b08b9 following the author's five targeted fixes: one newly identified expected-failure reporting gap in the unshare backend (inline). Existing confinement, deadline, claim and symlink findings remain on their original threads; this does not repeat, dismiss or approve them. I inspected the new confinement.py, parallel_round.py, CLI caller and available fixture tests; no real namespace, test, model or device run was performed. This is not an M0–M2 acceptance or launch authorization.
| stdin=subprocess.DEVNULL, | ||
| capture_output=True, | ||
| text=True, | ||
| timeout=remaining, |
There was a problem hiding this comment.
P2 — Persist a terminal unsettled receipt when the real namespace subprocess times out. subprocess.run(..., timeout=remaining) raises subprocess.TimeoutExpired on the normal time-limit path. Here only the finally cleanup runs; the exception escapes UnshareBackend.run. In serve_ticket, the claimed ledger has already been persisted, but its handler only catches ConfinementUnavailable and BudgetError, so this timeout exits without writing results/<ticket>.json. A subsequent status call incorrectly reports queued (until a new serve invocation detects the claim). Catch the expected subprocess timeout, preserve the claimed/charged ambiguity, write an explicit unsettled receipt with stage/reason, and test this with a stubbed timeout—no real namespace is needed. This is separate from proving OS confinement on a host.
Dong1017
left a comment
There was a problem hiding this comment.
Changes requested at 9b4b08b9: the new filesystem view still lacks the runtime files needed to start a dynamically linked Python command.
- One new P2 startup defect is detailed inline. It affects the real namespace backend on an otherwise namespace-capable Linux host.
- The existing deadline follow-up and timeout-receipt finding remain applicable; this review does not repeat them.
- The persisted pre-launch claim and explicit deletion refusal improve the earlier paths. They do not establish native confinement qualification or complete the parent task. Next action remains with @NumberWan: repair the current blockers and provide bounded software evidence.
Evidence and scope
Reviewed head: 9b4b08b9f7b4bde0f18d6e3db80d8e47cf059539; API base: f93df0548a26714f6514337a8525c7c14108acad; assessed main: 919ed42c0d01dc0aa3b613085b9041d735d2d4db.
Inspected the six-file repair delta, current prepare/submit/serve and CLI consumers, runtime-root/mount/exec paths, and relevant fixture tests against the existing execution contract. Current per-head evidence contains successful declaration checks; no Ready full-check or installed-package success was observed. Verification was source and existing hosted evidence only: no repository tests, namespaces, model/device runs or paid evaluation were executed. This review did not implement the PR and grants no launch, merge or goal-acceptance authorization.
|
|
||
| def runtime_roots() -> tuple[Path, ...]: | ||
| """Python prefix needed to start the interpreter. Never the host root.""" | ||
| candidates = [Path(sys.prefix).resolve(), Path(sys.executable).resolve().parent] |
There was a problem hiding this comment.
P2 — Include the executable's runtime dependency closure in the fresh root. runtime_roots() contributes only sys.prefix and the executable's parent. For a normal dynamically linked CPython with prefix /usr/local (or /usr), the ELF interpreter and some shared libraries are reached through /lib* paths outside those mounts. enter_namespace() creates only the planned bind targets before chroot and then calls execvpe; it does not create the required loader paths/symlinks. Consequently, a namespace-capable Linux host can build the jail successfully but fail even the initial Python build/trial with ENOENT for the missing ELF interpreter, before any task code runs. This is distinct from namespace availability and the existing timeout finding.
Construct an explicit read-only runtime image/allowlist that includes the interpreter's required loader and shared-library paths, while retaining the private-file boundary; do not restore the host-root bind. A model-free Linux smoke case should start the actual UnshareBackend with a trivial Python command and separately confirm that a private canary stays invisible. The current tests intercept subprocess.run or use the import-hook fixture, so they do not cover this startup path. Static path analysis, not an executed namespace reproduction; the loader/ENOENT behavior is documented by execve(2).
…e jail A namespace timeout now leaves an unsettled receipt, each launch reads a fresh clock, and the fresh root includes the dynamic loader and its libraries. Refs InferMatrixLab#249 Signed-off-by: NumberWan <wantszkin2003@gmail.com>
|
Addressed the three findings on 9b4b08b in fc9d232.
Still out of scope: OS-confinement qualification on this host, M3, incumbent advance, and shared drivers. |
There was a problem hiding this comment.
Reviewed commit: fc9d232
Coverage: Snapshot: 2221 files, 35993584 bytes; skipped 0 entries; Initial model context: 22 files; other accepted files available through tools; Git workspace contains a fixed snapshot, not full upstream history
| now=datetime.now(timezone.utc), | ||
| ) | ||
| elif args.command == "serve": | ||
| result = serve_ticket( |
There was a problem hiding this comment.
F1 — critical, must-fix, regression — serve CLI is broken: serve_ticket() still requires now, but this commit removed it from the only call site
- Where:
tools/parallel_round.py:53(signature atsrc/inferrsi/harness/parallel_round.py:394) - Problem:
serve_ticket(round_dir, ticket, incumbent, *, backend, now: datetime, clock=None)keepsnowas a required keyword-only parameter while this commit deletednow=datetime.now(timezone.utc)from the CLI call (tools/parallel_round.pydiff, -1 line) and moved the fresh-clock behaviour intocurrent().nowis now never read in the body, so the parameter is dead but still mandatory. Reproduced:PYTHONPATH=src python -m tools.parallel_round serve --round-dir /tmp/rd --ticket abc --incumbent /tmp→TypeError: serve_ticket() missing 1 required keyword-only argument: 'now'. - Conditions: every
serveinvocation throughtools/parallel_round.py, i.e. the documented coordinator path the PR advertises. All tests pass only because each call intests/test_parallel_round.pypassesnow=WHEN. - Impact: the coordinator cannot measure any ticket outside unit tests; the
servepath fails before reading the plan, so the PR's main entry point is unusable. - Fix: make the parameter optional or drop it, e.g.
now: datetime | None = None(keepcurrent()readingdatetime.now(timezone.utc)so the fresh-deadline contract from the earlier round stays), or passnow=datetime.now(timezone.utc)again from the CLI. Add a CLI-level regression test that callsmain(["serve", ...]), not only directserve_ticketcalls.
| assert "SECRET_CANARY" not in plan["env"] | ||
|
|
||
|
|
||
| @pytest.mark.skipif(not namespaces_available(), reason="user namespaces are unavailable") |
There was a problem hiding this comment.
F2 — high, must-fix, test gating — the live namespace smoke fails instead of skipping when unprivileged user namespaces are blocked
- Where:
tests/test_confinement.py:135 - Problem: the guard only reads
/proc/sys/user/max_user_namespaces. When that value is positive but namespace creation is denied, the test runs and errors:UnshareBackend().run→ConfinementUnavailable: unshare: unshare failed: Operation not permitted. Executed in this container:PYTHONPATH=src python -m pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.py→1 failed, 23 passed, the failure beingtest_unshare_starts_python_and_hides_a_private_canary. Heremax_user_namespaces= 498239 andkernel.apparmor_restrict_unprivileged_userns= 1, whileunshare --user truereturns EPERM. - Conditions: unprivileged containers (default seccomp profiles) and AppArmor-restricted unprivileged-userns hosts, which is the default on Ubuntu 24.04+ and therefore on common GitHub-hosted runners. This is unrelated to whether the host has namespaces at all.
- Impact: the repository's full
pytest(ready and nightly lanes) goes red on such hosts, and the PR's statement that the smoke "skips where user namespaces are unavailable" is wrong, so this test cannot serve as the confinement qualification evidence it claims to be. - Fix: probe the syscall rather than the sysctl, e.g.
subprocess.run(["unshare", "--user", "true"], ...)and skip on a non-zero return, or wrapUnshareBackend().run(...)intry/except ConfinementUnavailableandpytest.skip(str(exc)).
| ) | ||
| except (ConfinementUnavailable, BudgetError) as exc: | ||
| return _record(round_dir, ticket, status="unsettled", reason=str(exc)) | ||
| candidate_body = json.loads(candidate_result.read_text(encoding="utf-8")) |
There was a problem hiding this comment.
F3 — medium, suggestion, error handling — a trial that exits 0 without a valid {result} escapes as an uncaught exception and leaves no terminal receipt
- Where:
src/inferrsi/harness/parallel_round.py:446(unchanged since the previous reviewed commit) - Problem:
_launch.runonly checksrecord["status"] != "completed", thenserve_ticketreadstrial_dir/candidate.jsonandincumbent.jsonoutside theexcept (ConfinementUnavailable, BudgetError)handler. If the result file is missing or not valid JSON,FileNotFoundError/JSONDecodeErrorpropagate out ofserve_ticket; the ledger entry staysclaimedandresults/<ticket>.jsonis never written. This is the same receipt gap that was fixed forsubprocess.TimeoutExpiredin this round. - Conditions: a trial command that exits 0 without writing exactly the
{result}path (a stub, or a benchmark whose output path was changed/broken) or that writes a non-JSON body. - Impact:
serveaborts with a traceback,statusreportsqueueduntil anotherserverun detects the claim, and the attempt remains charged with nounsettledreceipt. - Fix: move the result read/parse into a
trythat records_record(round_dir, ticket, status="unsettled", reason=f"{stage} result missing/invalid")instead of letting the exception escape (the claim itself should stay as it is).
|
This PR adds the offline parallel-round coordinator (prepare/submit/serve/status), lane attempt budgets, a fail-closed confinement backend and its tests. Two blocking defects: the Reviewed commit: fc9d232 |
Summary
prepare/submit/servepath for one parallel round (Refs #249). Each attempt gets its own plan digest, the lane cap comes from the frozen plan, and the deadline is checked beforereserve.confined_executerefuses to run when user namespaces are unavailable. It does not fall back to a plain subprocess. The fixture backend used in tests only rejects argv paths outside the writable set; it is not an OS boundary.This is a draft slice, not completion of M0–M2. It does not add Claude Code or Codex drivers, a SwarmFlow template,
close, incumbent advancement, or campaign-template fields. #249 still has no assignee; this pull request does not treat that scope as accepted and does not authorize the paid pilot.Test plan
pytest tests/test_round_budget.py tests/test_confinement.py tests/test_parallel_round.pyruff checkon the new files