task: parallel optimization rounds with orthogonal lanes for multi-agent harnesses #249

Activity

1. david6666666

   added

   kind:taskBounded delivery task with a delegated sliceBounded delivery task with a delegated slice

   area:workbenchCLI workbench: profile, A/B loop, quality gates, evidence reportsCLI workbench: profile, A/B loop, quality gates, evidence reports

   area:processRepo-wide process, governance, privacy and security rulesRepo-wide process, governance, privacy and security rules

   on Oct 29, 2026
2. david6666666 commented on Oct 29, 2026

   

   david6666666

   on Oct 29, 2026

   CollaboratorAuthor

   More actions

   Full plan: parallel optimization rounds with multi-agent harnesses

   Sources inspected on 2026-10-29: InferRSI main 509531c6, openJiuwen agent-core b2a0a248 (the revision jiuwenswarm pins in pyproject.toml) and jiuwenswarm 053c82b0. Line links are permalinks to those revisions. This plan does not authorize any paid call, device time or publication. A Chinese translation follows in the next comment.

   1. Why rounds are serial today

   What parallel lanes need	What InferRSI has	Gap
   Every lane starts from the incumbent	Each round names its parent revision, builds the candidate in an isolated directory and records the import path (RFC). search_state holds incumbent_revision, next_candidate_parent, isolation and import_path (template).	One candidate parent per round, no lane list
   Measurement that no lane can game	run_experiment with AB/BA ordering (runner.py) and evaluate (evaluation.py)	Run by one operator; nothing queues candidates from several agents
   Devices shared safely	qualify_devices refuses a GPU that already has a compute process (cuda.py)	A check, not a lease: two lanes measuring at once fail or contend
   Several wins combined	Harness improvement composes at most two eligible skill-text edits, selects one on a whole-file conflict and requires a fresh rerun (harness-improvement.md, choose)	Skill text only, two edits, no engine code
   Writers isolated from each other	"concurrent writers use separate working directories/indexes and private scratch files. Branch names alone are insufficient." (agent-collaboration.md). Separate directories do not isolate Python imports (architecture.md).	A rule for contributors, not enforced for lanes
   Every lane on record, including failures	search_plan entries with status and evidence, and append-only search_plan_revisions (template). Negative experiment memory in the search policy.	No round-level record of parallel results

   SwarmFlow gives deterministic fan-out, but its defaults do not fit a campaign:

   • agent-core's worktree backend creates new worktrees from origin/<default branch> (backend.py), not from the campaign incumbent. Dirty ones stay "for the leader to merge" (worktree.py).
   • agent() takes no working directory, environment or device option (_ENGINE_OPTIONS).
   • A timed-out worker counts as one failed attempt and is retried twice more by default (swarmflow.md, runtime.py). A cache-hit replay skips the worker and does not recreate its worktree (swarmflow.md).
   • verify() is a reviewer vote with a one-vote veto and a 0.85 score threshold (swarmflow.md, verify.py). It is model judgment, not measurement.
   • Claude Code and Codex workers need enable_swarmflow: true and the external_cli_agents allowlist (guide). Codex members run without approvals or sandbox unless configured otherwise, "matching the Claude member's bypassPermissions behavior" (team.py, validator), and the shipped config disables the permission layer (config.yaml).

   Benchmarks are also slow relative to agent tool limits. In #220 a benchmark took 5 to 13 minutes, longer than Claude Code's 2-minute foreground limit, so every benchmark became a background job. agent-core's Bash tool defaults to 300 seconds (_tool.py). Submission therefore has to be asynchronous.

   (agent-core also ships an openjiuwen.rsi package, "Recursive self-improvement for standalone Expert Harnesses" (__init__.py). It shares the acronym only. This plan does not use it.)

   2. Principle: the orchestrator proposes, InferRSI decides

   Step	Owner	Rule
   Pick lanes from search_plan and freeze the round plan	Campaign owner, optionally with an agent drafting	Plan digest posted before fan-out
   Create lane checkouts and environments	InferRSI launch	From the incumbent SHA, one per lane
   Edit code and run cheap local checks	Lane agent: SwarmFlow worker, Claude Code or Codex	Inside its own checkout only
   Submit a candidate	Lane agent through the lane's .inferrsi/round submit wrapper	Returns a ticket at once
   Measure	InferRSI serve	AB/BA against the incumbent under a device lease
   Select, compose, rerun and advance the incumbent	InferRSI close	One lane per composition step
   Accept a campaign result	The campaign's acceptance issue and held-out split	Unchanged by this issue

   Agent narrative, agent-run timings and verify() votes are recorded but never decide anything, per "Evaluator + evidence win" (AGENTS.md).

   3. Choosing orthogonal lanes

   A lane is one search_plan entry plus the files it may change. The campaign owner chooses lanes before the round, from evidence. Nobody adds a lane mid-round.

   • Start from the profile. The campaign plan already records each entry's target stage, its nonoverlapping exposed share of end-to-end time and its conditional headroom (template).
   • Pair entries whose gains should add up. Good partners target different stages on the critical path, change different files and use different mechanisms. When two lanes remove exposed time from different stages of a serial critical path, the savings add up in time. Overlap, queueing and batching break that, which is why the template's headroom field asks about them.
   • Keep same-bottleneck entries apart. Two entries that attack the same stage or the same resource usually overlap. Take the one with more headroom and leave the other for a later round, where it is measured against the new incumbent.
   • Skip lanes that cannot show a result. A lane whose conditional headroom is below two sigma of the measured noise cannot produce a measurable win, so it gets no slot.
   • Derive globs from the code. A short read-only scouting pass can map each entry to the files it would touch, for example SwarmFlow's "Understand" shape of parallel readers producing a structured map (swarmflow.md). Its output is a proposal, and the owner freezes the lanes.
   • Prefer fewer lanes to colliding ones. If no clean split exists, run two lanes instead of four.

   4. Round protocol

   The commands live in tools/parallel_round.py. The library code goes under src/inferrsi/harness/ and obeys the existing module boundary tests. A round keeps its working state in a round directory outside the InferRSI tree: the lane clones under lanes/, the queue, the measurement checkouts and the ledger. close exports the evidence to evidence/<campaign-id>/rounds/<round-id>/.

   prepare reads the frozen campaign plan and writes round-plan.json:

   • The round ID, the campaign plan path and digest, and the incumbent SHA. The SHA must equal search_state.incumbent_revision.
   • Gates, by reference to the campaign's frozen test plan: ab_ba_pairs, repetitions, the rollback line, and the validator with its hashes. A round can tighten a gate but never loosen one.
   • Protected paths: the benchmark, the validator, metric parsing, build scripts and anything else the test plan names.
   • Measurement device sets, the build command and an optional smoke command.
   • For each lane:
     • its ID and the search_plan entry it pursues, whose plane and hypothesis it inherits;
     • its writable path globs;
     • its harness and model;
     • its agent budget (turns, tokens or USD, wall clock) and submission cap.
   • The composition policy and the round deadline.

   prepare refuses the plan in any of these cases:

   • Two lanes' globs match a common path in the incumbent tree. New files are checked when they arrive.
   • A lane's globs reach a protected path.
   • A lane's entry does not name exactly one plane.
   • Two lanes pursue the same entry.
   • An entry is not planned.
   • A matching negative record exists and the plan states no changed condition (RFC).

   The plan digest is posted on the campaign's issue before launch.

   launch sets up each lane:

   • A local clone of the campaign repository at lanes/<id>, detached at the incumbent SHA, with its origin remote removed. Git hardlinks the object files when the round directory is on the same filesystem. Each lane has its own index, refs and objects inside its directory, so no two lanes share an index, and without a remote a lane has nowhere to push. Commits are optional, because submit reads the working tree.
   • A private environment, such as a virtual environment or a container. An import probe must resolve the target package inside the lane clone, because --candidate-repo changes only the working directory (quickstart.md).
   • A .inferrsi/ directory holding lane.json, the round wrapper (a standalone standard-library script, so the lane environment needs no InferRSI install), scratch space, an outbox and a status mirror, listed in the clone's .git/info/exclude. Everything a lane writes stays inside its clone.

   It writes launch.json, the manifest the drivers in section 6 take as input. With --drive claude or --drive codex it also starts one driver per lane. --drive swarmflow renders a SwarmFlow script instead (section 6).

   submit (.inferrsi/round submit) is the only way a lane hands over a candidate. It compares the working tree with the incumbent commit and copies the changed files under the lane's globs into .inferrsi/outbox/, with a manifest of paths, modes, deletions and SHA-256 hashes. Changes outside the globs are listed back to the agent and never submitted. The wrapper does not need to write to .git, so it also works under a sandbox that keeps .git read-only. It returns a ticket at once:

   {"ticket": "r03-L2-7f3a9c1e", "status": "queued", "duplicate": false}

   Submission is idempotent per round digest, lane and manifest digest: the same files always return the same ticket. As a result, SwarmFlow retries, cache-hit replays and driver restarts cannot queue a candidate twice. The submission cap counts distinct manifests. The wrapper checks scope early so the lane gets fast feedback. serve repeats every check and trusts nothing inside a lane clone: it takes globs and gates from the round plan, never from lane.json or the wrapper, so editing them gains a lane nothing.

   status returns a ticket's state (queued, measuring, measured or rejected), plus the verdict and gain once it has been measured. serve mirrors each ticket's state into the lane's .inferrsi/status/, so the command also works inside a sandbox. status --wait <seconds> blocks until the ticket changes state or the time runs out. Drivers set the limit below the harness's foreground tool timeout: two minutes by default in Claude Code, 300 seconds for the agent-core Bash tool. A lane may read it to iterate. The authoritative copy is the ledger in the round directory.

   serve is one local process that owns the measurement devices. For each ticket:

   1. It picks up new outbox entries from every lane, recomputes every file hash, writes the files into a fresh checkout of the incumbent and commits the result as the candidate. Any of these rejects the ticket: a hash mismatch, a path outside the lane's globs, a file matching another lane's globs, a protected path, a symlink leaving the checkout, or a submodule change.
   2. It builds the candidate and runs the round plan's smoke command, if one is frozen, before taking a device. A broken build therefore never occupies a device.
   3. It takes a host-wide file lock on a free device set and runs qualify_devices. It then runs run_experiment, with the incumbent checkout as baseline and the materialized candidate as candidate, followed by evaluate. If a device is busy, the ticket stays queued instead of failing.
   4. It records the RunSpec, the repository_identity of both checkouts, the verdict and the gain against the ticket.

   The queue is served round-robin across lanes, so one fast lane cannot starve the others. Lane agents never hold measurement devices.

   Every k tickets, with k frozen in the plan, serve also measures the incumbent against itself. If that A/A check fails the noise gate, or device qualification drifts, the queue pauses, in line with the RFC's rule to stop a round when the stack or device state drifts (RFC).

   close runs once every lane has stopped or the round deadline has passed. Section 5 describes it.

   5. Closing a round: select, compose, rerun, record

   Orthogonality has two parts here:

   • Structural orthogonality means disjoint write scopes. It is declared in the plan and enforced at submission, so combining lanes never needs a textual merge.
   • Performance orthogonality means gains that still add up when combined. It can only be measured, and close measures it one lane at a time.

   close runs these steps:

   1. Select. Each lane contributes at most one candidate: the ticket with the largest gain among those evaluate passed. A gain below two sigma of the measured noise is "not measured" under the campaign test plan and counts as no gain. A lane with no passing ticket is a negative result.

   2. Order. Eligible lanes are sorted by measured gain, largest first, with the lane ID breaking ties.

   3. Compose stepwise. Let I0 be the incumbent and C0 = I0. For each lane k in order:

      • close writes the lane's files onto a fresh checkout of C(k-1), commits the result as one step, and measures it against C(k-1) with run_experiment and evaluate.
      • A pass makes it C(k). A failure skips the lane and keeps C(k-1).

      Step 1 reuses the lane's own measurement against I0, so composing m lanes costs m - 1 measurements. Each step adds one lane, so each step changes one plane, as AGENTS.md requires.

   4. Rerun. The final head runs once more against I0 from fresh checkouts.

      • If it passes, it becomes the new incumbent. This is harness improvement's "Composition must independently qualify" rule (harness-improvement.md). It also corrects for having picked the best of several noisy measurements.
      • If it fails, close reruns the best single lane against I0 once. If that also fails, I0 stays.

      Harness improvement keeps H0 after a failed rerun without trying another candidate. This plan allows exactly one fallback because every lane already passed on its own. A plan may freeze the stricter rule instead.

   5. Record. The round's results go into the campaign record:

      • The new incumbent is a chain of commits, one per accepted lane, each naming its ticket, lane, plane and evidence path.
      • round-ledger.json covers every lane: its entry, plane, harness, model and agent cost; every ticket with its verdict and gain; the selected ticket; the composition outcome; and the interaction ratio. Negative results stay in the ledger.
      • The campaign plan gets append-only search_plan_revisions rows. A composed lane becomes kept, and a lane with no passing ticket becomes rejected. A lane that passed alone but lost its gain in composition goes back to planned. For that lane, the new incumbent is the changed condition the RFC asks for, so a later round may retry it.
      • search_state.incumbent_revision and incumbent_evidence move to the new head and the ledger.
      • Charts use the branch layout for a round's lanes, which all start from one incumbent, and the chain layout for composition steps and for rounds over time (campaign-summary-chart.md).

   For each composition step, the interaction ratio is the step's measured improvement over C(k-1) divided by the lane's standalone improvement over I0, both in the metric's own unit:

   • Near 1, the lane was orthogonal in performance as well as in files.
   • Well below 1, it chased the same bottleneck as an earlier lane.
   • Above 1, the two lanes reinforce each other.

   The next round's prepare starts from the new incumbent. Lane clones are deleted once the evidence is exported.

   6. Lane drivers

   Every driver reads launch.json and runs one agent per lane inside that lane's clone. All drivers give the agent the same lane prompt, which states:

   • the entry's hypothesis;
   • the lane's globs and the protected paths;
   • the wrapper commands and the budget;
   • that the agent's own timings decide nothing.

   Each lane names its harness and model, so a round may mix them. The speed comparison in section 7 keeps them fixed.

   submit refuses candidates after the lane's deadline, and the deadline holds across harness retries. Each driver works without changes to the harness itself.

   Claude Code. The driver runs one claude -p process per lane, with the lane clone as its working directory. It uses the flags and environment of the ClaudeCodeClient in #248:

   • stream JSON output, a full model ID and effort;
   • --max-turns and --max-budget-usd;
   • --permission-mode dontAsk with a frozen --allowedTools list that includes the lane wrapper;
   • --strict-mcp-config.

   Each lane gets its own CLAUDE_CONFIG_DIR. The driver does not pass --worktree, which would create a second checkout under <repo>/.claude/worktrees/ that InferRSI does not track (CLI reference).

   Codex. The driver runs one codex exec --json -C <lane clone> per lane. It reuses the argv and isolated configuration of the existing CodexClient (argv, client_config()), in the long-horizon mode from #248.

   Both CLIs. The wrapper only reads and writes files inside the lane clone. It needs no network access, no socket and no write access to .git. Paid lanes run under the runner-enforced isolation of #248: only the lane clone and its environment are writable, and no measurement device is visible.

   jiuwenswarm SwarmFlow. launch --drive swarmflow renders a round script from the template in docs/examples/parallel-round/. The Leader of a team session runs it through script_path (swarmflow.md). The Process CLI does not accept Team or Workflow roots (Process_CLI_Machine.md), so a SwarmFlow round starts from an interactive Team-mode session, such as the TUI (guide). The template follows SwarmFlow's rules:

   • No file access in the script. The script cannot touch the filesystem or the network (swarmflow.md), so workers call the wrapper with their shell tool. The lane manifest arrives through args, a string the script parses with json.loads (swarmflow.md).
   • No working-directory option. agent() has none, so each worker's prompt names its lane clone by absolute path. M2 checks whether native workers' file tools reach paths outside the team workspace. If they do not, launch puts the round directory inside it. The template never sets isolation='worktree', because those worktrees start from the default branch and a replay does not recreate them.
   • Literal budget. META must be a pure literal (swarmflow.md), so launch writes the frozen token limit into workflow_token_limit as a literal. Per-lane wall-clock limits use the timeout option.
   • Third-party workers. Claude Code or Codex workers set agent_type. That type must be declared in external_cli_agents, and enable_swarmflow must be on (guide).
   • Concurrency. Up to min(16, CPU cores - 2) workers run at once (swarmflow.md). Further lanes queue.

   from swarmflow import agent, map_parallel, phase, compact
   import json
   
   META = {
       "name": "inferrsi-round",
       "description": "InferRSI parallel round: one worker per lane; InferRSI measures and decides",
       "phases": [{"title": "Lanes", "detail": "edit, then submit through the lane wrapper"}],
       "workflow_token_limit": 2000000,  # rendered from the frozen round plan
   }
   
   LANE_REPORT = {
       "type": "object",
       "properties": {
           "lane": {"type": "string"},
           "tickets": {"type": "array", "items": {"type": "string"}},
           "notes": {"type": "string"},
       },
       "required": ["lane", "tickets"],
   }
   
   
   async def run(args):
       manifest = json.loads(args)  # launch.json
       phase("Lanes")
   
       async def lane(item):
           options = {"timeout": item["timeout_s"]}
           if item.get("agent_type"):  # "claude" or "codex"
               options["agent_type"] = item["agent_type"]
           # The prompt names the lane clone, its globs, the wrapper and the budget.
           return await agent(item["prompt"], schema=LANE_REPORT, label=item["id"],
                              phase="Lanes", options=options)
   
       reports = await map_parallel(manifest["lanes"], lane)
       # Reports only. The round ledger is the record, and nothing here is a verdict.
       return {"round": manifest["round_id"], "lanes": compact(reports)}

   SwarmFlow's runtime behavior meets the round contract as follows:

   • Retries. A timed-out worker is retried twice more by default. The retry works in the same lane clone and finds the earlier attempt's files. The wrapper returns the existing ticket for any manifest already submitted, and the cap counts distinct manifests, so a retry cannot add submissions.
   • Replays. A cache-hit replay skips the worker. Nothing is lost, because lane state lives in the lane clone and the round ledger, not in the worker's return value.
   • Reviews. verify() may run as an optional review before submission, for example a readability check. Its votes are recorded and never gate a candidate.
   • Isolation. The shipped configuration disables the permission layer, and Codex members bypass their own sandbox unless configured otherwise (section 1). A paid SwarmFlow round therefore runs the whole team in a container whose only writable mounts are the lane clones. serve runs outside the container and reads each outbox from the shared mount. Inside the container, nothing at the OS level stops one worker from writing another lane's clone. serve rejects files outside the submitting lane's globs, which limits the damage but does not prevent interference. Rounds that need lanes isolated from each other use the Claude Code or Codex drivers.

   7. Measuring the speedup

   The question is whether parallel rounds reach a given gain sooner than serial rounds at the same agent budget, and what they cost in device time. The pilot answers it for one task, one harness and one model.

   • Arms.
     • P runs rounds of N lanes, as described above.
     • S runs the same lanes one per round, in the same frozen order and with the same per-lane budget. Every S round still runs close, so S is today's serial campaign with the same tooling. A real serial campaign could also reorder its directions after each result; this comparison does not capture that.
     • Both arms start from the same incumbent and use the same harness, model, prompts, gates and device sets. Both stop at the target or at the same total agent budget.
   • Task. One task with at least three search_plan entries that the owner expects to be orthogonal and whose files are disjoint. A reference gain G* is frozen before the pilot, from a reference solution as in #248 or from earlier campaign evidence.
   • Primary measure. Wall-clock time from the first launch until the incumbent first reaches a frozen fraction f of G*, for example f = 0.8, with a passing rerun. A run that never reaches it is censored at budget exhaustion, and the report says so.
   • Secondary measures.
     • The final incumbent gain when the budget runs out.
     • Measurement device-hours, agent tokens and USD.
     • Queue wait per ticket and device utilization.
     • Interaction ratios, and lanes that passed alone but lost their gain in composition.
     • Rejected submissions by cause: scope, protected path, build or gate.
   • Repetitions. Agent runs vary, so each arm runs at least three independent times with fresh sessions, in interleaved order. The report gives every run's numbers with medians and ranges. Three runs per arm can show a large, consistent difference or its absence. They cannot estimate a small one.
   • Decision rule, frozen in the plan.
     • "Faster": P reaches f·G* before S in every repetition pair, the median wall-clock ratio is below a frozen threshold, and P's final gain is not lower.
     • "Slower": the mirror case.
     • Anything else: "not distinguishable".

   As a rough guide, suppose each lane spends A minutes of agent work and submits s candidates that take M minutes each to measure. On one device set:

   • N serial rounds take about N(A + sM).
   • One parallel round takes about max(A + sM, N·s·M).

   The speedup approaches N when agent work dominates and approaches 1 when measurement does. Each extra device set divides the measurement term. Composition costs the same in both arms: a parallel round measures N - 1 steps and one rerun, and N serial rounds measure N reruns. The report includes queue wait so readers can see whether devices were the limit.

   The speed pilot is separate from #248, which asks whether InferRSI helps an agent at all. It reuses that issue's adapters, isolation and claim-reliability measures for each lane.

   8. Milestones

   M0 to M2 need no paid calls and can merge before anyone decides on M3.

   M0, round tooling.

   • Deliverables:
     • tools/parallel_round.py with prepare, launch, serve, status and close.
     • The standalone lane wrapper with submit and status.
     • The round plan, ticket and ledger schemas.
     • Two campaign plan template fields, budget.lanes_per_round and search_state.last_round_ledger, with layout tests, coordinated with #30.
     • An operation guide entry.
   • Tests run on the CPU backend against a synthetic multi-lane repository:
     • Each lane owns one knob file. A benchmark script turns the knobs into metrics.latency_ms, like the existing synthetic fixture.
     • Devices come from a fixture probe, as in test_module_boundaries.py.
   • Scripted lanes cover these cases:
     • an honest gain;
     • a write outside the lane's globs;
     • an edit to the benchmark;
     • a resubmission of the same files;
     • a regression;
     • a pair of lanes whose gains overlap, so the second fails its composition step;
     • a lane past its submission cap;
     • two lanes measuring at once, which must never share a device set.
   • A test also confirms that close advances search_state and appends search_plan_revisions rows.

   M1, Claude Code and Codex drivers.

   • Tests use stub claude and codex executables that perform the scripted lanes.
   • They check argv, working directory and environment, and that --worktree is never passed.
   • They confirm that a lane cannot write outside its clone under the isolation from #248.

   M2, SwarmFlow template.

   • Deliverables: the template and its guide.
   • A dependency-free test renders a script and checks the loader rules that matter here: a literal META, and no time, random or uuid calls (loader.py).
   • An optional job pins agent-core at b2a0a248 and runs the script through run_workflow (runner.py) with MockBackend fixtures keyed by lane label. The fixtures are callables, so each one can perform a scripted lane (mock.py).

   M3, speed pilot. The pilot from section 7. Its plan digest is posted here first, and every run is retained. Paid calls and device time need separate authorization.

   9. Risks and mitigations

   Risk	Mitigation
   Lanes look orthogonal but share a bottleneck, such as memory bandwidth, a cache or the scheduler	Every composition step is measured and its interaction ratio reported. A lane that loses its gain returns to planned
   A lane games the measurement through the benchmark, validator, metric parsing or environment	Those paths are protected. serve builds from the incumbent plus the lane's files under its globs, and takes the benchmark command and environment from the plan
   Picking the best of several noisy tickets inflates the gain	Submission caps, A/A checks and a fresh rerun against I0 before any incumbent update. The report shows both the ticket gain and the rerun gain
   Lane builds disturb measurements on the same host	Lanes never see measurement devices. serve pins its CPU set where the plan says so, and A/A checks pause the queue on drift
   Lanes cannot profile or test on accelerators	The plan grants separate exploration devices, an open decision in the issue body, or lanes rely on serve results
   Harness retries, replays or restarts submit twice	Tickets are idempotent per manifest, and the cap counts distinct manifests
   Benchmarks outlast tool timeouts	Submission is asynchronous, and status --wait stays below each harness's foreground limit
   Greedy composition misses a better combination	Accepted as a simplification. The ledger keeps every lane, and a later round can retry any planned lane
   An unsandboxed harness damages the host or reads secrets	Paid rounds run in a container where only the lane clones are writable and no credential is mounted beyond the model API key. serve treats every outbox as untrusted data
   serve runs candidate code	Inherent to measuring code. Each candidate runs the way a campaign runs any candidate: in a fresh measurement checkout, with the plan's environment
   N lanes multiply spend	Per-lane caps, the round deadline, SwarmFlow's workflow_token_limit and a cost line per lane in the ledger
   A round result is mistaken for acceptance	The new incumbent is development evidence. Acceptance stays with the campaign's acceptance issue and its held-out split

   10. What a pilot result will and will not show

   A positive pilot shows that, for one task, harness and model, parallel rounds reached a frozen fraction of a known gain sooner than serial rounds at the same agent budget, and what that cost in device-hours. It does not show that the lanes' optimizations generalize, that one harness beats another, or that the final engine passes acceptance. A null result still guides the next step: if queue wait dominates, add measurement device sets before adding lanes.
3. Dong1017 commented on Oct 29, 2026

   

   Dong1017

   on Oct 29, 2026

   Collaborator

   More actions

   Proposal review of the full plan, against the Issue body updated 2026-10-29T07:50:08Z. Assessed main: a8215b3ec314b8bc5cee152cdebbf4bb76ba4188; plan source pins: InferRSI 509531c681479fbdd7ddd0e8b0032239e64a44c1, agent-core b2a0a248c554753ba31727415fe17a03eed60b46, jiuwenswarm 053c82b07056cf503e2b0229e1cbbb45a9c2bf7f. Design/source inspection only; no model, device or workflow execution.

   Central measurement ownership, asynchronous submissions, stepwise composition and retaining negative results are sound directions. Please resolve these two protocol gaps:

   P2 — Manifest idempotence does not make worker retries budget-idempotent (§4, §6). The pinned runtime allows two extra attempts. If attempt 1 submits bytes D1 and times out, attempt 2 can edit the same clone into D2 and obtain a different ticket while under the distinct-manifest cap. Thus “a retry cannot add submissions” is false; retries can also consume additional agent time and tokens without submitting anything. Retain content-based deduplication, but freeze a cumulative lane budget and absolute lane deadline across attempts/restarts, with authoritative submission counts and expiry checks outside the writable lane clone. Record all attempt costs. Add an offline timeout-after-submit → changed-bytes retry case, plus a restart after budget exhaustion.

   P2 — Extend execution isolation to the candidate measurement path (§4, §9). The plan puts lane agents in a container but leaves serve outside it, then builds and executes lane-supplied code in fresh measurement checkouts. A fresh checkout and source hashes establish provenance; they do not stop candidate build/import code from writing the coordinator's ledger, other checkouts or trusted grading files. The current executor supplies a working directory and process environment, not filesystem confinement. Specify the execution boundary for build/smoke/benchmark/validation: candidate-private writable scratch and outputs, protected grading inputs, and no write access to coordinator state or other candidates. Keep plan validation, lease ownership and verdict persistence in the trusted coordinator. An offline candidate that attempts an out-of-checkout write should demonstrate that boundary before a paid-host pilot.

   For §7, P starts all lanes from I0 while S advances the incumbent between lanes. This is a useful comparison of the two campaign policies, but it also changes the code/history later agents see; label the result as a workflow-policy comparison rather than isolated concurrency speedup. No additional paid arm is needed for that narrower claim.

   Next: @david6666666 and a named accountable/acceptance owner should settle these contracts and the adapter responsibility shared with #248 before authorizing implementation. Offline M0/M1 and the separately authorized pilot remain distinct; no incumbent advance, campaign acceptance or resource authorization results from this review.
4. david6666666

   added

   help:wantedNeeds an owner or contributorNeeds an owner or contributor

   on Oct 29, 2026
5. NumberWan commented on Oct 30, 2026

   

   NumberWan

   on Oct 30, 2026

   More actions

   I would like to take this task. Please assign @NumberWan.

   I am asking for the offline slices only: M0 (round tooling, schemas, synthetic fixture), M1 (Claude Code and Codex drivers against stub executables), and M2 (SwarmFlow template and its loader checks). I am not asking to own M3. Paid model calls, accelerator time, and the speed pilot stay with the goal-acceptance owner as a separate authorization. This comment does not request a budget, and it does not start implementation.

   Static reading of main 6e6ee245 (2026-10-30) and of the plan pinned at 509531c6. I did not run a model, a device, or a workflow.

   P2 — attempt budget, not manifest identity

   Agreed that content-addressed tickets do not make SwarmFlow retries budget-idempotent. A second attempt can edit the same clone into new bytes, take a new ticket under the distinct-manifest cap, and a timed-out attempt can spend tokens without submitting.

   tools/skill_eval_budget.py already has the transition algebra this needs. The charge unit there is an attempt binding, not a content digest:

   • Binding is (plan_digest, run_id, worst_case). Its comment requires run_id to include the workflow attempt, not only the run number.
   • reserve is idempotent only for the exact same binding. The same plan digest with a different run_id is a BudgetError, so a retry is not a free extra submission.
   • advance(..., "claimed") is deliberately not idempotent: a second claimant must not execute.
   • Ledger.charged sums worst_case over every entry, and no phase frees capacity, so a timed-out attempt still occupies the lane budget.
   • AtomicStore / claim_once keeps that count outside the writable tree, behind a compare-and-set token.

   Manifest idempotence stays, and only deduplicates measurement. The lane budget is a sequence of attempt bindings.

   Two adaptations, without changing the transitions:

   • PR_LIMIT_MICROUSD is a module constant. Per the repository rule that thresholds stay out of the tree, the cap becomes a field on the ledger, supplied by the frozen round plan. No numeric default in code.
   • PRKey becomes (round_digest, lane_id).

   Offline cases I would add in M0, matching the review:

   1. Timeout after submit, then a retry that changes bytes. The second attempt is a new binding against the same cumulative lane budget, and it is refused once that budget is exhausted.
   2. Restart after budget exhaustion. claim_once does not authorize another attempt. Every attempt's reserved cost is in the ledger, including attempts that submitted nothing.

   P2 — confine the measurement path through the existing executor

   Agreed that a fresh checkout and source hashes establish provenance and do not stop candidate build or import code from writing the coordinator ledger, another checkout, or trusted grading files. execute_command in src/inferrsi/infra/executors/local.py sets a working directory and an environment. It does not confine the filesystem.

   I would not add a second runner. run_experiment already takes execute with the same signature execute_trial uses, and tests/test_module_boundaries.py already injects that callable. serve would pass a confinement wrapper of that signature. docs/architecture.md already requires the offline harness lane to reuse run_trial and the local executor rather than create another execution backend. Plan validation, lease ownership, and verdict persistence stay in the trusted coordinator.

   The writable set cannot be "the candidate checkout, and nothing else." The executor rewrites {result} to trial_dir/result.json and sets INFERRSI_RESULT_PATH, so the candidate command has to be able to write that trial directory. The boundary I would freeze:

   • Writable: scratch inside the candidate checkout, and that trial's trial_dir.
   • Readable: the benchmark, the validator, metric parsing, and the other protected paths named by the frozen plan.
   • Not visible: the coordinator ledger, other lane clones, other trials' directories, and credentials.

   Offline case before any paid-host pilot: a candidate command that writes outside the checkout and its own trial_dir is rejected, and the ledger is unchanged.

   Where the device lease lives

   Section 4 places the lease in src/inferrsi/harness/ next to the queue. docs/architecture.md gives device qualification and process ownership to Infra, and gives the harness AB/BA order, deadlines, evidence, and adjudication. I would put the lease under src/inferrsi/infra/, beside processes.py, and leave the queue and selection in the harness.

   qualify_devices is CUDA-specific (src/inferrsi/infra/platforms/cuda.py). The lease key should be the platform-neutral identity from inspect_device, so a round on another platform can take a lease too. Today that CUDA check refuses a GPU that already has a compute process, so two lanes measuring at once fail rather than silently contend. The lease turns that failure into a queue, which is what section 4 already requires. Lane agents still never hold measurement devices.

   Section 7

   I accept the label. P starts every lane from I0 while S advances the incumbent between lanes, so later agents see different code. The pilot is a workflow-policy comparison, not an isolated concurrency speedup. No extra paid arm.

   A zero-model replay can bound the measurement term: resubmit the content-addressed manifests from a finished ledger and measure them again. It does not reproduce agent adaptation, reordering, or a lane that would have written different bytes after seeing an earlier result. I would report it as a measurement-queue bound, not as a third policy that replaces the pilot.

   If this scope is accepted and the issue is assigned, M0 is a separate PR. This comment does not advance an incumbent and does not close any completion criterion.
6. Dong1017 commented on Oct 30, 2026

   

   Dong1017

   on Oct 30, 2026 – with ChatGPT Codex Connector

   Collaborator

   More actions

   Technical follow-up to @NumberWan's offline M0–M2 proposal, checking source at 6e6ee245d9e92ee9365976f2e4e262304999f147. The cumulative-charge direction, separate trial output, and workflow-policy comparison label address the intent of the previous review. Two contract details remain before treating those P2 concerns as cleared:

   1. Retry identity and expiry. Ledger validation/reserve rejects the same plan_digest with a new run_id, even when budget remains. Changing only the cap and PRKey therefore does not admit the proposed second attempt. Specify an immutable, distinct attempt-plan binding rooted in the unchanged round/lane contract (or explicitly choose no retries), retaining all prior charges. Also retain the original review's absolute lane deadline and authoritative submission/expiry checks across restarts; reservation accounting alone enforces neither. Include a retry with budget remaining as well as the exhausted-budget case. AtomicStore is a storage contract, not itself an implemented filesystem boundary.

   2. Apply confinement before measurement too. The execute hook in run_experiment is an appropriate reuse point, but plan §4 builds and smoke-tests the candidate before calling it. Those stages can execute candidate code as well. Explicitly route build/import/smoke through the same protected boundary, with their own bounded writable outputs. Demonstrate denied coordinator writes from a build step as well as a trial command, and fail closed when confinement is unavailable. Passing a callable/cwd is not confinement evidence.

   The offline-only split is clear, and M3 remains separate. Assignment and acceptance of M0–M2 still require the responsible maintainer's decision; this reply neither assigns the task nor authorizes implementation, paid execution, device replay or incumbent advancement. Coordinate the driver contract with #248 rather than creating competing adapters.
7. NumberWan commented on Oct 30, 2026

   

   NumberWan

   on Oct 30, 2026

   More actions

   Thank you, both points land. The previous reply's reserve() claim does not hold, and the confinement answer stopped one stage too early. Concrete fixes, checked against main c97f25e2 and the plan pinned at 509531c6. Still no code, no PR, no branch.

   Retry identity: an attempt-scoped plan_digest, zero changes to reserve/advance/Ledger

   Confirmed the collision: reserve() rejects any entry whose plan_digest matches an existing one, regardless of run_id:

   if (entry.binding.plan_digest == binding.plan_digest
           or entry.binding.run_id == binding.run_id):
       raise BudgetError("Plan or run already bound differently")

   Treating plan_digest as the round plan's digest, as the previous reply did, means attempt 2 collides with attempt 1 under that same check, regardless of budget. The fix is not to change this function; it is to stop overloading plan_digest with the round's identity.

   • plan_digest = SHA256(round_digest, lane_id, attempt_ordinal). Each attempt mints its own plan identity. reserve()'s existing "one entry per plan_digest" rule then means exactly "one reservation per attempt," with no change to the transition algebra — this is the "immutable, distinct attempt-plan binding rooted in the unchanged round/lane contract" from the review.
   • run_id carries the workflow run id and the attempt ordinal together, per the existing comment requirement.
   • worst_case is the lane's frozen per-attempt cap from the round plan, not a shared value across attempts.
   • PRKey becomes LaneKey(round_digest, lane_id); one ledger per lane, shared across its attempts.
   • PR_LIMIT_MICROUSD becomes a cap field on that ledger, sourced from the lane's frozen budget in the round plan, not a module constant.

   Ledger.charged already sums worst_case over every entry regardless of phase, so retaining all prior charges, including attempts that submitted nothing, needs no new mechanism; it already works that way.

   This still leaves time separate from money. The ledger has no notion of an absolute deadline. The round plan gets a frozen deadline_utc per lane, and serve, as the trusted coordinator, checks now > deadline_utc before calling reserve() for any new attempt, independent of remaining budget. That check reads the frozen round-plan.json on disk, not an in-memory timer a restarted lane process could reset.

   Offline cases:

   1. Retry with budget remaining: attempt 1 goes reserved to cancelled on timeout; attempt 2 reserves under a new attempt-scoped plan_digest and succeeds; charged sums both worst_case values and stays at or under cap.
   2. Exhausted budget: a further attempt's reserve() fails at Ledger.__post_init__ before any claim is possible.
   3. Deadline past, budget remaining, after a coordinator restart: the deadline check, read from the frozen plan, refuses the attempt regardless of remaining budget.

   On AtomicStore: agreed that it is a storage contract. claim_once's compare-and-set gives one property only, that a second claimant cannot execute against the same stored ledger state. It says nothing about what a running candidate can write to the filesystem. That is a separate mechanism, addressed below, not a consequence of the storage contract.

   Confinement: one primitive, applied before the build and the smoke command too

   Agreed execute is the right reuse point for the trial phase, and that it is not itself confinement evidence. The gap is earlier: plan section 4 builds and smoke-tests the candidate before any call into run_experiment, and both stages execute candidate code that execute's injection point never sees.

   One primitive, called at both points, with the enforcement at the OS level rather than inside the callable:

   def confined_execute(
       argv: tuple[str, ...],
       cwd: Path,
       writable_paths: tuple[Path, ...],
       readable_paths: tuple[Path, ...],
       deadline: float,
   ) -> dict:
       """Deny filesystem effects outside writable_paths at the OS level;
       refuse to run rather than execute unconfined."""

   • Trial phase: this is the execute= value passed into run_experiment. writable_paths is the candidate checkout's scratch plus that trial's trial_dir, which has to stay writable because the executor rewrites {result} into it and sets INFERRSI_RESULT_PATH there.
   • Build and smoke phase: serve calls the same primitive directly, before any run_experiment call exists. writable_paths is the candidate checkout plus its own build-output directory. No trial_dir exists yet at this point.
   • Both calls share the same readable set (benchmark, validator, metric parsing) and the same invisible set (the coordinator ledger, other lanes' checkouts, other trials' directories, credentials).

   src/inferrsi/infra/executors/local.py's execute_command today is a plain subprocess.Popen with a cwd and an env; nothing stops a child process from writing anywhere its OS permissions allow. Confinement has to come from outside the Python call, for example a mount namespace that bind-mounts everything except writable_paths read-only, or a container whose only writable mount is writable_paths. Section 6 of the plan already asks for exactly this around the worker itself; this extends the same boundary to the build and smoke stages the coordinator runs locally.

   Fail closed: if the host cannot set up that boundary, confined_execute refuses to run the candidate at all. It does not fall back to an unconfined subprocess.

   Offline cases, in addition to the trial-write denial the plan already lists:

   1. A build step that attempts to write the coordinator's ledger path or another lane's checkout is denied through the same mechanism, and the ledger is unchanged.
   2. With the confinement mechanism unavailable in the test environment, confined_execute refuses the run and reports why, instead of executing unconfined.

   #248

   Agreed. The Claude Code and Codex lane drivers would share the argv and isolation contract with #248's adapters rather than create a second one.

   No code, no branch, and no PR accompany this comment. Assignment and acceptance of M0 through M2 stay with the responsible maintainer.
8. Dong1017 commented on Oct 30, 2026

   

   Dong1017

   on Oct 30, 2026 – with ChatGPT Codex Connector

   Collaborator

   More actions

   Follow-up to 6516547428, source checked at 37166b6f7323c76a091e423a7163ac99c4ae01ac:

   The distinct attempt identity and explicit build/smoke/trial OS boundary resolve the earlier proposal ambiguity in direction; they remain designs, not implemented or tested clearance. Two acceptance cases still need correction:

   • Timeout after execution is not reserved → cancelled. The existing transition table requires claiming before execution; a claimed attempt can only finish with trusted settlement, and a crash stays ambiguous/charged. Cancellation applies only before claim. Make the timeout-after-submit fixture start from claimed, retain that charge/ambiguity, and admit any separately authorized next attempt with a new binding and sufficient cumulative capacity—never replay or cancel the old execution claim.

   • Check the persistent deadline beyond reservation. Reserving immediately before expiry does not prevent a later claim/launch or a late changed-byte submission after expiry. Freeze the boundary semantics and recheck in the trusted coordinator at claim/launch and submission intake, and bound running work by the same absolute deadline. Add “reserved before expiry, launch after expiry” and “started before expiry, submit new manifest after expiry” cases, including coordinator restart. Reading the deadline only before reserve() does not yet close the original expiry/submission concern.

   Keep the attempt digest canonically encoded and bound to the frozen round contract; the ordinal must not let a restart reset identity or accounting. The confinement-denial tests must exercise actual OS enforcement (or fail closed when unavailable), not only a mocked callable.

   M0–M2 assignment/scope acceptance remains a maintainer decision; no implementation completion, paid/device authorization or M3 acceptance is inferred from this discussion.
9. david6666666

   assigned

   NumberWan

   on Oct 30, 2026
10. NumberWan

    added 2 commits that reference this issue on Nov 2, 2026

    fix(harness): close the parallel-round execution boundary

    ...

    9b4b08b

    fix(harness): keep parallel-round launches inside the deadline and th…

    ...

    fc9d232