[SamGu-NRX]

fleet-packet: burst14-bs-stage-selection

What changed

Until now, HSS_ORCH_STAGE_STUBS=1 was all or nothing: a local run answered every stage with its stand-in from server/orchestrator/stubs/<stage>.py, so the result never depended on the scan and no real stage code could be exercised locally without running the full real orchestration (Modal, GPU, credentials). This change adds the middle setting and makes the choice observable:

1. HSS_ORCH_REAL_STAGES (new backend/server/orchestrator/stage_selection.py) names the stages whose real implementations run even with stage stubs on, per stage. A local or dev run can now exercise one real stage at a time against stand-in upstreams — e.g. HSS_ORCH_REAL_STAGES=validate runs the real packet validation and records which code served every other stage.
2. Every stage publishes an implementation marker at run-relative key impl/<stage>.json (stored at captures/{capture_id}/runs/{run_id}/impl/<stage>.json) under the pinned contract hss-stage-impl/v1, recording implementation (real or stand-in) and the exact dotted callable that executed — read off the resolved callable itself, so a per-run selection override (the capture-placement runner's) is recorded as what it ran, not what a global setting implies. Unset, behavior is byte-for-byte today's: every non-exempt stage records stand-in.
3. The result stage's declared inputs include impl/ (graph.py STAGE_INPUTS), so the composer receives every marker of its run and downstream result consumers can name which implementation produced each input. The pinned contract is shared with obv/burst14-bs-result-names-stand-ins; it is defined once in runner.py (IMPL_SCHEMA) and copied verbatim there.

The cpu alias selects every stage whose real path is deterministic CPU work (no GPU, no model weights, no paid call): validate poses reads scale dense coverage scene objects_3d placement criteria result. objects is the one deliberate exclusion — its real path loads OWLv2/SigLIP weights — and listing it is refused unless HSS_OBJECTS_DEVICE=cpu opts into local weights.

Why

A per-run answer to "which code produced this?" needs two halves: the ability to run real stages locally at all (the selection), and a durable per-run record of what actually ran (the markers). The selection is dev-only by construction — refused together with HSS_ORCH_MODAL_REMOTE=1 (a remote run already runs everything real) and outside SMP_ENVIRONMENT=dev — so production cannot silently run a half-real orchestration. Refusals raise at runner import, which is the startup of every process that can run a stage, and name the offending variable and value.

The markers land in impl/ inside the run directory rather than in run metadata so they ride the existing artifact publication, hydration, and result-input machinery with no new transport, and survive in storage next to the artifacts they describe.

How it works

flowchart LR
    env["HSS_ORCH_REAL_STAGES\n(comma list or 'cpu')"] --> parse["stage_selection.py\nparse + refusals"]
    parse -->|"is_real(stage)"| sc["runner.stage_callable"]
    sc -->|"selected / stubs_exempt"| real["real module\nserver.<stage>"]
    sc -->|"not selected"| stub["stand-in\nserver.orchestrator.stubs.<stage>"]
    real --> marker
    stub --> marker
    marker["run_stage → impl/<stage>.json\nhss-stage-impl/v1"] --> result["result stage inputs\ninclude impl/"]

Loading

Caption: the selection only changes which callable stage_callable resolves; the marker records what actually ran, and the result stage receives every marker.

UI changes

N/A — orchestration backend only; no client-visible surface changes.

Verification

• New: backend/server/orchestrator/tests/test_stage_selection.py — 14 tests: parsing (unset/blank, whitespace, cpu alias covering exactly CPU_STAGES, alias + objects = all 12), refusals (unknown stage, HSS_ORCH_MODAL_REMOTE=1, non-dev environment, objects without local-weights device — each message names the variable), the selection cache following the environment, callable selection under stubs (selected → real, unselected → stub, stubs_exempt placement stays real, stubs-off unaffected), runner-import-time validation of a bad value, and two in-process journey runs on the committed backend/data/fixtures/packets/mini-arkit packet: with validate selected the run reaches today's terminal state through the real packet validation with mixed impl markers (validate real/server.packet.run, poses stand-in, placement real via stubs_exempt), and unset the markers record exactly today's implementations; the result stage's hydration carries the whole impl/ directory. No Modal, GPU, network, or credentials.
• pytest server/orchestrator/tests — 195 passed.
• pytest backend/tests/test_capture_real_downstream.py — 12 passed (the feat(backend): --real-downstream runs the real criteria and result through the capture front door #292 capture-real-downstream journey, which overrides stage_callable per run, is unchanged and its override is what the marker's callable field is designed to record honestly).
• pytest backend/tests/test_runtime_join_journey.py — 6 passed.
• ruff check . — clean; the new test file is ruff format-clean.
• Full make test — 4 failed, 1520 passed, 19 skipped, 14 errors in 10:43: the failure set is the branch base's documented baseline exactly — 4 server/eval/tests/test_gates.py (SD-96 gates) failures plus 14 collection errors in tests/test_capture_cold_restart.py, every name in files this branch does not touch; zero failures attributable to this change.
• Coverage limit: the objects real path with local weights is refused-by-default and not exercised end-to-end here (no weights on the dev box); the refusal itself is tested.

Risk

• A real stage fed by stand-in upstreams can meet inputs it was not built for (the stub poses stage produces a single identity camera, for example); the docs (backend/docs/real-cpu-stages.md) name this and recommend selecting upstream stages together. This is a dev-only knob and fails at run time, not silently.
• impl/ artifacts are new keys under the run directory; only the result stage's declared inputs change, and _publish callers treat unknown keys uniformly. Runs that fail a stage publish no marker for it, same as any other artifact.
• Pre-existing on alpha, untouched here: ruff format --check flags server/orchestrator/tests/test_greptile_followup.py; make lint will report it on the base branch too.

Checklist

• The scope is focused; related changes belong together
• I explained what changed and why
• N/A — no visible changes (backend orchestration only; before/after images not applicable)
• N/A — no motion or interaction changes (no video applicable)

Human author: Sam Gu (sgu07966@gmail.com)

🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact

---

Opened by the maintainer from the Obvious thread's own description, so every review bot reviews it (CodeRabbit skips PRs authored by bots). The work and its commits are by Obvious.

Co-authored-by: obvious-autobuild[bot] <262744130+obvious-autobuild[bot]@users.noreply.github.com>

[SamGu-NRX]

@codex review

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 49c06985-5213-449d-b830-1f8eefde9f36

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-23T16:44:16.318150Z	—	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 098d8fdfc7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[SamGu-NRX]

@greptileai review this at plus

[greptile-apps]

Confidence Score: 3/5

[Medium impact] Adds per-stage real implementation selection for local development.

Fix the CPU alias and dev-only environment guard before merging.

Findings

1.  CPU alias needs models ▶
2.  Production passes the dev guard ▶
3.  Remote selection skips refusal ▶
4.  Marker delivery lacks coverage ▶

Fix with agent prompt

### Issue 1
backend/server/orchestrator/stage_selection.py:46-48
The `cpu` alias selects stages that still need more than CPU-only work. On photo-only packets, real `dense` reaches model loaders that default to `cuda`, so a dev box without weights or a GPU cannot use the advertised weight-free alias. Real `reads` can also make paid calls if `VISION_EXTERNAL_CALLS=on`.

Exclude these paths from `cpu` or enforce the settings that keep them resource-free.

### Issue 2
backend/server/orchestrator/stage_selection.py:75-76
Missing or blank `SMP_ENVIRONMENT` passes this guard as `dev`, but `Settings` treats those same values as `prod`. With stage stubs and a real-stage selection enabled, normal service setup therefore permits mixed real/stand-in work in a production-configured process.

Require an explicit `dev` value and match the meaning used by `Settings`.

### Issue 3
backend/server/orchestrator/stage_selection.py:70
This refusal misses supported true values for `HSS_ORCH_MODAL_REMOTE`. For example, `true`, `yes`, and `on` enable remote work in `load_orch` but pass this check, allowing the unsupported combination with `HSS_ORCH_REAL_STAGES`. Developers using these spellings miss the promised startup error.

Use the same boolean rules as `load_orch` and test a nonnumeric true value.

### Issue 4
backend/server/orchestrator/tests/test_stage_selection.py:235-238
This assertion only filters `run.artifacts` after the run finishes. It never checks that the result stage received marker files, so it still passes if downloading `impl/` breaks. It also includes `impl/result.json`, which is written only after the result callable returns.

Check the upstream marker files inside the result callable so the test covers actual delivery.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds per-stage real implementation selection under stubs and publishes hss-stage-impl/v1 records alongside run outputs. The result stage now downloads upstream markers.

• Dev runs can choose which stages use real code.
• Each successful stage records which implementation ran.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Env["HSS_ORCH_REAL_STAGES"] --> Guard["Parse and check environment"]
    Guard --> Resolver["stage_callable"]
    Resolver --> Real["Real stage"]
    Resolver --> Stub["Stand-in stage"]
    Real --> Publish["Publish outputs and impl marker"]
    Stub --> Publish
    Publish --> Storage["Run artifacts"]
    Storage --> Download["Download upstream impl markers"]
    Download --> Result["Result callable"]
    Result --> ResultMarker["Publish result outputs and marker"]

Loading

_{Reviews (1) · Last reviewed commit: "docs: record the verified full-suite res..." · Reviewed by Greptile}

[SamGu-NRX]

@codex review

[chatgpt-codex-connector]

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

An unknown error occurred

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[SamGu-NRX]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 5546656c5c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[codemetrics-ai]

Note

Review skipped

Codemetrics is installed on this repository, but it hasn't been added to your team yet.

To enable AI code reviews for this repo, go to your dashboard and add BaseScanning to your team's repositories.