Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions PR.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ Until now, `HSS_ORCH_STAGE_STUBS=1` was all or nothing: a local run answered eve
2. **Every stage publishes an implementation marker** at run-relative key `impl/<stage>.json` (stored at `captures/{capture_id}/runs/{run_id}/impl/<stage>.json`) under the pinned contract `hss-stage-impl/v1`, recording `implementation` (`real` or `stand-in`) and the exact dotted `callable` that executed — read off the resolved callable itself, so a per-run selection override (the capture-placement runner's) is recorded as what it ran, not what a global setting implies. Unset, behavior is byte-for-byte today's: every non-exempt stage records `stand-in`.
3. **The result stage's declared inputs include `impl/`** (`graph.py` `STAGE_INPUTS`), so the composer receives every marker of its run and downstream result consumers can name which implementation produced each input. The pinned contract is shared with `obv/burst14-bs-result-names-stand-ins`; it is defined once in `runner.py` (`IMPL_SCHEMA`) and copied verbatim there.

The `cpu` alias selects every stage whose real path is deterministic CPU work (no GPU, no model weights, no paid call): `validate poses reads scale dense coverage scene objects_3d placement criteria result`. `objects` is the one deliberate exclusion — its real path loads OWLv2/SigLIP weights — and listing it is refused unless `HSS_OBJECTS_DEVICE=cpu` opts into local weights.
The `cpu` alias selects every stage whose real path is deterministic CPU work (no GPU, no model weights, no paid call): `validate poses reads scale coverage scene objects_3d placement criteria result`. `objects` is excluded because its real path loads OWLv2/SigLIP weights, and listing it is refused unless `HSS_OBJECTS_DEVICE=cpu` opts into local weights. `dense` is excluded because photo-only packets fall back to π³/MoGe/DA3 weights; it can still be listed by name. The alias is refused while `VISION_EXTERNAL_CALLS` is on, because `reads` and `criteria` would then call a paid vision model.

## Why

A per-run answer to "which code produced this?" needs two halves: the ability to run real stages locally at all (the selection), and a durable per-run record of what actually ran (the markers). The selection is dev-only by construction — refused together with `HSS_ORCH_MODAL_REMOTE=1` (a remote run already runs everything real) and outside `SMP_ENVIRONMENT=dev` — so production cannot silently run a half-real orchestration. Refusals raise at runner import, which is the startup of every process that can run a stage, and name the offending variable and value.
A per-run answer to "which code produced this?" needs two halves: the ability to run real stages locally at all (the selection), and a durable per-run record of what actually ran (the markers). The selection is dev-only by construction — refused together with a true `HSS_ORCH_MODAL_REMOTE` in any spelling `load_orch` accepts (a remote run already runs everything real) and unless `SMP_ENVIRONMENT` is exactly `dev`; unset or blank counts as prod, as it does in `Settings` — so production cannot silently run a half-real orchestration. Refusals raise at runner import, which is the startup of every process that can run a stage, and name the offending variable and value.

The markers land in `impl/` inside the run directory rather than in run metadata so they ride the existing artifact publication, hydration, and result-input machinery with no new transport, and survive in storage next to the artifacts they describe.

Expand Down
37 changes: 26 additions & 11 deletions backend/docs/real-cpu-stages.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ scan. `HSS_ORCH_REAL_STAGES` adds the middle setting: it names the stages whose
exercise one real stage at a time against stand-in upstreams.

```sh
export HSS_ORCH_STAGE_STUBS=1 HSS_ORCH_REAL_STAGES=validate
export SMP_ENVIRONMENT=dev HSS_ORCH_STAGE_STUBS=1 HSS_ORCH_REAL_STAGES=validate
# or every stage whose real path needs no GPU, weights or paid call:
export HSS_ORCH_REAL_STAGES=cpu
```
Expand All @@ -16,18 +16,28 @@ The value is a comma-separated list of stage names, or the alias `cpu`.
Whitespace and a trailing comma are tolerated; stages run real on top of
whatever their upstreams produced, which under stubs are stand-in artifacts.

## The `cpu` alias, and why `objects` is excluded
## The `cpu` alias, and why `objects` and `dense` are excluded

`cpu` selects every stage whose real path is deterministic CPU work with no
GPU, no model weights and no paid call:

`validate`, `poses`, `reads`, `scale`, `dense`, `coverage`, `scene`,
`objects_3d`, `placement`, `criteria`, `result`
`validate`, `poses`, `reads`, `scale`, `coverage`, `scene`, `objects_3d`,
`placement`, `criteria`, `result`

`objects` is the one stage the alias must not pull in: its real path loads the
OWLv2 and SigLIP weights (`server/objects/run.py`), which a CPU dev box does
not carry. Listing `objects` explicitly is refused unless
`HSS_OBJECTS_DEVICE=cpu` is also set to run those weights locally.
Two stages stay out:

- `objects` loads the OWLv2 and SigLIP weights (`server/objects/run.py`), which
a CPU dev box does not carry. Listing `objects` explicitly is refused unless
`HSS_OBJECTS_DEVICE=cpu` is also set to run those weights locally.
- `dense` falls back to π³/MoGe/DA3 weights on a CUDA device whenever the
packet has no LiDAR and no points were injected (`_acquire_cloud` in
`server/dense/run.py`), and fails with `input_missing:weights` without
`HSS_DENSE_WEIGHTS`. List it by name (`cpu,dense`) when the packet or the
machine can serve it.

`reads` and `criteria` stay in the alias because their real paths make no
vision call while `VISION_EXTERNAL_CALLS` is off, which is the default. The
alias is refused when that variable is on.

## Refusals

Expand All @@ -36,9 +46,14 @@ when the orchestrator runner first loads, and refused with a message that names
the variable and the bad value when:

- a listed stage name is unknown;
- `HSS_ORCH_MODAL_REMOTE=1` — a remote run already runs every stage for real;
- `SMP_ENVIRONMENT` is anything other than `dev`;
- `objects` is listed without `HSS_OBJECTS_DEVICE=cpu`.
- `HSS_ORCH_MODAL_REMOTE` is true in any spelling `load_orch` accepts (`1`,
`true`, `yes`, `on`, any case): a remote run already runs every stage for
real;
- `SMP_ENVIRONMENT` is anything other than exactly `dev`, including unset or
blank. `Settings` reads those values as `prod`, so the selection does too;
- `objects` is listed without `HSS_OBJECTS_DEVICE=cpu`;
- the `cpu` alias is used while `VISION_EXTERNAL_CALLS` is true (`on`, `1`,
`true`, `yes`).

A stage selected this way can still fail at run time: a real stage fed by
stand-in artifacts can meet inputs it was not built for (the stub poses stage
Expand Down
62 changes: 44 additions & 18 deletions backend/server/orchestrator/stage_selection.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,43 +9,53 @@

Grammar: a comma-separated list of stage names, or the alias ``cpu``, which
selects every stage whose real path needs no GPU, no model weights and no paid
call. ``objects`` is the one stage ``cpu`` must not pull in: its real path loads
OWLv2/SigLIP weights (``server/objects/run.py``), so listing it needs
``HSS_OBJECTS_DEVICE=cpu`` as well.
call. Two stages stay out of ``cpu``:

The variable is dev-only by construction: it is refused together with
``HSS_ORCH_MODAL_REMOTE=1`` (a remote run already runs everything real) and
outside ``SMP_ENVIRONMENT=dev``. Refusals raise as early as the process can see
them — :mod:`server.orchestrator.runner` validates at import, which is the
startup of every process that can run a stage.
- ``objects`` loads OWLv2/SigLIP weights (``server/objects/run.py``), so
listing it needs ``HSS_OBJECTS_DEVICE=cpu`` as well.
- ``dense`` falls back to π³/MoGe/DA3 weights on a CUDA device whenever the
packet carries no LiDAR and no points were injected
(``server/dense/run.py`` ``_acquire_cloud``). It can still be listed by name.

``reads`` and ``criteria`` call a paid vision model when
``VISION_EXTERNAL_CALLS`` is on, so ``cpu`` is refused in that case.

The variable is dev-only by construction: it is refused together with a true
``HSS_ORCH_MODAL_REMOTE`` (a remote run already runs everything real) and
unless ``SMP_ENVIRONMENT`` is exactly ``dev``. Refusals raise as early as the
process can see them — :mod:`server.orchestrator.runner` validates at import,
which is the startup of every process that can run a stage.

This module reads the environment itself and does not extend
``server/orchestrator/config.py``: that file is another packet's file this
wave.
wave. It does reuse that file's boolean parser, so a value means the same thing
in both places.
"""

from __future__ import annotations

import os
from collections.abc import Mapping

from server.orchestrator.config import _flag
from server.orchestrator.graph import STAGE_NAMES

ENV_VAR = "HSS_ORCH_REAL_STAGES"
CPU_ALIAS = "cpu"
REMOTE_VAR = "HSS_ORCH_MODAL_REMOTE"
ENVIRONMENT_VAR = "SMP_ENVIRONMENT"
OBJECTS_DEVICE_VAR = "HSS_OBJECTS_DEVICE"
VISION_VAR = "VISION_EXTERNAL_CALLS"

#: Stages whose real path is deterministic CPU work: no GPU, no local weights,
#: no paid or networked model call. ``objects`` is the deliberate omission —
#: its real path loads OWLv2/SigLIP weights, so it never rides the alias.
#: no paid or networked model call (the last holds while VISION_EXTERNAL_CALLS
#: is off, which the parser enforces for the alias). ``objects`` and ``dense``
#: are the deliberate omissions; see the module docstring.
CPU_STAGES: tuple[str, ...] = (
"validate",
"poses",
"reads",
"scale",
"dense",
"coverage",
"scene",
"objects_3d",
Expand All @@ -56,7 +66,7 @@

# Cache key: the env vars the parse reads. A cache entry survives only while
# they are unchanged, so tests can monkeypatch the environment freely.
_CACHE_KEYS = (ENV_VAR, REMOTE_VAR, ENVIRONMENT_VAR, OBJECTS_DEVICE_VAR)
_CACHE_KEYS = (ENV_VAR, REMOTE_VAR, ENVIRONMENT_VAR, OBJECTS_DEVICE_VAR, VISION_VAR)
_cache: tuple[tuple[tuple[str, str], ...], frozenset[str]] | None = None


Expand All @@ -67,16 +77,23 @@ def parse_real_stages(environ: Mapping[str, str] | None = None) -> frozenset[str
raw = (env.get(ENV_VAR) or "").strip()
if not raw:
return frozenset()
if (env.get(REMOTE_VAR) or "").strip() == "1":
remote = env.get(REMOTE_VAR, "0")
# load_orch parses this variable with _flag, so every spelling it treats as
# true (1, true, yes, on, any case) is refused here too.
if _flag(remote):
raise ValueError(
f"{ENV_VAR}={raw!r} selects real stages for a local run, but {REMOTE_VAR}=1 already runs "
f"{ENV_VAR}={raw!r} selects real stages for a local run, but {REMOTE_VAR}={remote!r} already runs "
"every stage for real; set one or the other, never both."
)
environment = (env.get(ENVIRONMENT_VAR) or "dev").strip() or "dev"
# Fail closed, matching Settings.SMP_ENVIRONMENT (server/common/config.py):
# a missing, blank or unknown value is prod there, so only an exact "dev" is
# dev here. An unconfigured production process therefore refuses a selection.
environment = env.get(ENVIRONMENT_VAR)
if environment != "dev":
shown = "unset" if environment is None else f"{environment!r}"
raise ValueError(
f"{ENV_VAR}={raw!r} is a dev-only selection; {ENVIRONMENT_VAR}={environment!r} runs the real "
"orchestration, where the variable has nothing to select. Unset one of the two."
f"{ENV_VAR}={raw!r} is a dev-only selection, but {ENVIRONMENT_VAR} is {shown}, which Settings "
f"reads as prod. Set {ENVIRONMENT_VAR}=dev exactly, or unset {ENV_VAR}."
)
known = frozenset(STAGE_NAMES)
selected: set[str] = set()
Expand All @@ -85,6 +102,15 @@ def parse_real_stages(environ: Mapping[str, str] | None = None) -> frozenset[str
if not name:
continue
if name == CPU_ALIAS:
vision = env.get(VISION_VAR, "")
# reads switches on for "on"; criteria also for 1/true/yes. _flag
# covers both sets, so the alias never pulls in a paid call.
if _flag(vision):
raise ValueError(
f"{ENV_VAR} uses the alias {CPU_ALIAS!r}, which promises no paid call, but "
f"{VISION_VAR}={vision!r} lets reads and criteria call a vision model. Set "
f"{VISION_VAR}=off, or list the stages by name."
)
selected.update(CPU_STAGES)
continue
if name not in known:
Expand Down
Loading