pgw#1142 (§4.32 item 4): an operator can turn compiled serving off on a LIVE worker — and back on

changelog.d/pgw1142.md

@@ -0,0 +1 @@
+- **pgw#1142 (DESIGN-RULINGS §4.32 item 4): the operator can turn compiled serving OFF on a live worker — and back on.** Paul, verbatim: *"The consumer should have the ability to turn off compile entirely, and serve-eager only, if the compile is broken or they just don't care. I.e., they can send some sort of command to the worker to serve eager rather than serve compiled. This is useful for the hub and cozy-local."* Half the vocabulary already existed and is REUSED rather than duplicated: `eager_only` is a Plan `ExecutionSpec` backend (`STEADY_BACKEND_EAGER_ONLY` → `arm_ordered`: *"eager_only arms nothing, by order"*), and pgw#714's `operator_eager_pin` is the hub-resolved lane pinned in config. Both are decided BEFORE a pod serves and neither can be taken back, which is exactly what "the compile is broken" — a thing you discover mid-flight — needs. There was no env toggle to migrate (`GEN_WORKER_FORCE_EAGER` and friends do not exist, grep-verified); this is the first explicit mechanism, and it is a **command over the existing scheduler control channel** (`ServePosture` on `SchedulerMessage`, th#1808), never an env: §1.17 — an env may carry a VALUE, not a DECISION. **Scope, decided and recorded:** WHOLE-WORKER (the operator's question is "is compile broken on this pod"; per-function needs a second addressing vocabulary for a case nobody has had), RUNTIME (a boot-only switch means recycling the pod to use it, which is the cost it exists to avoid), and REVERSIBLE — which is what fixes the enforcement point. The order is read AT THE CALL in `aot_serve`'s wrapper, so an armed artifact is never unwrapped and releasing the order resumes compiled serving on the very next request with no re-arm, no re-materialize and no re-mint; unwrapping would have produced the same first half and a permanently eager pod afterwards. **One posture, one gate:** `compile_cache.arming_block` — the ONE precondition authority — answers the order first, so adoption, JIT intake, cold compile and every self-mint are suppressed by one check rather than by a check per call site; `fleet_cells.enable_compiled` short-circuits BEFORE the policy so no cell is resolved, downloaded or minted for a gate that has already decided; `arm_ordered` obeys the order over the hub's own Plan by arming nothing and serving (never `OrderedArmError` — killing the function is the opposite of "serve eager instead"); and the executor's boot path skips boot-adopt so nothing is fetched at all. **It composes with §4.31's sticky de-arm without being mistakable for it:** same eager posture, two triggers — the de-arm is automatic, per-cell, evidence, irreversible for the boot; this is operator-explicit, whole-worker, policy, reversible — and releasing the order deliberately does NOT resurrect a cell that de-armed itself for cause. **Telemetry says which:** a new `EagerPhase.OPERATOR_EAGER_ONLY` (`operator_eager_only`, pinned in the wire-contract test), aliased into `serving_mode` and `boot_adopt`'s gate reasons, plus a `serve_posture` activity kind carrying `eager_only_engaged`/`eager_only_released` with the operator's own words. `serving_mode.resolve` reports a suppressed request as `eager` rather than `aot_cell` — the cell is still armed, so naive classification reads exactly the wire lie pgw#1082/#1093 spent two pods closing — and it is not counted as a fallback, because nothing fell back. **cozy-local:** `gen-worker serve --eager-only` / `run --eager-only` for the boot, and a `{"posture": {...}}` control frame on the serve socket for a warm serve (cl#49 drives both). Named `--eager-only`, never `--eager`, which means eager LOADING and is a false friend this lane nearly inherited.

docs/local-dev.md

@@ -258,6 +258,23 @@ uses):
   If launching `serve` in the background, pass `--no-stdin` so it doesn't
   consume the parent shell's stdin.
 
+  **Turning compiled serving off** (DESIGN-RULINGS §4.32 item 4). Start with
+  `--eager-only` and the serve arms no compiled cell and mints none — for a
+  cell that turns out to be broken, or simply to skip a mint you do not want
+  to spend your machine's minutes on. On an already-running serve, send the
+  posture control frame; it is reversible, and the reverse costs nothing
+  because the artifact is never unwrapped, only bypassed:
+
+  ```bash
+  $ printf '{"posture":{"eager_only":true,"reason":"the cell crashes"}}\n' \
+      | nc -U .gen-worker.sock
+  {"ok":true,"eager_only":true,"changed":true,"posture":"compiled serving suppressed by cozy-local-cli"}
+  ```
+
+  (`cozy serve --eager-only` and `cozy eager-only <owner>/<ep> [on|off]` are
+  the cozy-local spellings of the same two calls.) Not to be confused with
+  `--eager`, which decides when `setup()` runs, not how forwards execute.
+
 - **stdin/stdout NDJSON (default, single process):** pipe a batch of
   newline-delimited JSON requests in; get one NDJSON response line each. Logs
   go to stderr. The process exits when stdin closes.

proto/PROTO_DIGEST

@@ -3,4 +3,4 @@
 # BOTH repos commit this same digest, so a one-sided edit fails scripts/proto-drift-check.sh
 # offline, in the repo where it happened. Regenerate with:
 #   sha256sum internal/orchestrator/grpc/proto/worker_scheduler.proto
-b0c9964eb10d7b333ed69ebc99fd88b91ae6ad957a358621192d36a85dbd5e10
+c173533e248d44a4917080f80d1d32582886c5fb0a5027fe59d6d75ae8b2cb19

proto/worker_scheduler.proto

@@ -114,6 +114,8 @@ message SchedulerMessage {
     // behaves exactly as before. At the coordinated cutover RunJob is deleted
     // and field 2 + the name "run_job" become `reserved` (§1.27(f)).
     RunAttempt   run_attempt   = 7;
+    // §4.32 item 4 / th#1808 / pgw#1142: the operator's eager-only command.
+    ServePosture serve_posture = 8;
   }
 }
 
@@ -1782,3 +1784,38 @@ message TokenRefresh {
   string token = 1;           // replacement worker JWT
   int64  expires_at_unix = 2; // exp claim of the replacement (observability)
 }
+
+// ---------------------------------------------------------------------------
+// Serve posture (DESIGN-RULINGS §4.32 item 4)
+// ---------------------------------------------------------------------------
+
+// Orchestrator -> worker: the OPERATOR's eager-only command. Paul, §4.32 item
+// 4: *"The consumer should have the ability to turn off compile entirely, and
+// serve-eager only, if the compile is broken or they just don't care."*
+//
+// It is a COMMAND, not config and not an env var (§1.17): whole-worker,
+// runtime, and REVERSIBLE in both directions on a live stream — "the compile is
+// broken" is discovered while serving, and so is "it was not the compile".
+//
+// Distinct from the two eager postures that already exist, deliberately:
+//   * `ExecutionSpec`'s `STEADY_BACKEND_EAGER_ONLY` is PLAN-carried — chosen
+//     per dispatch, before the pod is serving, and reports
+//     `hub_ordered_eager`;
+//   * §4.31's sticky de-arm is AUTOMATIC and per-cell — a cell-attributable
+//     failure, evidence, irreversible for the boot.
+// This one is operator-explicit, whole-worker and reversible, and it reports
+// its own token (`operator_eager_only`) so a suppressed pod's telemetry is
+// never mistakable for a broken adopt path.
+//
+// No reply message; the worker emits a typed activity event on each transition.
+// Additive proto3 oneof field: a worker that predates it ignores it.
+message ServePosture {
+  // true  = serve eager only: arm nothing, mint nothing, and answer every
+  //         request from the eager forward even where a cell is already armed
+  //         (the arm is NOT unwrapped, which is what makes false reversible).
+  // false = release the suppression; already-armed cells serve compiled again
+  //         on the next request, and later arming decisions run normally.
+  bool   eager_only = 1;
+  string reason     = 2; // operator's words, carried into the worker's event
+  string actor      = 3; // who ordered it (admin principal / "cozy-local")
+}

src/gen_worker/activity.py

@@ -87,6 +87,14 @@
 # ``detail`` naming the identifiers (ref/function/label/request) + the
 # exception. Fail-soft BEHAVIOR is unchanged — these are confessions.
 KIND_SERVE_DEGRADE = "serve_degrade"
+# pgw#1142 / §4.32 item 4: the OPERATOR's eager-only order changed state on this
+# worker. Two phases, both transitions and neither a degradation:
+# `eager_only_engaged` (compiled serving suppressed — armed cells stay armed and
+# are not called) and `eager_only_released`. Deliberately its own kind rather
+# than a `serve_degrade` phase: every other member of that kind is something
+# that went wrong, and an operator exercising a supported control must not land
+# in a defect population the fleet reads as breakage.
+KIND_SERVE_POSTURE = "serve_posture"
 KIND_LORA_HYGIENE = "lora_hygiene"
 # pgw#794: the serve-side adapter-fidelity gate. `phase=refused` is a
 # fail-CLOSED decision (the request also gets a typed error); `phase=degraded`

src/gen_worker/aot_serve.py

@@ -95,6 +95,7 @@
 from . import cell_key as cell_key_mod
 from . import host_isa
 from .cell_adopt import AdoptOutcome
+from . import serve_posture
 from . import shape_growth
 from .compile_cache import (
     AdoptError,
@@ -2452,6 +2453,19 @@ def aot_forward(*args: Any, **kwargs: Any) -> Any:
         artifact_lora, eager_lora = adapter_call_kwargs(module, runner)
         eager_kwargs = {**kwargs, **eager_lora}
         kwargs = {**kwargs, **artifact_lora}
+        if serve_posture.eager_only():
+            # pgw#1142 / §4.32 item 4: an operator ordered eager. This is THE
+            # reversibility seam — the artifact is not unwrapped and `state`
+            # is not touched, so releasing the order resumes compiled serving
+            # on the very next call with no re-arm, no re-materialize and no
+            # re-mint. Unwrapping here would have been the same posture for
+            # one boot and a lie afterwards.
+            #
+            # Ordered BEFORE the `failed` check only for cost; the two are
+            # independent and stay independent — releasing the order never
+            # resurrects a cell de-armed for cause (§4.31), because that
+            # de-arm is evidence and this is policy.
+            return original(*args, **eager_kwargs)
         if state["failed"]:
             return original(*args, **eager_kwargs)
         try:

src/gen_worker/boot_adopt.py

@@ -99,8 +99,19 @@
     # boots without asking. Correct, and previously indistinguishable from a
     # boot-adopt that asked and was refused.
     "eager_only",
+    # pgw#1142 / §4.32 item 4: an OPERATOR ordered this worker eager-only, so
+    # the boot did not ask for a cell it is forbidden to call. Distinct from
+    # `eager_only` above because that one is a property of the pod that no
+    # command can lift, and this one is a decision that can be taken back —
+    # the same pod, re-asked after the order is released, adopts normally.
+    "operator_eager_only",
 )
 
+#: The reason token for :data:`GATE_REASONS`' operator entry. Named so the
+#: executor references the vocabulary instead of re-typing the literal — the
+#: drift channel `EagerPhase` was created to close, one module over.
+OPERATOR_EAGER_ONLY = "operator_eager_only"
+
 #: Step 1.5 (pgw#1127 S2) — THIS MACHINE's own store, addressed by the DERIVED
 #: key, before the hub is asked at all. §4.28: *"local cell, local repo-CAS,
 #: reused across its own boots — never uploaded, never requested."*

src/gen_worker/cell_adopt.py

@@ -166,6 +166,15 @@ class EagerPhase(StrEnum):
     #: `state_blocked_idle`, two replacements bought.
     ADOPTED_CELL_REFUSED = "adopted_cell_refused"
 
+    #: pgw#1142 / §4.32 item 4: an OPERATOR ordered this worker to serve eager
+    #: only, over the scheduler's control channel or the cozy-local CLI. It is
+    #: neither a defect nor a degradation — it is the answer, and it is
+    #: reversible — so it must never be counted with the failure classes above
+    #: or with `hub_ordered_eager` (which is one PLAN's backend, not a standing
+    #: order about this pod). A worker holding this token has cells it could be
+    #: serving from, still armed, deliberately not called.
+    OPERATOR_EAGER_ONLY = "operator_eager_only"
+
     #: `_fail_closed`'s default, for a caller that has not classified its exit.
     #: A new decline landing here rather than on its own member is the
     #: regression pgw#824 exists to catch.

src/gen_worker/cli/run.py

@@ -34,6 +34,7 @@
     from .. import local_serve
 
 from ..api.errors import CanceledError
+from .. import serve_posture
 from ..models import provision
 from .local_context import build_local_context
 from ..discovery.project import load_project_config
@@ -103,6 +104,14 @@ def add_subparser(sub: argparse._SubParsersAction[Any]) -> None:
             "missing model weights from tensorhub / huggingface."
         ),
     )
+    p.add_argument(
+        "--eager-only", dest="eager_only", action="store_true",
+        help=(
+            "Serve EAGER ONLY: never arm a compiled cell and never mint one "
+            "(DESIGN-RULINGS §4.32). One invocation is where a cold machine "
+            "would otherwise spend minutes minting."
+        ),
+    )
     p.add_argument(
         "--device", dest="device", default=None,
         help="Override the torch device (e.g. 'cuda:0', 'cpu').",
@@ -1068,6 +1077,11 @@ async def _drain() -> int:
 # --------------------------------------------------------------------------
 
 def _handle_run(args: argparse.Namespace) -> int:
+    # pgw#1142 / §4.32 item 4: before anything can arm or mint. `run` does its
+    # setup() per invocation, so the order has to stand before the first one.
+    if bool(getattr(args, "eager_only", False)):
+        serve_posture.apply_command(
+            True, actor="cozy-local-cli", reason="--eager-only")
     try:
         return _run_inner(args)
     except _UsageError as e:

src/gen_worker/cli/serve.py

@@ -50,6 +50,7 @@
 import msgspec
 
 from ..api.errors import CanceledError
+from .. import serve_posture
 from ..discovery.names import slugify_name
 from ..models import memory
 from ..models.residency import Residency, Tier
@@ -162,6 +163,17 @@ def add_subparser(sub: argparse._SubParsersAction[Any]) -> None:
             "fail-fast / pre-warming; default is lazy per-function loading."
         ),
     )
+    p.add_argument(
+        "--eager-only", dest="eager_only", action="store_true",
+        help=(
+            "Serve EAGER ONLY: never arm a compiled cell and never mint one "
+            "(DESIGN-RULINGS §4.32). For a broken compile, or a machine whose "
+            "owner does not want minutes of it spent compiling. Reversible "
+            "while the serve runs — send {\"posture\":{\"eager_only\":false}} "
+            "on the socket. NOTE: unrelated to --eager, which is about when "
+            "setup() runs, not about how forwards execute."
+        ),
+    )
     p.add_argument(
         "--vram-budget", dest="vram_budget", type=float, default=0.0,
         metavar="GB",
@@ -682,6 +694,8 @@ def _parse_frame(line: bytes) -> Dict[str, Any]:
     Kinds:
       - ``{"kind":"request","function","payload","request_id"}``
       - ``{"kind":"cancel","request_id"}``  (control frame: ``{"cancel":{...}}``)
+      - ``{"kind":"posture","eager_only","reason"}``
+        (control frame: ``{"posture":{"eager_only":true,"reason":"..."}}``)
       - ``{"kind":"error","message"}``
     """
     try:
@@ -690,6 +704,22 @@ def _parse_frame(line: bytes) -> Dict[str, Any]:
         return {"kind": "error", "message": f"request is not valid JSON: {e}"}
     if not isinstance(obj, dict):
         return {"kind": "error", "message": "request must be a JSON object"}
+    if "posture" in obj:
+        # pgw#1142 / §4.32 item 4: the cozy-local half of the eager-only
+        # command. It rides the control-frame shape `cancel` already
+        # established rather than opening a second channel — one socket, one
+        # protocol, and `cozy` learns one more frame instead of a new client.
+        p = obj.get("posture") or {}
+        if not isinstance(p, dict) or not isinstance(p.get("eager_only"), bool):
+            return {
+                "kind": "error",
+                "message": "posture.eager_only (boolean) is required",
+            }
+        return {
+            "kind": "posture",
+            "eager_only": bool(p["eager_only"]),
+            "reason": str(p.get("reason") or ""),
+        }
     if "cancel" in obj:
         c = obj.get("cancel") or {}
         rid = c.get("request_id") if isinstance(c, dict) else None
@@ -736,6 +766,18 @@ def _emit_stdout(s: str) -> None:
             found = endpoint.interrupt_request(frame.get("request_id"))
             _write_response_line(_emit_stdout, {"ok": True, "canceled": found})
             continue
+        if frame["kind"] == "posture":
+            changed = serve_posture.apply_command(
+                bool(frame["eager_only"]),
+                actor="cozy-local-cli",
+                reason=str(frame.get("reason") or ""),
+            )
+            current = serve_posture.order()
+            _write_response_line(_emit_stdout, {
+                "ok": True, "eager_only": current.active, "changed": changed,
+                "posture": current.describe(),
+            })
+            continue
         rid = frame.get("request_id")
         if frame.get("stream"):
             terminal = endpoint.dispatch(
@@ -875,6 +917,27 @@ def _send(env: Dict[str, Any]) -> None:
         except OSError:
             pass
         return
+    if frame["kind"] == "posture":
+        # pgw#1142 / §4.32 item 4, the RUNTIME half: a warm serve holding a
+        # broken cell can be told to stop using it without being restarted,
+        # and told to use it again afterwards. The reply reports the posture
+        # that now stands, so a client never has to infer it.
+        changed = serve_posture.apply_command(
+            bool(frame["eager_only"]),
+            actor="cozy-local-cli",
+            reason=str(frame.get("reason") or ""),
+        )
+        current = serve_posture.order()
+        try:
+            _send({
+                "ok": True,
+                "eager_only": current.active,
+                "changed": changed,
+                "posture": current.describe(),
+            })
+        except OSError:
+            pass
+        return
     if frame["kind"] == "error":
         try:
             _send(_error_envelope("usage", frame["message"]))
@@ -988,6 +1051,14 @@ def _serve_inner(args: argparse.Namespace) -> int:
         candidates, getattr(args, "functions", None),
     )
 
+    # pgw#1142 / §4.32 item 4: the order is installed BEFORE setup(), because
+    # setup() is where a cozy-local serve arms and (on a miss) mints. A flag
+    # that only took effect afterwards would still have spent the compile the
+    # operator asked us not to spend.
+    if bool(getattr(args, "eager_only", False)):
+        serve_posture.apply_command(
+            True, actor="cozy-local-cli", reason="--eager-only")
+
     # 2. Boot the endpoint — setup() once per class, hold instances warm.
     endpoint = _Endpoint(
         offline=bool(args.offline),

src/gen_worker/compile_cache.py

@@ -73,7 +73,7 @@
 
 from . import (
     cell_key, dist_records, env_seal, guard_closure, hot_swap,
-    settings_authority,
+    serve_posture, settings_authority,
 )
 from .api.errors import RetryableError
 from .models import w8a8_lora
@@ -2328,6 +2328,18 @@ def arming_block(
     Deliberately side-effect free — :func:`apply` still owns the arming
     mutations; this only names.
     """
+    # pgw#1142 / §4.32 item 4, and it is FIRST because it is the cheapest and
+    # the most authoritative: an operator has said this worker serves eager.
+    # Routing the command through the one precondition authority is what makes
+    # it suppress adoption, JIT intake, cold compile and every self-mint
+    # without a check per call site. Unlike every other reason here it is not
+    # deterministic for the life of the process — it can be released — which is
+    # sound for the callers that classify: a mint refused under the order is
+    # refused because the operator does not want one, and if that changes the
+    # next arming pass mints normally.
+    ordered_eager = serve_posture.block()
+    if ordered_eager:
+        return ordered_eager
     if _PROCESS_COMPILES_DISABLED:
         return f"process compiles are disabled: {_PROCESS_COMPILES_DISABLED}"
     if operator_eager_pin(pipeline):