feat: add shared engineering evaluation and dual-NPU MoE task

.github/workflows/ci-ready.yml

@@ -68,6 +68,37 @@ jobs:
           PR_BODY: ${{ github.event.pull_request.body }}
         run: python3 tools/check_pr_references.py
 
+  benchmark-artifact-controls:
+    # Software correctness only: no model, secrets, device or performance gates.
+    runs-on: ubuntu-22.04
+    timeout-minutes: 10
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+        with:
+          persist-credentials: false
+      - name: Install isolated artifact test dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install --yes --no-install-recommends bubblewrap
+          sudo /usr/bin/python3 -m pip install "numpy>=1.24"
+          /usr/bin/python3 -m pip install "pytest>=7.4" "PyYAML>=6.0"
+          /usr/bin/python3 -I -c "import numpy; print(numpy.__version__)"
+      - name: Real NumPy sandbox and authored two-arm software controls
+        env:
+          INFERRSI_REQUIRE_BENCHMARK_SANDBOX: "1"
+        run: |
+          /usr/bin/python3 -m pytest tests/test_skill_eval_benchmark.py tests/test_skill_eval_benchmark_compare.py --junitxml=benchmark-controls.xml
+      - name: Retain software-control results including failures
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: benchmark-artifact-controls-${{ github.run_id }}-${{ github.run_attempt }}
+          path: benchmark-controls.xml
+          if-no-files-found: error
+          retention-days: 14
+
   kv-task-qualification:
     # Bounded software-control qualification; not a branch-protection or efficacy gate.
     if: github.event.pull_request.draft == false

docs/architecture.md

@@ -1,5 +1,20 @@
 # Modular architecture
 
+The [shared engineering benchmark adapter](engineering-benchmarks.md) lives in
+`harness/benchmarks`, with task data in `data/engineering_benchmarks`. It binds
+task-owned oracles and workloads to the existing Workbench runner and offline
+SkillEval verifier contract. It adds neither a search controller nor another
+execution platform. The current device pilot runs trusted affine RMSNorm on
+Ascend through Workbench; the formal artifact sandbox remains CPU NumPy only.
+Each task's actual qualification and limits are recorded separately.
+
+The [two-rank MoE scenario](moe-forward-benchmark.md) has its own distributed
+task/grader contract under `data/engineering_scenarios` and reuses
+`harness.scenarios.external.execute_trial`. Its paired, slowest-rank timing is
+not the single-operation TaskPackage/SkillEval adapter. Historical Agent runs
+and their published evidence remain distinct from the portable runner's current
+software and device qualification.
+
 The local experiment path now has concrete Harness, Model and Infra modules. The
 CLI, RunSpec, knowledge retrieval and existing command contracts remain stable.
 The diagrams below describe implemented code, with the historical compatibility

docs/engineering-benchmarks.md

@@ -0,0 +1,155 @@
+# Shared engineering benchmarks
+
+The `benchmarks` commands check and measure an implementation against a versioned
+task package. Packages define semantics, inputs, correctness rules and timing;
+the existing Workbench runs paired comparisons and retains evidence. Agent
+execution, knowledge selection and optimization search remain caller-owned.
+
+For a complete distributed layer rather than a single-operation task, see the
+[two-NPU MoE forward scenario](moe-forward-benchmark.md). Its local paired runner
+uses the existing external executor with a distributed grader. It has a separate
+task contract and is not accepted by the single-operation `benchmarks` adapter.
+The [six-session MoE comparison](../evidence/moe-agent-comparison-20261104/README.md)
+publishes historical measurements, candidate snapshots, token costs and curves;
+it does not establish device validation of the newly packaged runner.
+
+| Task | Candidate entry point | Execution boundary |
+| --- | --- | --- |
+| [Ascend affine RMSNorm](../src/inferrsi/data/engineering_benchmarks/rmsnorm-npu/README.md) | `compute(x, weight, *, eps)` | Reviewed source on one caller-allocated NPU; no formal device sandbox |
+| [CPU Softmax](../src/inferrsi/data/engineering_benchmarks/softmax-cpu/README.md) | `compute(x, *, eps)` | Reviewed NumPy source; a separate SkillEval artifact sandbox also supports this task |
+
+The task READMEs retain source attribution, licenses and workload limits. To
+optimize an implementation, copy the task's `baseline.py` to a separate
+`implementation.py`. The packaged `candidate.py` is a reference control, not an
+Agent-produced optimization. Keep task, oracle and baseline fixed while changing
+the implementation. Review source before using the trusted worker: content
+hashes detect drift, not malicious code.
+
+## Run an NPU implementation
+
+Discovery imports neither torch nor torch_npu. Execution requires the target's
+compatible installed torch/torch_npu/CANN stack; InferRSI does not install or
+replace that runtime. Enter a caller-managed single-NPU allocation before any
+device execution. The worker requires one `ASCEND_RT_VISIBLE_DEVICES` entry,
+uses logical device 0 and never falls back to CPU.
+
+```bash
+inferrsi benchmarks list
+inferrsi benchmarks show rmsnorm-npu
+inferrsi benchmarks check rmsnorm-npu --candidate <implementation.py> --output-dir <new-check-dir> --timeout-seconds <timeout>
+inferrsi benchmarks compare rmsnorm-npu --candidate <implementation.py> --repo <checkout> --output-dir <new-run-dir> --repetitions <pairs> --max-minutes <budget> --min-improvement-pct <gain-gate>
+```
+
+RMSNorm uses contiguous 2D float32 inputs and nonuniform weights. Input and
+weight generation, scalar double/math.fsum oracle computation and transfers
+occur outside timing. The clock covers eager dispatch and completion
+synchronization after an untimed pre-synchronization. It measures synchronized
+eager-call latency, not pure kernel time. Shape, dtype, device, numerical values
+and input/weight preservation are checked outside the timer for every call.
+
+Profiling is optional and separate from measurement:
+
+```bash
+inferrsi benchmarks profile rmsnorm-npu --candidate <implementation.py> --output-dir <new-profile-dir> --timeout-seconds <timeout>
+```
+
+This retains real device events in `npu-trace.json` and a path-free kernel
+summary, with no performance verdict. Distinguish verification copies from
+compute kernels. Review raw traces for private paths/identifiers before sharing.
+
+## Use the CPU software control
+
+Install the development checkout with `pip install -e ".[dev,benchmarks]"`.
+Discovery and planning do not import NumPy; CPU execution requires it.
+
+```bash
+inferrsi benchmarks show softmax-cpu
+inferrsi benchmarks check softmax-cpu --candidate <implementation.py> --output-dir <new-check-dir> --timeout-seconds <timeout>
+inferrsi benchmarks measure softmax-cpu --candidate <implementation.py> --output-dir <new-measure-dir> --timeout-seconds <timeout>
+inferrsi benchmarks grade softmax-cpu --candidate <implementation.py> --output-dir <new-grade-dir> --timeout-seconds <timeout>
+inferrsi benchmarks plan softmax-cpu --candidate <implementation.py> --repo <checkout> --output-dir <new-run-dir> --repetitions <pairs> --max-minutes <budget> --min-improvement-pct <gain-gate>
+```
+
+The Softmax contract reduces axis 1 and includes strided and boundary inputs.
+The independent oracle uses scalar double precision and `math.fsum`. `check`
+evaluates the base cases and the same varying-input sequence used by measurement.
+Every timed input has distinct retained objects and bytes, so an earlier call
+cannot prime an identity/content memo for a later sample. This public suite does
+not prevent hostile implementations or establish held-out generalization.
+
+Input generation, oracle evaluation, validation, startup and warmups are outside
+timing; allocations inside `compute` are included. The score sums per-case
+medians for the task's selected workloads. Inspect individual cases as well as
+the aggregate. The zero-input case checks correctness but contributes no timing
+score. CPU `profile` retains `profile.txt`, not a device timeline or speed score.
+
+## Comparison and retained results
+
+Replace `plan` with `compare` to execute through Workbench. `plan` prints a
+RunSpec; `compare` constructs its own RunSpec rather than consuming a previously
+printed one. Freeze the actual inputs and gates before candidate search using
+the [acceptance procedure](contributing.md#acceptance-checklist). All placeholders
+above come from that experiment, not repository defaults. Single-action output
+directories must be new.
+
+`TaskPackage` freezes task and grader identities. Commands also bind candidate
+bytes and runtime fingerprints, checked before and after worker execution. The
+Git checkout supplied by `--repo` identifies source context; task and candidate
+files have additional hashes. Runtime fingerprints alone do not establish an
+uncontended host, equivalent device/CANN installations or native dependencies.
+
+Results retain the planned `input_sequence`, actual `executed_calls` and sample
+input hashes. Failed candidates may execute only a prefix. Wrong outputs produce
+no usable performance metric. Preserve unsuccessful trials and inspect
+`execution.json`, per-trial `result.json`, `evaluation.json`, `status.json` and
+source-integrity records.
+
+`compare` returns `run_dir` and the retained status. Its exit codes are PASS = 0,
+FAIL = 1 and INCONCLUSIVE = 2. A complete comparison missing its gain gate is
+FAIL. A wrong candidate preventing measurement leaves the comparison
+INCONCLUSIVE, while independent artifact correctness can be FAIL. Exceptions,
+missing dependencies, drift and timeouts must not become successful comparisons.
+Workbench statistics are descriptive; these results do not establish service
+gains, statistical significance or Agent/knowledge efficacy.
+
+`grade` returns artifact-correctness PASS/FAIL/INCONCLUSIVE with SkillEval's
+`passed`/`assertions` structure. The lower-level
+`benchmarks.skill_verifier.task_descriptor` binds an already reviewed artifact
+for `skill_evaluation.run_trial`; its answer supplies `candidate_sha256`.
+For actual CPU file handoff through `tools.skill_eval_compare`, see
+[engineering task artifacts](engineering-task-artifacts.md). That formal adapter
+checks correctness only and rejects NPU tasks. It does not automatically send
+generated NPU code to this trusted worker. Provider generation mode and the
+offline harness-improvement controller do not yet support this task adapter.
+
+## Qualify the tools with authored controls
+
+The NPU utility exercises correct, wrong, replay, same-code and reference
+implementations through the worker and Workbench:
+
+```bash
+python -m tools.npu_benchmark_validate --repo <checkout> --output-dir <new-evidence-dir> --repetitions <pairs> --min-improvement-pct <gain-gate> --max-aa-difference-pct <noise-gate> --max-minutes <comparison-budget> --timeout-seconds <action-timeout>
+```
+
+It freezes policy and implementation identities before execution, retains all
+samples and checks actual device/CANN/runtime consistency. The utility includes
+profiling as a tool control; this does not make profiling mandatory for ordinary
+optimization. `--timeout-seconds` caps its standalone actions, while paired
+Workbench trials use the comparison deadline. It does not enforce a separate
+whole-utility budget or the standalone cap on each paired trial. Establish the
+required outer allocation/process deadlines and verify cleanup before relying
+on those budget boundaries.
+
+The CPU integration utility dynamically selects the packaged NumPy task:
+
+```bash
+python -m tools.engineering_benchmark_validate --repo <checkout> --output-dir <new-evidence-dir> --repetitions <pairs>
+```
+
+It retains profiles and AB/BA samples and runs the existing SkillEval trial path
+with correct and wrong authored controls. Neither utility generates Agent
+candidates. Their reports separate software controls from campaign acceptance.
+Preserve experiment plans and original results with their tested revisions using
+the [evidence conventions](contributing.md#evidence-and-claims); review/redact
+copies of raw logs before publication. A successful control run does not finish
+the Agent-to-NPU handoff or prove an optimization benefit.

docs/engineering-task-artifacts.md

@@ -0,0 +1,115 @@
+# Engineering task artifacts in SkillEval
+
+`engineering-benchmark-v1` connects the packaged `softmax-cpu` task to
+`tools.skill_eval_compare` in **update mode**. It freezes two skill versions,
+retains each producer's actual `implementation.py`, and grades correctness with
+the shared task oracle. It does not measure speed. Reviewed implementations can
+later enter a separate [Workbench comparison](engineering-benchmarks.md).
+
+## Prepare the task and comparison
+
+Run from a development checkout. Generate a canonical case rather than editing
+its contract or relying on a same-named directory in the working tree:
+
+```bash
+python -c "from inferrsi.harness.skill_eval_benchmark import task_descriptor; from inferrsi.infra.artifacts import write_json; from pathlib import Path; write_json(Path('cases.json'), [task_descriptor('softmax-cpu')])"
+```
+
+Create `config.json` beside `cases.json`:
+
+```json
+{
+  "adapter": "engineering-benchmark-v1",
+  "mode": "update",
+  "skill": "<skill-id>",
+  "skill_root": "<repo-relative-skill-root>",
+  "cases": "cases.json",
+  "snapshots": {
+    "baseline": {"git_revision": "<old-skill-full-commit>"},
+    "candidate": {"git_revision": "<new-skill-full-commit>"}
+  },
+  "model": {"name": "<approved-model>", "effort": "<frozen-effort>"},
+  "limits": {
+    "repeats": 1,
+    "max_task_calls": 2,
+    "timeout_seconds": 60,
+    "campaign_seconds": 180,
+    "token_stop_new_trials": 10000
+  }
+}
+```
+
+The example limits are illustrative; choose and freeze the actual bounds before
+execution. Use full commits available in the local repository. Do not add a
+provider client, custom schema or grader. See [comparison configuration](skill-eval-comparison.md)
+for Git prerequisites, snapshot rules and budget semantics.
+
+```bash
+python -m tools.skill_eval_compare prepare --config config.json --repo <skill-checkout> --output <new-evidence-directory>
+```
+
+Preparation freezes task, starter, schema, verifier/controllers and skill
+bundles. Formal loading always uses canonical packaged resources. Retained task
+copies must match that descriptor. The supplied CPU starter is already correct;
+leaving it unchanged can pass, so correctness alone cannot identify optimization.
+
+## Hand off and retain actual files
+
+Stage a fresh workspace outside the evidence tree for one arm:
+
+```bash
+python -m tools.skill_eval_compare execute --output <evidence-directory> --authorize <plan-sha256> --stage-local <new-baseline-workspace> --task-id softmax-cpu --condition baseline --repeat 1 --current-head <new-skill-full-commit>
+```
+
+Stage does not launch an Agent. Run the producer under the caller's established
+permissions and budget. Only `implementation.py` may change; keep task,
+instructions and assigned knowledge frozen. The file implements
+`compute(x, *, eps)` with self-contained NumPy/math helpers. Its final answer is:
+
+```json
+{"artifact": "implementation.py", "summary": "Describe the actual change"}
+```
+
+Stop the producer before capture. Retain `stdout.log`, `stderr.log` and
+`execution.json` in a separate submission directory, following the exact fields
+in [author-run handoff](local-skill-evaluation.md#author-run-handoff).
+Model, access, resources and usage in these files are author-reported, not
+authenticated observations. Missing usage stays unknown.
+
+```bash
+python -m tools.skill_eval_compare execute --output <evidence-directory> --authorize <plan-sha256> --capture-local <baseline-workspace> --submission <baseline-logs> --task-id softmax-cpu --condition baseline --repeat 1 --current-head <new-skill-full-commit>
+```
+
+Repeat stage/produce/capture for `--condition candidate` with distinct workspace
+and log directories, then run:
+
+```bash
+python -m tools.skill_eval_compare report --output <evidence-directory>
+```
+
+The plan hash binds this attempt; it does not authorize model spending. Changed
+inputs require a new plan. Capture preserves actual source bytes and hashes;
+report rechecks retained artifacts without rerunning the producer. Failed or
+incomplete attempts remain recorded and cannot be overwritten as clean results.
+
+## Execution boundary and interpretation
+
+The trusted parent owns inputs, the scalar oracle and assertions. Submitted
+source executes in a Linux bubblewrap child with system Python, matching NumPy
+and a bounded runner; the repository, grader and other arm are not mounted.
+Every returned array is checked by the parent. The public suite does not prove
+hidden-test generalization, hostile-protocol resistance or independent tracing
+of every candidate operation. Producer permissions are a separate boundary.
+
+Windows, missing bubblewrap/namespace support or an incompatible NumPy runtime
+produce `artifact_sandbox_unavailable` / INCONCLUSIVE without host fallback.
+NPU tasks are explicitly rejected. CPU replay qualification does not establish
+NPU isolation, and the default unrestricted Agent client is refused for automatic
+engineering runs. Provider generation and the offline improvement controller do
+not support this adapter.
+
+A PASS means correctness on the public input sequence, not faster code. A local
+handoff report showing `artifact_improved` establishes a difference in retained
+artifact checks, not a causal skill benefit or verified total cost. Performance,
+Agent efficacy and service claims need their own controlled evidence. Profiling
+is optional and is not invoked by this adapter.