template: FC_SKIP_PYTEST and FC_SKIP_SNAPSHOT for timing-only replay trials

[ArjunS07]

Purpose

Timing-only replay trials ("retime", no agent) only need the timing pass. This PR lets the trial environment turn off pytest and the snapshot check with FC_SKIP_PYTEST=1 and FC_SKIP_SNAPSHOT=1. Attempts and measure jobs never set these variables, so their behaviour does not change.

How it works

flowchart LR
  M[LSV measure] --> S{FC_SKIP_SNAPSHOT=1?}
  S -- yes --> SM[write snapshot_skipped.json]
  S -- no --> SV[snapshot baseline / download / verify]
  SM --> P{FC_SKIP_PYTEST=1?}
  SV --> P
  P -- yes --> PM[write skip marker to test_results.json]
  P -- no --> PR[pytest_runner.py]
  PM --> R[parser.py: reward.json]
  PR --> R

Loading

Only the exact value 1 turns a step off. An unset variable or any other value (0, true, empty) runs every step as before.

A skipped step is recorded as not run. It is never recorded as a pass:

Field in reward.json	pytest skipped	snapshot skipped
tests_passed	null	unchanged
tests_ran (new)	false	unchanged
pytest_skipped (new)	"FC_SKIP_PYTEST"	unchanged
snapshots_passed	unchanged	null
snapshot_verify_ran	unchanged	false
snapshot_skipped (new)	unchanged	"FC_SKIP_SNAPSHOT"

When nothing is skipped, tests_ran is true if test_results.json exists, and pytest_skipped and snapshot_skipped are null.

Changes

1. test.sh: with FC_SKIP_SNAPSHOT=1, the oracle snapshot baseline, the snapshot download and snapshot-tool verify do not run. The script writes snapshot_skipped.json and logs one line.
   Effect: no snapshot work and no snapshot result in a replay trial.
   Before: the snapshot steps always ran when their inputs were present.
2. test.sh: with FC_SKIP_PYTEST=1, pytest_runner.py does not run. The script writes {"pytest_skipped": "FC_SKIP_PYTEST", "tests_ran": false, "tests_passed": null} to test_results.json and logs one line. This also replaces the "Tests skipped as per configuration" passing result of tasks rendered without pytest.
   Effect: no pytest work in a replay trial, and no fake pass.
   Before: pytest always ran (or the configured passing result was written).
3. test.sh: when both flags are set, the trial does not take the measure gate slot for the tests step.
   Effect: a replay trial does not wait for a slot it does not use.
   Before: the slot was always taken.
4. parser.py: reads the two skip markers. A skipped pytest gives tests_passed=null, an empty pytest block and the new top-level tests_ran and pytest_skipped fields. A skipped snapshot check gives snapshots_passed=null and snapshot_verify_ran=false even if a summary file is present, plus the new snapshot_skipped field. reward.txt still gives the failure penalty when tests_passed is not true.
   Effect: a skip cannot read as a pass.
   Before: these fields did not exist.
5. test.sh: the two flags are read on the first lines, before any file is sourced, into readonly variables _fc_skip_pytest and _fc_skip_snapshot. Then FC_SKIP_PYTEST and FC_SKIP_SNAPSHOT are unset. All later checks use only the readonly variables.
   Effect: a file under /etc/profile.d cannot turn a skip on or off, by export, by plain assignment or by declare -g.
   Before: (first commit of this PR) the flags were read after sourcing, so the agent could add export FC_SKIP_PYTEST=1 to a sourced file and skip pytest.
6. tests/docker/test_template_skip_flags.py (new) and test_template_pytest_step.py (block extraction updated for the new if).

Verification

• PYTHONPATH=src .venv/bin/python -m pytest tests/docker: 448 passed, 10 skipped.
• Tamper tests: the test harness runs the real script head, which sources a profile.d file. A file that sets both flags (by export, by assignment and by declare -g) does not skip any step. A file that sets them to 0 or unsets them does not undo an incoming skip. Both tests fail when the readonly and unset lines are removed.
• New tests: the rendered script is valid bash for both run_pytest values; unset, empty, 0 and true run every step in the same order; each flag skips only its own step; both flags take no gate slot; the parser output for each skip and for no skip.
• parser.py run on Python 3.8 (uv run --no-project --python 3.8) with both skip markers gives tests_passed=None, tests_ran=False, pytest_skipped='FC_SKIP_PYTEST', snapshots_passed=None, snapshot_verify_ran=False, snapshot_skipped='FC_SKIP_SNAPSHOT'.

Notes

• FC_SKIP_SNAPSHOT=1 also skips the oracle snapshot baseline recording, because it is part of the same snapshot step.
• The three new fields are always present in reward.json. Existing field names and values do not change.
• test.sh sources /etc/profile.d/asv_utils.sh and /etc/profile.d/asv_build_vars.sh after the agent phase, and the agent (root) can write to them. This PR protects only the two skip flags. The other values that test.sh and its scripts read after sourcing need a separate review:
  • test.sh: ENV_NAME, BENCHMARK_DIR, FC_PYTEST_TIMEOUT, FORMULACODE_SNAPSHOT_FILTER, FORMULACODE_SNAPSHOT_TIMEOUT, FORMULACODE_SNAPSHOT_WORKERS, FORMULACODE_NO_UPLOAD, LSV_OUTPUT_DIR, MEASURE_GATE_URL, MEASURE_GATE_CONNECT_TIMEOUT, MEASURE_GATE_ACQUIRE_WAIT, SUPABASE_URL, SUPABASE_ANON_KEY.
  • lsv_measure.py: BENCHMARK_DATA, LSV_OUTPUT_DIR, FC_REBUILD_CMD (set by test.sh after sourcing).
  • pytest_runner.py: PYTHONPYCACHEPREFIX, T_BENCH_TASK_LOGS_PATH, T_BENCH_CONTAINER_LOGS_PATH.
  • parser.py: DATASMITH_VERIFY_CONSTANT_FACTOR_MIN, DATASMITH_VERIFY_CONSTANT_FACTOR_MAX, DATASMITH_VERIFY_DILUTION_RATIO_MAX, DATASMITH_VERIFY_MEASURE_GEOMEAN_MIN, FORMULACODE_BASELINE_FROM_CACHE, FORMULACODE_EXPECTED_N, T_BENCH_TASK_LOGS_PATH, SUPABASE_URL, SUPABASE_ANON_KEY.
  • upload.py: SUPABASE_SERVICE_KEY as well.
  • rebuild.sh: CFLAGS, CPPFLAGS, CONDA_PREFIX, NP_INC, PYTHONPATH, TILEDB_PATH.
  • Not only variables: a sourced file can also change PATH and PYTHONPATH, or define shell functions such as python, timeout, snapshot-tool or mg_acquire, which test.sh then calls.