| status | active | ||||||
|---|---|---|---|---|---|---|---|
| owner | release-qa | ||||||
| reviewed | 2026-11-15 | ||||||
| summary | Testing entry point and evidence boundaries; platform guides own execution, the development workflow owns the edit loop, and delivery research stays opt-in. | ||||||
| sourceOfTruth |
|
Start with the current checkpoint and primary roadmap, not a dated run command. Source, project settings, contracts, and repository scripts outrank this guide. This page routes work; it does not duplicate platform procedures or establish another gate.
When a lane is replaced or retired, preserve tested user protections and evidence, not legacy implementation details or unsupported scores. A passing inherited suite does not prove correctness; a newer suite is not exempt from independent validation. Replace or consolidate demonstrated weaknesses within their existing owner, then retire the obsolete execution path.
| Task | Authoritative procedure | Boundary |
|---|---|---|
| Routine edit | Development workflow | scripts/dev.sh check, scoped commit on main; push on explicit request; targeted acceptance at checkpoints |
| macOS UI acceptance | macOS testing | Existing scripts/ui_test.sh macos lanes at coherent checkpoints |
| Generic iOS SDK compile | Host prerequisite | No phone or Simulator; matching Xcode components required |
| iPhone control / generation / candidate checks | iOS testing | Physical-device XCUITest; candidate proof remains distinct from diagnostics |
| Checks XCUITest cannot cover (system settings, permission prompts, installs, audio routing) | Checks XCUITest does not cover | Every UI-driven check is an XCUITest lane (a new scenario when none covers it); the rest is the maintainer's, by hand |
| Device deadline / pause / resume / lock | Pause and resume | Frozen progress stays untracked; validate restoration and exact identities |
| Model download lifecycle | Model delivery | Explicit isolated root; preserve canonical data and failed evidence |
| Benchmark operation / publication | Benchmarking procedure | Exact-source qualified PASS only; no automatic Git changes; the UI benchmark and perf lanes publish on a PASS and say so when they start, --no-publish keeps a probe run local |
| Telemetry fields / schema / knobs | Telemetry reference | Interpretation, not another operator runbook |
| Delivery bench | Benchmarking procedure §4.6 | Paired instructed/neutral takes with fail-closed instruction receipts; acoustic effects, not listener-proven delivery |
| Audio QC | QC v2 | Models run one at a time after TTS exits; report execution, coverage and qualification separately. Missing, stale or unqualified required evidence is NOT_ASSESSED; frozen thresholds and held-out qualification are required for acceptance |
| Release / submission programme | Release-first plan | Implementation, candidate verification, publication approval are separate |
| Gate or contract changes | Development workflow, docs/reference/tooling-and-evidence.md |
Add a check only for a product invariant; prove rejection as well as success; never assert another file's wording |
Routine edits use scripts/dev.sh; no skill is required. The six vocello-* workspace skills in
.agents/plugins/vocello/skills/ select the existing iOS, macOS UI, device diagnostics, benchmark,
debugging and read-only release-readiness procedures; they add no gate or authority
(tool routing).
Antigravity reads the artifacts directly, or hands the run directory to the read-only vocello-xcresult-triage
subagent, and stops when the deciding evidence is clear:
- Read
run.json, the required-step ledger and aggregate result. A missing required step is not PASS. Steps the lane runs recordstartedAtbesidecompletedAt, so a step's duration comes from the ledger itself (steps recorded after the fact withrequired_step_record, such as a skippedui-buildorresult-retention, carry none). The lane's own verdict (verdict.txt, or itstest verdict:/<platform> <lane> PASSline) decides, never a wrapper's or pipe's exit code; when a lane runs in the background, capture its exit status directly. - Inspect the failed test and xcresult summary, then nearby log context. Use the existing bootstrap and external-interruption classifiers when applicable; zero launched cases can be infrastructure.
- Check the crash delta and attachment manifest for relevant control observations.
- Review
screenshots/index.json(readable names, test, time, failure flag): open every failure frame and the key states for clipping, overlap, a wrong state or an unexpected dialog. Report-only; each finding is classified as a release blocker or a backlog line. - Report run id, verdict, deciding step and artifact paths. The verdict classes are PASS, product failure, infrastructure failure, external interruption, restoration gap and incomplete evidence; this list is the one the triage agent and the debug skill use. Never turn a failed run into a pass or silently rerun it.
The SwiftPM runtime lane keeps XCTest stdout in runtime.log and diagnostic stderr in
runtime.stderr.log, without merging concurrent writes. macOS test verdicts record native statuses
and separate core_summary / runtime_summary collection statuses. A complete nonempty
All/Selected tests trailer is required for these summaries; native success with incomplete evidence
fails the lane. Inspect both streams before diagnosing a runtime failure. Historical failed or
inconsistent summaries remain unchanged.
UI runs retain ui-finalization.json and its log separately from native test status. After an
attempted test, failure or interruption still triggers available crash-delta, strict summary and
attachment collection, followed by owned-process cleanup. Each collector has a bounded budget;
timeout, missing evidence or cleanup failure prevents PASS. The first native failure/signal remains
the exit status. Signals during finalization are replayed after remaining collectors, owned-process
cleanup and receipt writing; an earlier native failure retains precedence over a later collection signal.
Read the collection receipt as well as the required-step ledger; neither a clean
receipt nor passed test cases alone establishes successful run completion.
A full macOS smoke, perf or benchmark run also owes every test
config/ui-lane-tests.json lists for its lane. A listed test that
is missing from the results or was skipped ends the run without a PASS (xctest_summary.py --expect-lane exits 4), and test-results.json names it under expected. A --scenario run is one
journey by design and checks nothing. The file changes in the same commit as any renamed, added or
deleted journey (macOS testing).
A benchmark or perf run started with --no-publish holds publication-skipped.json in its run
directory instead of a tracked record; its verdict and artifacts are those of a published run.
Generation UI tests require visible Settings readiness, Generate enabled, and the exact required
Clone reference. macOS fixture repair uses scripts/macos_test.sh models ensure or install
only after visible readiness fails. iOS model repair uses visible Settings controls; no headless
inventory substitutes for UI readiness. The isolated model-download lane is explicitly selected,
never an implicit smoke/benchmark bootstrap. Restart the affected lane after repair.
- XCTest owns visible assertions, activities, screenshots and attachments. The runner additionally owns source/app/device identity, crash deltas, required collection, validators and restoration.
config/orchestration-contract.jsondefines required steps. Missing, interrupted, duplicate, failed or unknown required evidence prevents PASS, even when every XCTest case passed.- Preserve the first failed result. No automatic retries, seed substitution or merging a later pass into the failed bundle. The platform guide defines typed infrastructure classification.
- Resume uses the existing source/build/device/plan-bound validator, never a guessed row count. A zero-observation shard provides no cursor; changed source cannot reuse old acceptance.
- Generated audio, transcripts, database copies, raw telemetry, screenshots, traces and result bundles remain untracked. Compact published benchmark records must be privacy-safe and qualified.
- On the Mac UI benchmark, playback is part of the verdict: a silent played-audio capture, a
planned capture missing from a run that armed capture, a playback continuity failure or an
underrun above
--max-playback-underruns(default 0) fails the take. The iPhone checker reports continuity failures and underruns without changing the verdict. Outside the measured lanes a take must be seen playing before its playback counts as ended, and smoketest02asserts it (macOS testing). - Partial-utterance ASR consensus is a verifier gap, not product failure or PASS. Warnings do not become clean promotion evidence merely because an operational lane completed.
config/build-output-policy.json owns free-space floors, caches, artifact lifetimes and cleanup.
Use scripts/clean_build_caches.sh --routine --dry-run before bounded cleanup; never assume all
artifacts or caches are disposable. Multi-run evidence is pinned before launch.
Native Xcode/SwiftPM commands are serialized by the host-wide native lock across every checkout and
agent worktree. Generation and heavy analyzers run in the lead session, and no evaluator runs beside
a resident generator: neural evaluators start only after the TTS process exits, and the canonical
Mac mini M6 runs QC v2's models one at a time under build/cache/qc/run.lock (the 8 GB Mac is a
product floor, not an evaluator host). Timing and memory lanes refuse to start on a busy host
(host preflight).
Ordinary CI and scripts/dev.sh check are deterministic. Models, a phone and UI tests never block
commits, pushes or candidate packaging. Push CI, and scripts/dev.sh check when UI-test sources or
project.yml change, compile the XCUITest bundles and never run them
(Daily loop). Public promotion separately requires all
applicable exact-source acceptance lanes. Consult the platform release guide for command-bound
evidence, signing and artifact verification; a development build is not a processed distribution
candidate.
Former recipes, including operator-local ML setup, are in git history (the runbook before
2026-09-06). Dated pins and commands there are not current authority. For current analyzer
configuration use QC v2 (qc.md) and its checked-in config/qc/ contracts.