Skip to content

Latest commit

 

History

History
153 lines (130 loc) · 11.4 KB

testing-runbook.md

File metadata and controls

153 lines (130 loc) · 11.4 KB
status active
owner release-qa
reviewed 2026-11-15
summary Testing entry point and evidence boundaries; platform guides own execution, the development workflow owns the edit loop, and delivery research stays opt-in.
sourceOfTruth
scripts/check_project_inputs.sh
scripts/macos_test.sh
scripts/ios_device.sh
scripts/ui_test.sh
config/orchestration-contract.json
config/ui-lane-tests.json

Testing runbook

Start with the current checkpoint and primary roadmap, not a dated run command. Source, project settings, contracts, and repository scripts outrank this guide. This page routes work; it does not duplicate platform procedures or establish another gate.

When a lane is replaced or retired, preserve tested user protections and evidence, not legacy implementation details or unsupported scores. A passing inherited suite does not prove correctness; a newer suite is not exempt from independent validation. Replace or consolidate demonstrated weaknesses within their existing owner, then retire the obsolete execution path.

Choose the route

Task Authoritative procedure Boundary
Routine edit Development workflow scripts/dev.sh check, scoped commit on main; push on explicit request; targeted acceptance at checkpoints
macOS UI acceptance macOS testing Existing scripts/ui_test.sh macos lanes at coherent checkpoints
Generic iOS SDK compile Host prerequisite No phone or Simulator; matching Xcode components required
iPhone control / generation / candidate checks iOS testing Physical-device XCUITest; candidate proof remains distinct from diagnostics
Checks XCUITest cannot cover (system settings, permission prompts, installs, audio routing) Checks XCUITest does not cover Every UI-driven check is an XCUITest lane (a new scenario when none covers it); the rest is the maintainer's, by hand
Device deadline / pause / resume / lock Pause and resume Frozen progress stays untracked; validate restoration and exact identities
Model download lifecycle Model delivery Explicit isolated root; preserve canonical data and failed evidence
Benchmark operation / publication Benchmarking procedure Exact-source qualified PASS only; no automatic Git changes; the UI benchmark and perf lanes publish on a PASS and say so when they start, --no-publish keeps a probe run local
Telemetry fields / schema / knobs Telemetry reference Interpretation, not another operator runbook
Delivery bench Benchmarking procedure §4.6 Paired instructed/neutral takes with fail-closed instruction receipts; acoustic effects, not listener-proven delivery
Audio QC QC v2 Models run one at a time after TTS exits; report execution, coverage and qualification separately. Missing, stale or unqualified required evidence is NOT_ASSESSED; frozen thresholds and held-out qualification are required for acceptance
Release / submission programme Release-first plan Implementation, candidate verification, publication approval are separate
Gate or contract changes Development workflow, docs/reference/tooling-and-evidence.md Add a check only for a product invariant; prove rejection as well as success; never assert another file's wording

Antigravity QA shortcuts

Routine edits use scripts/dev.sh; no skill is required. The six vocello-* workspace skills in .agents/plugins/vocello/skills/ select the existing iOS, macOS UI, device diagnostics, benchmark, debugging and read-only release-readiness procedures; they add no gate or authority (tool routing).

Read a finished run

Antigravity reads the artifacts directly, or hands the run directory to the read-only vocello-xcresult-triage subagent, and stops when the deciding evidence is clear:

  1. Read run.json, the required-step ledger and aggregate result. A missing required step is not PASS. Steps the lane runs record startedAt beside completedAt, so a step's duration comes from the ledger itself (steps recorded after the fact with required_step_record, such as a skipped ui-build or result-retention, carry none). The lane's own verdict (verdict.txt, or its test verdict: / <platform> <lane> PASS line) decides, never a wrapper's or pipe's exit code; when a lane runs in the background, capture its exit status directly.
  2. Inspect the failed test and xcresult summary, then nearby log context. Use the existing bootstrap and external-interruption classifiers when applicable; zero launched cases can be infrastructure.
  3. Check the crash delta and attachment manifest for relevant control observations.
  4. Review screenshots/index.json (readable names, test, time, failure flag): open every failure frame and the key states for clipping, overlap, a wrong state or an unexpected dialog. Report-only; each finding is classified as a release blocker or a backlog line.
  5. Report run id, verdict, deciding step and artifact paths. The verdict classes are PASS, product failure, infrastructure failure, external interruption, restoration gap and incomplete evidence; this list is the one the triage agent and the debug skill use. Never turn a failed run into a pass or silently rerun it.

The SwiftPM runtime lane keeps XCTest stdout in runtime.log and diagnostic stderr in runtime.stderr.log, without merging concurrent writes. macOS test verdicts record native statuses and separate core_summary / runtime_summary collection statuses. A complete nonempty All/Selected tests trailer is required for these summaries; native success with incomplete evidence fails the lane. Inspect both streams before diagnosing a runtime failure. Historical failed or inconsistent summaries remain unchanged.

UI runs retain ui-finalization.json and its log separately from native test status. After an attempted test, failure or interruption still triggers available crash-delta, strict summary and attachment collection, followed by owned-process cleanup. Each collector has a bounded budget; timeout, missing evidence or cleanup failure prevents PASS. The first native failure/signal remains the exit status. Signals during finalization are replayed after remaining collectors, owned-process cleanup and receipt writing; an earlier native failure retains precedence over a later collection signal. Read the collection receipt as well as the required-step ledger; neither a clean receipt nor passed test cases alone establishes successful run completion.

A full macOS smoke, perf or benchmark run also owes every test config/ui-lane-tests.json lists for its lane. A listed test that is missing from the results or was skipped ends the run without a PASS (xctest_summary.py --expect-lane exits 4), and test-results.json names it under expected. A --scenario run is one journey by design and checks nothing. The file changes in the same commit as any renamed, added or deleted journey (macOS testing).

A benchmark or perf run started with --no-publish holds publication-skipped.json in its run directory instead of a tracked record; its verdict and artifacts are those of a published run.

Model readiness

Generation UI tests require visible Settings readiness, Generate enabled, and the exact required Clone reference. macOS fixture repair uses scripts/macos_test.sh models ensure or install only after visible readiness fails. iOS model repair uses visible Settings controls; no headless inventory substitutes for UI readiness. The isolated model-download lane is explicitly selected, never an implicit smoke/benchmark bootstrap. Restart the affected lane after repair.

Evidence and verdicts

  • XCTest owns visible assertions, activities, screenshots and attachments. The runner additionally owns source/app/device identity, crash deltas, required collection, validators and restoration.
  • config/orchestration-contract.json defines required steps. Missing, interrupted, duplicate, failed or unknown required evidence prevents PASS, even when every XCTest case passed.
  • Preserve the first failed result. No automatic retries, seed substitution or merging a later pass into the failed bundle. The platform guide defines typed infrastructure classification.
  • Resume uses the existing source/build/device/plan-bound validator, never a guessed row count. A zero-observation shard provides no cursor; changed source cannot reuse old acceptance.
  • Generated audio, transcripts, database copies, raw telemetry, screenshots, traces and result bundles remain untracked. Compact published benchmark records must be privacy-safe and qualified.
  • On the Mac UI benchmark, playback is part of the verdict: a silent played-audio capture, a planned capture missing from a run that armed capture, a playback continuity failure or an underrun above --max-playback-underruns (default 0) fails the take. The iPhone checker reports continuity failures and underruns without changing the verdict. Outside the measured lanes a take must be seen playing before its playback counts as ended, and smoke test02 asserts it (macOS testing).
  • Partial-utterance ASR consensus is a verifier gap, not product failure or PASS. Warnings do not become clean promotion evidence merely because an operational lane completed.

Storage and concurrency

config/build-output-policy.json owns free-space floors, caches, artifact lifetimes and cleanup. Use scripts/clean_build_caches.sh --routine --dry-run before bounded cleanup; never assume all artifacts or caches are disposable. Multi-run evidence is pinned before launch. Native Xcode/SwiftPM commands are serialized by the host-wide native lock across every checkout and agent worktree. Generation and heavy analyzers run in the lead session, and no evaluator runs beside a resident generator: neural evaluators start only after the TTS process exits, and the canonical Mac mini M6 runs QC v2's models one at a time under build/cache/qc/run.lock (the 8 GB Mac is a product floor, not an evaluator host). Timing and memory lanes refuse to start on a busy host (host preflight).

CI and release

Ordinary CI and scripts/dev.sh check are deterministic. Models, a phone and UI tests never block commits, pushes or candidate packaging. Push CI, and scripts/dev.sh check when UI-test sources or project.yml change, compile the XCUITest bundles and never run them (Daily loop). Public promotion separately requires all applicable exact-source acceptance lanes. Consult the platform release guide for command-bound evidence, signing and artifact verification; a development build is not a processed distribution candidate.

Historical procedures

Former recipes, including operator-local ML setup, are in git history (the runbook before 2026-09-06). Dated pins and commands there are not current authority. For current analyzer configuration use QC v2 (qc.md) and its checked-in config/qc/ contracts.