Add Artifacts backend and update dependencies - #3
Open
mizchi wants to merge 17 commits into
Open
Conversation
Independent vrt/ module with 116 tests for visual regression testing combined with accessibility semantic verification. Key components: - 3-track parallel pipeline (Diff Intent / Visual Semantic / A11y Semantic) - Cross-validation matrix (Visual x A11y x Intent) - 2-tier expectations (short-cycle per-commit + long-cycle spec invariants) - Introspect: auto-generate UI spec from a11y trees - Reasoning chains: expectation -> change -> realization verification - Dep graph integration for cost optimization (skip unaffected checks) - 10 fixture scenarios with harness quality tests (40 assertions) - Scoring system (usability / practicality / fixSteps / quality / tokens) - CLI workflow: init / capture / verify / approve / report Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Generate invariants for all landmarks (form, region, search, contentinfo) not just banner/main/nav - Add element-count invariant to detect role changes (e.g. link→button) - Harness now detects form-removed and role-changed scenarios: 10/10 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Export LANDMARK_ROLES, INTERACTIVE_ROLES from a11y-semantic.ts (remove duplicated definitions from introspect.ts) - Export STOP_WORDS, SYNONYMS from expectation.ts (remove inline copies from reasoning.ts, align fuzzy matching) - Remove dead code: unused matchedExpIdx, unmatchedExp, hunkRe, matchResult - Remove unreachable fallback in cross-validation.ts (replace with throw) - Add missing expectedA11yChanges field in vrt-cli.ts fallback intent - Net: -12 lines, consistent matching behavior across modules Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lper - Goal Runner: multi-step goal verification with per-step reasoning chains, cumulative scoring, retry logic, and final invariant checks - 3-step "dark mode" fixture scenario (theme vars → apply → a11y) - Tests: 6 cases including failure detection and scoring - CLI commands: `introspect` and `spec-verify` - `vrt introspect`: generate spec.json from current a11y snapshots - `vrt spec-verify`: verify spec invariants with dep graph skipping - Playwright helper: `nlAssert()` and `nlAssertOnFailure()` - onlyOnFailure pattern for cost optimization - dep graph integration via `nlAssertWithDepCheck()` - Heuristic fallback when LLM not available - LLM-based screenshot + a11y assertion (pluggable provider) - New fixtures: step1-theme-vars, step2-dark-applied, step3-dark-a11y Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Goal Runner (HIGH): - Pass retryCount to loadSnapshot for real agent retry awareness - Cache snapshot to avoid double-call bug (line 145) - Guard empty steps array (crash prevention) - Add retry behavior tests (succeed on 2nd attempt, exhaust retries) Matching logic (HIGH+MED): - Structured match uses best-score instead of greedy findIndex - Fix matchedExpIndices double-tracking in expectation.ts (single pass) - Tighten fuzzy match to matched>=2 (prevent false positives) - inferA11yMode: use word boundaries to avoid "add" matching "additional" Tests & cleanup (MED): - Add playwright-helper heuristic unit tests (6 tests) - Fix role-changed wrong expectation (was identical to correct) - Replace dead sentinel test with scenario coverage assertion - Remove unused HarnessResult interface 134 tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Visual pipeline (7 tests): - Real PNG generation + pixelmatch diff + heatmap output - Color change, layout shift, text change, whiteout detection - Validates the entire visual path with actual images Roundtrip tests (9 tests): - introspect → spec.json → verifySpec for all fixture scenarios - Confirms: nav-removed/form-removed/label-broken/role-changed → FAIL - Confirms: style-only/search-added → PASS (invariants preserved) vrt expect command: - Auto-generates expectation.json from baseline vs snapshot a11y diff - Detects regressions and classifies change types - Eliminates manual expectation authoring 150 tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Visual pipeline (7 tests): real PNG generation → pixelmatch → heatmap → visual-semantic classification. Validates full visual path. - Roundtrip (9 tests): introspect → spec → verify for all fixtures. Long-cycle invariants correctly detect nav/form/label/role changes. - `vrt expect`: auto-generate expectation.json from a11y diff - `vrt demo`: kitty graphics protocol demo with 5 scenarios, showing baseline → snapshot → heatmap → a11y diff → reasoning 150 tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
tsx is not globally installed; use npx tsx to match the agent section. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Demonstrates the full VRT repair cycle with kitty inline images: Phase 1: Establish baseline (10 spec invariants) Phase 2: Detect regression (4 a11y changes, 3 label-missing, 2 spec violations) Phase 3: Generate fix plan from reasoning (3 action items) Phase 4: Apply fix → re-verify (REALIZED, ALL PASS) Run: just vrt-demo-fix Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 3 now calls Anthropic API for AI diagnosis: - Root cause analysis from a11y diff + spec violations - Impact assessment (user-facing problems) - Step-by-step fix plan - Verification guidance Phase 4 adds AI fix evaluation: - Assesses fix adequacy - Identifies remaining concerns - Rates fix quality (1-10) Falls back to heuristic when ANTHROPIC_API_KEY is not set. Usage: ANTHROPIC_API_KEY=sk-ant-... just vrt-demo-fix Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scenario A: Refactor → labels broken → AI diagnosis → fix → PASS - 4 a11y regressions detected, 3 label-missing issues - AI explains root cause and fix plan - Fix verified: all labels restored, spec passes Scenario B: Feature with a11y gaps → AI review → re-implement → PASS - Search form added but missing labels (searchbox, button) - Reasoning: PARTIALLY-REALIZED (landmark found, labels missing) - AI points out specific aria attributes needed - Re-implemented with labels: REALIZED Scenario C: Cascade failure → fix one, break another → 2 AI diagnoses → PASS - Step 1: link→button role change (spec: 2 violations) - Step 2: fix attempt removes form entirely (spec: 3 violations, NEW) - Step 3: AI diagnoses cascade, gives precise fix - Step 4: proper fix restores both links and form New fixture: snapshot-search-no-label.a11y.json Usage: just vrt-demo-multi # heuristic mode ANTHROPIC_API_KEY=... just vrt-demo-multi # with AI reasoning Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New multi-step demo: just vrt-demo-multi (3-scenario)
just vrt-demo-multistep (6-step dashboard)
Dashboard rebuild scenario (6 steps):
Step 1: FAIL — divs instead of ARIA tab roles (AI diagnoses)
Step 2: PASS — proper tablist/tab/tabpanel applied
Step 3: PASS — analytics table added (no headers detected)
Step 4: PASS — column headers and table caption added
Step 5: FAIL — search filter without labels (AI diagnoses)
Step 6: PASS — search labels added, dashboard complete
Improvements:
- Expanded SYNONYMS for tab/tablist/tabpanel/table/column/header
- Removed "added"/"removed"/"changed" from STOP_WORDS (important in VRT)
- Removed type-name synonyms (added→landmark-changed) to prevent false positives
- 6 new dashboard fixtures (step0-step5)
150 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mizchi
marked this pull request as ready for review
July 24, 2026 08:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
just testjust checkjust buildwrangler deploy --dry-run