Skip to content

test(parsers): check linear growth instead of fixed wall-clock ceilings in the timing tests - #787

Merged
aliasunder merged 10 commits into
mainfrom
test/parser-timing-growth-checks
Nov 13, 2026
Merged

aliasunder merged 10 commits into
mainfrom
test/parser-timing-growth-checks

Conversation

@aliasunder

@aliasunder aliasunder commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

What does this PR do?

The parser timing tests (tasks-linear, tasks-clear-linear, headings-linear, tasks-tag-dedup) fail CI at random with no change to the parsers. The timed child processes asserted two things about their medians:

  • every median under a fixed millisecond ceiling (250 to 2,000 ms), and
  • the median at the largest size within a bounded multiple of the one at the smallest (the growth check).

A slow runner breaks the first with no regression in the code, and a fixed per-test child deadline (10 to 30 s) breaks the same way. Failures seen on CI: status stamp clear exceeded ceiling 259, 559, 1176 (while that ceiling was 1,000 ms, before #755 raised it to 2,000 ms) and packed hashtag edits exceeded ceiling 64, 226, 1025, and the status-stamp child killed at its 15 s deadline.

This PR keeps the growth check as the speed test:

  • The fixed ceilings go wherever a growth check already exists. The growth check compares a run against itself, so a slower runner scales every median together.
  • The packed-hashtag test had only a ceiling. It gains a growth check scaled by the line's length, because its size sets both the hashtag count and each hashtag's length.
  • Every timing child in these four files shares one 60 s deadline (__tests__/helpers/linear-timing.ts), with the vitest timeout 10 s above it so a child that finishes inside its deadline never fails as a vitest timeout. The slowest child takes about 7 s on a fast laptop.
  • The 110,000-case Dataview corpus test gets an explicit 30 s timeout like its sibling corpus test; the 5 s default ran out under local load.
  • LINEAR_GROWTH_ALLOWANCE, copied in four files, moves to the same helper. The two files that passed a bare 10 as the growth check's noise floor name it, with the same value.
  • The heading and field-grammar timing tests assert their whole output, and the status-stamp test prints its medians like the other eight.

Type of change

  • New feature (a tool, a prompt, a setting, a parameter, or a change to what a tool does)
  • Bug fix
  • Refactoring
  • Documentation
  • CI / workflow change
  • Infrastructure (SST, Docker)
  • Other: tests only

Verification

  • The four files pass locally: 28 tests. On a passing CI run tasks-clear-linear took 27.6 s against 11.5 s on a fast laptop, which is the margin the old fixed deadlines lacked.
  • Mutation check: with tasks.ts and headings.ts from before fix(parsers): keep task, heading, link, frontmatter and canvas parsing linear #753 (995118e^), all nine timing tests fail.
    • The dense-edits test fails its growth check (126 → 2,466 ms over a 5× size range).
    • The two field-clear tests crash in the old recursive removal (a stack overflow, or running out of memory).
    • Five fail at the 60 s deadline.
    • The field-grammar test compiles a table the old file lacks, so it was run with the old recursive removal swapped in instead; its growth check fails.

Checklist

  • The pre-commit hook passed: typecheck, knip, and ESLint, Prettier, markdownlint and ShellCheck on the staged files
  • npm test passes (CI)
  • npm run build succeeds (CI)
  • If this adds a feature, the description links the feature request it came from
  • New MCP tools follow the naming and description conventions in AGENTS.md
  • README or ARCHITECTURE.md updated (if user-facing behavior changed)

🤖 Generated with Claude Code

… the parser timing tests

The timing tests asserted medians under fixed millisecond ceilings next to
a growth-ratio check. A slow CI runner exceeds a fixed ceiling and the
fixed child deadlines with no regression in the code. The growth check
alone catches the superlinear behaviour these tests guard, so the
ceilings go, the packed-hashtag test gains a growth check scaled by line
length, and every timing child shares one deadline with room for slow
runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@umm-actually

umm-actually Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

umm-actually re-reviewed at 90a2246

1 new finding(s) posted (8 tracked finding(s) across all runs).


umm-actually · z-ai/glm-5.3-flash

aliasunder and others added 5 commits November 12, 2026 19:49
…d field-grammar timing tests

The heading timing test checked only the variant names, and the field-grammar
timing test matched each fixture name with expect.any(String). Both outputs are
deterministic apart from the medians, so each test now matches the whole
result: the variant, sizes and three medians per heading series, and the
24 fixture names in the order the child times them.

Ship-Check: test-audit · claude-opus-5-5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ld's deadline

Vitest cannot interrupt a test blocked in execFileSync, and a killed child's own error is what vitest reports whatever the test timeout. The margin keeps a child that finishes inside its deadline from failing as a vitest timeout, which vitest checks once a synchronous test returns.

Ship-Check: bug-check · claude-opus-5-5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he status-stamp timings

The heading and tag-deduplication timing tests passed a bare 10 as the
growth check's noise floor; it is now a named constant like the other two
files' floor, with the same value. The status-stamp test was the only
timing test that never printed its medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t their deadline

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@umm-actually

umm-actually Bot commented Nov 13, 2026

Copy link
Copy Markdown

Give the corpus regression test an explicit timeout like its siblings
Low severity · tests · medium confidence

src/vault-mcp/obsidian-markdown/__tests__/headings-linear.test.ts:68 — beyond the diff's line ranges, in code the changes touch or depend on.

This PR gives the 110,000-case Dataview corpus test an explicit 30 s timeout because 'the 5 s default ran out under local load', but the 10,000-case fixed-seed traversal corpus test in this same file still rides vitest's 5 s default. Its per-case work (a legacy-oracle parse, the current parse, a deep equality check, and per-task re-extraction over up to 24 lines) is far heavier than the regex execs the raised test runs, so a loaded runner can push it past 5 s and fail it spuriously with no parser change — the failure class this PR removes from the timing children.

Failure scenario: A CI runner under load — the condition this PR exists for, where the old children 'fail CI at random with no change to the parsers' — runs the 10,000-case corpus test slowly enough to exceed vitest's 5 s default; the test red-balls with a timeout although the parsers are correct, the same '5 s default ran out under local load' failure the PR fixes for the sibling Dataview corpus test, and a local deep run with PARSER_CORPUS_CASES raised hits it sooner.

Suggested fix
Add an explicit timeout matching the sibling corpus tests (e.g. close the test with `}, 30_000)`), or hoist a shared CORPUS_TEST_TIMEOUT_MS into helpers/linear-timing.ts beside the other timing constants.

umm-actually · z-ai/glm-5.3-flash

…and correct the floor comments' arithmetic

The four fixed-seed corpus tests that still relied on vitest's 5 s default
get the same 30 s as their siblings. The 10 ms floor comments now state
each file's real margin: the heading traversal's quadratic would pass a
25 ms floor, while the tag-dedup fixture would fail it by only about 10%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aliasunder

Copy link
Copy Markdown
Owner Author

Issue comment 6840872688 (umm-actually, corpus test timeout): fixed in 32339fc. Every fixed-seed corpus test in the four timing files now carries an explicit 30 s timeout, the heading ranges test it named and the trailing-tag, emoji-recurrence and tag-removal-order corpora with it.


🔍 ship-check · pr-monitor · claude-opus-5-5

aliasunder and others added 2 commits November 12, 2026 20:47
The 10 ms and 14 ms smallest medians hold on an idle fast machine; a
loaded or slower runner measures several times more, and there the
smallest median clears the 25 ms floor anyway. The arithmetic is
unchanged.

Ship-Check: test-audit · claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ut as its siblings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… its two facts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ship-Check: code-quality · claude-opus-5-5
@aliasunder
aliasunder merged commit 19aba9b into main Nov 13, 2026
18 checks passed
@aliasunder
aliasunder deleted the test/parser-timing-growth-checks branch November 13, 2026 02:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant