Repository navigation
Move the Markdown directive tests onto Nullean.Snapshots - #4723
Merged
Merged
Conversation
The ApiExplorer pilot (#4654) used Nullean.Snapshots.Html and .TUnit 0.2.0 in one project. This makes the library available to all unit and integration test projects at 0.4.0 and adds the shared pieces the next migration steps build on: - tests-shared/Elastic.Documentation.Testing: DocsHtml (page section without the fixture H1 and htmx attributes), DiagnosticsSnapshot, NavigationDump, MockFileSystemDump. A scrubber for temp and repo paths is compiled into each test project. - The snap CLI as a local tool; CI uploads received snapshots when a test run fails; the build fails on orphaned snapshots after a unit run. - AGENTS.md describes the workflow. The PR review prompt no longer claims the repo uses xUnit. The five TestDiagnosticsCollector copies stay: they differ in lifecycle (Write vs HandleItem), so merging them is a behaviour change. DiagnosticsSnapshot works on any of them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ShouldBeHtml and ShouldContainHtml (Elastic.Markdown.Tests) and ConvertsToHtml, ConvertsToContainingHtml and the per-file ToHtml and ContainsHtml (Elastic.Authoring.Tests) keep their names and call sites, and now run on BeHtml and ContainHtml with the shared DocsHtml options. That removes the second copy of the AngleSharp.Diffing diff code and the AngleSharp.Diffing package. DiffPlex stays in Authoring for the plain text assertions. One test passed without testing anything. "marks exactly two options as a fork" gave the bare text `is-fork` to ConvertsToContainingHtml. The old normaliser printed only element children, so a text-only fragment became an empty string and every page contains an empty string. It now uses ConvertsToContainingRawHtml, like the test above it. Checked by corrupting one ShouldContainHtml and one ConvertsToHtml expectation: both fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The dropdown title and applies_to tests each asserted a few Contain and NotContain fragments on the rendered HTML. They are now one whole-output snapshot per class (inline for the small dropdown outputs, a file for the nested-content dropdown and the applies_to admonitions, whose popover payload makes them long). The tests became stricter: every old fragment was checked to appear in (or, for NotContain, be absent from) the accepted snapshot before it was accepted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Contain and NotContain fragment tests became whole-output snapshots per class, so the tests are stricter: every old fragment was checked to appear in (or, for NotContain, be absent from) the accepted snapshot before it was accepted. Small outputs are inline snapshots, the rest are files. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Contain and NotContain fragment tests became whole-output snapshots per class, so the tests are stricter: every old fragment was checked to appear in (or, for NotContain, be absent from) the accepted snapshot before it was accepted. Small outputs are inline snapshots, the rest are files. The changelog tests that only asserted NotContain on an otherwise empty output (empty bundles, filtered-out types) now assert the empty output and an empty diagnostics list, which a failing directive no longer satisfies. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ybook and table HTML The Contain and NotContain fragment tests became whole-output snapshots per class, so the tests are stricter: every old fragment was checked to appear in (or, for NotContain, be absent from) the accepted snapshot before it was accepted. Small outputs are inline snapshots, the rest are files. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The remaining single-fragment Contain and NotContain tests of these classes became the class's whole-output snapshot, so the tests are stricter: every old fragment was checked to appear in (or, for NotContain, be absent from) the accepted snapshot before it was accepted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The remaining single-fragment Contain and NotContain tests of these classes became the class's whole-output snapshot, so the tests are stricter: every old fragment was checked to appear in (or, for NotContain, be absent from) the accepted snapshot before it was accepted. The NotContain-only tests of changelog output that ends up empty also assert an empty diagnostics list now. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… sizing and applicability switch HTML The remaining single-fragment Contain and NotContain tests of these classes became the class's whole-output snapshot, so the tests are stricter: every old fragment was checked to appear in (or, for NotContain, be absent from) the accepted snapshot before it was accepted. Mermaid snapshots scrub the diagram file name hash, which tracks the renderer output rather than the markup. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… snapshots The remaining single-fragment Contain and NotContain tests of these classes became the class's whole-output snapshot, so the tests are stricter: every old fragment was checked to appear in (or, for NotContain, be absent from) the accepted snapshot before it was accepted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…k tests that also assert on the block Tests that asserted on the parsed block and on fragments of the rendered HTML keep the block assertions and check the HTML against a snapshot, so they are stricter: every old fragment is in the accepted snapshot, and the NotContain fragments are absent. The button group's Contain(Primary).And.Contain(Secondary) test is covered by the group snapshot. Tests whose HTML expectation is derived from the block (TOC slugs, catalog titles and URLs, element order and counts) keep their assertions. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Collector assertions (empty, counts, severity and message-substring predicates) became a snapshot of the whole diagnostics text (error|warning|hint file:line message), so the tests are stricter: they now also pin the file and line and fail on any additional diagnostic. Every old message fragment and severity was checked to appear in the accepted text, and the empty-diagnostics tests are MatchInlineSnapshot(""). Tests that became identical were merged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Collector assertions (empty, counts, severity and message-substring predicates) became a snapshot of the whole diagnostics text (error|warning|hint file:line message), so the tests are stricter: they now also pin the file and line and fail on any additional diagnostic. Every old message fragment and severity was checked to appear in the accepted text, and the empty-diagnostics tests are MatchInlineSnapshot(""). Tests that became identical were merged. The unsupported directive tests differ per derived class, so they use a file snapshot each.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
![infra-vault-gh-plugin-prod[bot]](https://avatars.githubusercontent.com/u/6764390?s=60&v=4){kind=small}
The repository and temp folder scrubbers replaced only the root, so a Windows run printed {repo}\docs\index.md where the verified text says
{repo}/docs/index.md, and about 160 diagnostics snapshots failed there. The path that follows the root is now written with forward slashes.
The temp scrubber also swallowed everything up to the next whitespace, including the :line after a file name. A path now ends at a colon too,
and the repository root is tried before the temp root because the repository can sit under the temp folder. The doc comment claimed it
scrubbed git SHAs; it does not, and no longer says so.
Tested with a Windows-style root, which also runs on Linux and macOS. Removing the separator replacement makes that test fail.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A test that works in a Guid-named folder in the temp directory printed that name in its diagnostics, so a snapshot of them could not be stable.
The temp scrubber now replaces a run of 32 hex digits in the path with {id}.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The stack was rebased onto fixes for how snapshots scrub paths (Windows separators, random temp folder names). So that the pushed history is not rewritten, this commit has the rebased content and has both the previously pushed tip and the updated base as parents. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The stack was rebased onto fixes for how snapshots scrub paths (Windows separators, random temp folder names). So that the pushed history is not rewritten, this commit has the rebased content and has both the previously pushed tip and the updated base as parents. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ting system CI on Windows failed five of the stacked PRs for reasons a laptop never shows: - DiagnosticsSnapshot now writes paths with forward slashes and drops a leading drive letter, so `C:\docs\toc.yml`, `C:/docs/toc.yml` and `/docs/toc.yml` read the same, and `testing\req.md` inside a message reads `testing/req.md`. - ToSnapshotText(firstLineOnly: true) keeps only the first line of each message, for diagnostics that carry an exception: its stack trace has source line numbers and the exception type differs per platform. - SnapshotPathLengthTests fails when a verified snapshot's path, with the received file's longer folder and the CI checkout root, would pass Windows' 260 characters. git on the runner cannot check such a file out, so the whole job failed before any test ran. Each has a test. Lowering the length limit makes the guard fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Three snapshots held a .NET stack trace: the storybook registry error and the two malformed applies-switch YAML errors. A trace has source line numbers and library frames, so it breaks when StorybookBlock.cs is edited or SharpYaml is upgraded, and on Windows the storybook case throws a different exception altogether (a path with a colon is not a legal path there). They now snapshot the first line of each diagnostic. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
DiagnosticsSnapshot drops a leading drive letter and writes forward slashes, so on Windows D:\a\docs-builder\docs-builder\docs\index.md reaches the
{repo} scrubber as /a/docs-builder/docs-builder/docs/index.md, which it no longer recognised. Every diagnostics snapshot that names a file failed on
the Windows runner. The scrubber now also matches the root without its drive letter. Removing that variant makes the new test fail.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
Docs preview (local build)Handbook preview: https://docs-v3-preview.elastic.dev/elastic/docs-builder/pull/4723/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4719 and #4720. The rest of Elastic.Markdown.Tests is in a sibling PR; they touch different folders.
Moves
tests/Elastic.Markdown.Tests/Directives/onto Nullean.Snapshots.Contain/NotContainfragments.Collector.Diagnostics.ToSnapshotText()); a clean run isMatchInlineSnapshot("").UnsupportedTestsdiffers per derived class, so it uses 9.verified.txtfiles; its two tests became one.index-<hash of the svg>.svg. A per-file scrubber turns the hash into{hash}so a renderer change does not rename every snapshot.2,119 tests, down from 2,196 because merged tests were removed. These tests are stricter; each commit says so.
Found by snapshot testing
NotContainon an output that is empty, so they passed even when the directive failed. They are now an empty HTML snapshot plus an empty diagnostics snapshot (the same forOmitsVersionFromRenderedOutput).ButtonInvalidTypeTestsusedContainSingle, which hid a second diagnostic (a missing/testlink). The snapshot records it.ImageTestsprintsresolved to \Nullean.ScopedFileSystem.ScopedFileInfo``, a type name instead of a path. Not fixed here.ChangelogEntryWithoutInlineContentTests.ErrorIsNotAWarningandMathBlockEmptyTests.EmitsErrorForEmptyContenthad become identical to a sibling test and are removed.TocTreeDirectiveTestspasses"solution", not"toctree", so it duplicatesSolutionDirectiveTests. Not changed.Checked
snap statusclean,snap orphansempty.NotContain).Left as they were
Tests whose expectation is derived from the block (changelog TOC slugs,
RelatedLearningTests,IndexOf/CountOccurrencesorder and count checks).Tab,Image,ImageCarousel,PageCard,VersionandAdmonitionBasetests only assert on blocks and never checked HTML; adding HTML snapshots there would be new coverage, so I did not.Not checked
Linux and Windows.
🤖 Generated with Claude Code