Move the rest of the Markdown tests onto Nullean.Snapshots by Mpdreamz · Pull Request #4724 · elastic/docs-builder

Mpdreamz · 2026-11-05T12:50:01Z

Stacked on #4719 and #4720. The Directives/ folder is in a sibling PR.

Moves the rest of Elastic.Markdown.Tests (Inline/, CodeBlocks/, FileInclusion/, AppliesTo/, Exporters/, DocSet/, Search/, SettingsInclusion/, TaskList/, FrontMatter/ and the root files) onto Nullean.Snapshots.

The 17 ShouldBeHtml/ShouldContainHtml sites become inline HTML snapshots.
About 190 inline HTML snapshots and 12 files (anything over 25 lines is a file).
About 110 inline diagnostics snapshots.
10 Should().Be("""…""") JSON sites became inline JSON snapshots and passed with their literals unchanged. 15 more tests became whole-JSON snapshots.
4 OKF index tests became inline text snapshots.
About 30 tests were merged (2,196 → 2,166 across the project).

These tests are stricter; each commit says so.

Found by snapshot testing

Probable product bug: SubstitutionTest.MutationOperatorsWorkInLinks asserted Contain("ELASTICSEARCH 9.0"), which the literal text [Link text with mutation](ELASTICSEARCH 9.0) satisfied. The mutation-operator link text does not render as a link. The old test hid that. Not fixed here.
DocumentationDocumentSerializationTests.SerializeDocumentWithNullAppliesToOmitsField asserted true.Should().BeTrue() in one branch, so it passed whether the field was omitted or null. The snapshot pins that it is omitted.
The CallOutTests literal had a stray </div> and indented lines inside <pre>; the old structural comparison tolerated that and the exact inline comparison does not. It now holds the real output.
LinkWithUnresolvedInterpolationError checked a truncated message (he url contains…). The full message is pinned.
Many fragments were weak (Contain("7.17.0"), Contain("https://")).

Caveats

HTML snapshots collapse whitespace inside <pre>, so a change that only alters newlines in code would not fail these tests. The old exact-text checks would have. I will raise this with the library.
The 12 DocumentationDocument JSON snapshots each repeat every field, so adding a field means re-accepting 12 files.

Checked

2,166 tests pass, three runs in a row. snap status clean, snap orphans empty.
Mutations of the link rel attribute, the heading wrapper class, the code-callout class, the settings class, the AppliesTo JSON sub_type key, the OKF heading, a front-matter diagnostic and a tracking message each failed with readable diffs.
A script checked every old Contain fragment against the accepted snapshots, and flagged a deliberately broken one.

Left as they were

Pure-function and [Arguments] tests, the CLI reference tests, PageTitleTests, StyleGuidePreviewTests, and tests whose diagnostics embed stack traces with line numbers.

Not checked

Other operating systems.

🤖 Generated with Claude Code

The ApiExplorer pilot (#4654) used Nullean.Snapshots.Html and .TUnit 0.2.0 in one project. This makes the library available to all unit and integration test projects at 0.4.0 and adds the shared pieces the next migration steps build on: - tests-shared/Elastic.Documentation.Testing: DocsHtml (page section without the fixture H1 and htmx attributes), DiagnosticsSnapshot, NavigationDump, MockFileSystemDump. A scrubber for temp and repo paths is compiled into each test project. - The snap CLI as a local tool; CI uploads received snapshots when a test run fails; the build fails on orphaned snapshots after a unit run. - AGENTS.md describes the workflow. The PR review prompt no longer claims the repo uses xUnit. The five TestDiagnosticsCollector copies stay: they differ in lifecycle (Write vs HandleItem), so merging them is a behaviour change. DiagnosticsSnapshot works on any of them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ShouldBeHtml and ShouldContainHtml (Elastic.Markdown.Tests) and ConvertsToHtml, ConvertsToContainingHtml and the per-file ToHtml and ContainsHtml (Elastic.Authoring.Tests) keep their names and call sites, and now run on BeHtml and ContainHtml with the shared DocsHtml options. That removes the second copy of the AngleSharp.Diffing diff code and the AngleSharp.Diffing package. DiffPlex stays in Authoring for the plain text assertions. One test passed without testing anything. "marks exactly two options as a fork" gave the bare text `is-fork` to ConvertsToContainingHtml. The old normaliser printed only element children, so a text-only fragment became an empty string and every page contains an empty string. It now uses ConvertsToContainingRawHtml, like the test above it. Checked by corrupting one ShouldContainHtml and one ConvertsToHtml expectation: both fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…nHtml were used The 17 ShouldBeHtml/ShouldContainHtml call sites outside Directives become MatchInlineHtmlSnapshot with DocsHtml.Body(). The ShouldBeHtml sites kept their literals unchanged and pass as they are. The ShouldContainHtml sites compared a fragment of the page; the snapshot is the whole rendered output, so those tests became stricter. Every old fragment appears in the accepted snapshot (the htmx attributes are removed by DocsHtml.Body(), as ShouldContainHtml already did). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…agments Output of the Inline fixtures now lives in inline snapshots. Where a test (or a cluster of tests of one class) asserted Contain/NotContain fragments on the rendered HTML, it is one MatchInlineHtmlSnapshot of the whole output; where it asserted a count or message on Collector.Diagnostics, it is a MatchInlineSnapshot of the diagnostics text ("" when the run is clean). These tests became stricter: they now fail on any unexpected change in the output, not only on the fragments they named. Before accepting, every old Contain fragment was checked to appear in the accepted snapshot (and no NotContain fragment to appear); the htmx attributes on links are removed by DocsHtml.Body(), as ShouldContainHtml already did. Tests of one class that only asserted fragments of the same output are merged into one test; their explanatory comments moved onto it. Kept as they were: the footnote index/regex-count tests, the raw-output nested anchor check (the HTML parser would split nested anchors in a snapshot), and BeNullOrWhiteSpace. Found: MutationOperatorsWorkInLinks asserted Contain("ELASTICSEARCH 9.0"), which a literal "[text](ELASTICSEARCH 9.0)" in the output satisfied, so the test passed while the link did not render as a link. The snapshot now shows it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…snapshots Same expectations as before (the accepted inline snapshots, moved to .snapshots/Inline/*.verified.html) so the test file stays readable. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Contain/NotContain fragments on the rendered HTML become one MatchInlineHtmlSnapshot per test and diagnostics counts/prefix checks become MatchInlineSnapshot of the diagnostics text, so these tests became stricter. Every old Contain fragment was checked to appear in the accepted snapshot. Caveat: the HTML snapshot collapses whitespace inside pre, so a change in the line structure of a code block alone is not caught by it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…gments Fragment checks on the included output become HTML snapshots (files for the larger ones) and the diagnostics assertions become diagnostics snapshots. These tests became stricter; every old Contain fragment appears in the accepted snapshot. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ttings The settings fixtures rendered several hundred lines each, so those snapshots are files. The tests of one fixture that asserted fragments of the same output are one snapshot test now; they became stricter. Every old Contain fragment appears in the accepted snapshot. RandomFileEmitsAnError keeps its targeted assertions: the diagnostic message embeds a YAML parser stack trace, which a snapshot should not pin. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

The Assembler and Codex tests are about the htmx attributes and the link target on cross links, so their snapshots keep those attributes (no DocsHtml.Body() normalisation). Fragment checks became whole-output snapshots, so the tests became stricter. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…output The nine Should().Be("""...""") JSON comparisons (AppliesTo serialisation, RepositoryLinks) become MatchInlineJsonSnapshot with their literals unchanged, and they pass as they were. SerializeAllLifecycles, SerializeComplex and SerializeValidatesJsonStructure checked fragments or walked the parsed JSON; they are whole-output snapshots now, so they became stricter (every old fragment and property value appears in the snapshots). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

The applies_to tests walked the parsed JSON property by property; they are one JSON snapshot of the whole serialised document each, so they became stricter. Every property the old tests asserted appears in the snapshots. SerializeDocumentWithNullAppliesToOmitsField was vacuous: its else branch asserted true.Should().BeTrue(), so it passed whether the field was omitted or null. The snapshot pins that the field is omitted. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

The four RenderIndexContent tests checked Contain/StartWith fragments of a few lines of text; they are inline text snapshots now and became stricter. Every old fragment appears in the snapshots. The pure-function tests and the YAML round-trip tests stay as they are. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Count and message-substring assertions on the collector become snapshots of the diagnostics text, so they pin the severity, location and whole message and became stricter. The four products-suggestion tests keep their old assertions: their diagnostic embeds an exception stack trace with source line numbers, which a snapshot should not pin. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…idation errors The three tests that checked an error by substring now snapshot the diagnostics text (the temp path is scrubbed to {temp}), so they pin the severity and the whole message and became stricter. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

infra-vault-gh-plugin-prod · 2026-11-05T12:54:31Z

 	[Test]
-	public void Render() => Html.Should().Contain("<span class=\"math\">a &lt; b &amp; b &gt; c</span>");
+	public void Render() =>
+		Html.Should().MatchInlineHtmlSnapshot(


Severity: P3 (Low)

Keep an escaping-sensitive check in MathRoleEscapesContentTests. The HTML snapshot parses entities, so the unescaped output a < b & b > c produces the same text node as the expected escaped output. Replacing WriteEscape with Write would therefore stop failing this test. Retain a raw-output assertion/text snapshot, or use tag-like input that makes missing escaping change the DOM.

_{Generated by Libra. React with 👍 or 👎 to give feedback on this comment.}

infra-vault-gh-plugin-prod · 2026-11-05T12:54:31Z

 	public void EmitsWarning() =>
-		Collector.Diagnostics.Should().Contain(d => d.Severity == Severity.Warning && d.Message.Contains("invalid-deployment"));
+		Collector
+			.Diagnostics
+			.ToSnapshotText()
+			.Should()
+			.MatchInlineSnapshot(
+				"""
+		warning {repo}/docs/index.md:3 Unknown deployment filter 'invalid-deployment'. Valid values are: ech, ece, eck, self.
+		error {repo}/docs/testing/kibana-general-settings.yml:2 Substitution key {ecloud} is undefined


Severity: P3 (Low)

EmitsWarning is meant to check the unknown invalid-deployment filter warning. The new snapshot also pins about 140 unrelated Substitution key {kib}/{es}/{ecloud} is undefined errors from the shared kibana-general-settings.yml fixture. Any edit to that fixture, or to how substitutions are reported, will break this test even though the behaviour it is named for still works. The test also no longer reads as a check of the warning. Either define the missing substitutions in the test docset, or limit the snapshot to warning-severity diagnostics.

_{Generated by Libra. Specialist: libra-complexity. React with 👍 or 👎 to give feedback on this comment.}

infra-vault-gh-plugin-prod · 2026-11-05T12:54:32Z

+	public void RendersUncheckedCheckbox() =>
+		Html.Should().MatchInlineHtmlSnapshot(
+			"""
+			<ul class="contains-task-list">
+				<li class="task-list-item">
+					<input disabled="disabled" type="checkbox">
+					A pending task</li>
+				<li class="task-list-item">
+					<input disabled="disabled" type="checkbox" checked="checked">
+					A completed task</li>
+			</ul>
+			""",
+			DocsHtml.Body()
+		);

 	[Test]
-	public void RendersUncheckedCheckbox() => Html.ShouldContainHtml("""<input disabled="disabled" type="checkbox">""");
-
-	[Test]
-	public void RendersCheckedCheckbox() => Html.ShouldContainHtml("""<input disabled="disabled" type="checkbox" checked="checked">""");
+	public void RendersCheckedCheckbox() =>
+		Html.Should().MatchInlineHtmlSnapshot(
+			"""
+			<ul class="contains-task-list">
+				<li class="task-list-item">
+					<input disabled="disabled" type="checkbox">
+					A pending task</li>
+				<li class="task-list-item">
+					<input disabled="disabled" type="checkbox" checked="checked">
+					A completed task</li>
+			</ul>
+			""",
+			DocsHtml.Body()
+		);


Severity: P3 (Low)

GeneratesHtml, RendersUncheckedCheckbox and RendersCheckedCheckbox now all assert the same inline HTML snapshot. The two extra tests no longer check anything separately, and any markup change has to be re-accepted in three places. Merge them into GeneratesHtml, as the PR did for other classes.

_{Generated by Libra. Specialist: libra-complexity. React with 👍 or 👎 to give feedback on this comment.}

The repository and temp folder scrubbers replaced only the root, so a Windows run printed {repo}\docs\index.md where the verified text says {repo}/docs/index.md, and about 160 diagnostics snapshots failed there. The path that follows the root is now written with forward slashes. The temp scrubber also swallowed everything up to the next whitespace, including the :line after a file name. A path now ends at a colon too, and the repository root is tried before the temp root because the repository can sit under the temp folder. The doc comment claimed it scrubbed git SHAs; it does not, and no longer says so. Tested with a Windows-style root, which also runs on Linux and macOS. Removing the separator replacement makes that test fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

A test that works in a Guid-named folder in the temp directory printed that name in its diagnostics, so a snapshot of them could not be stable. The temp scrubber now replaces a run of 32 hex digits in the path with {id}. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

The stack was rebased onto fixes for how snapshots scrub paths (Windows separators, random temp folder names). So that the pushed history is not rewritten, this commit has the rebased content and has both the previously pushed tip and the updated base as parents. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ting system CI on Windows failed five of the stacked PRs for reasons a laptop never shows: - DiagnosticsSnapshot now writes paths with forward slashes and drops a leading drive letter, so `C:\docs\toc.yml`, `C:/docs/toc.yml` and `/docs/toc.yml` read the same, and `testing\req.md` inside a message reads `testing/req.md`. - ToSnapshotText(firstLineOnly: true) keeps only the first line of each message, for diagnostics that carry an exception: its stack trace has source line numbers and the exception type differs per platform. - SnapshotPathLengthTests fails when a verified snapshot's path, with the received file's longer folder and the CI checkout root, would pass Windows' 260 characters. git on the runner cannot check such a file out, so the whole job failed before any test ran. Each has a test. Lowering the length limit makes the guard fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…-rest

DiagnosticsSnapshot drops a leading drive letter and writes forward slashes, so on Windows D:\a\docs-builder\docs-builder\docs\index.md reaches the {repo} scrubber as /a/docs-builder/docs-builder/docs/index.md, which it no longer recognised. Every diagnostics snapshot that names a file failed on the Windows runner. The scrubber now also matches the root without its drive letter. Removing that variant makes the new test fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…-rest

github-actions · 2026-11-05T14:30:57Z

Docs preview (local build)

Handbook preview: https://docs-v3-preview.elastic.dev/elastic/docs-builder/pull/4724/

Mpdreamz and others added 14 commits November 5, 2026 13:42

Markdown tests (Inline): footnote outputs over 25 lines move to file …

ab17567

…snapshots Same expectations as before (the accepted inline snapshots, moved to .snapshots/Inline/*.verified.html) so the test file stays readable. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Mpdreamz requested a review from a team as a code owner November 5, 2026 12:50

Mpdreamz requested a review from reakaleek November 5, 2026 12:50

Mpdreamz added the chore label Nov 5, 2026

infra-vault-gh-plugin-prod Bot approved these changes Nov 5, 2026

View reviewed changes

Mpdreamz and others added 10 commits November 5, 2026 14:09

Merge branch 'test/snapshots-infra' into test/snapshots-html-normalisers

fe5a8bf

Merge branch 'test/snapshots-html-normalisers' into test/snapshots-md…

c413d42

…-rest

Merge branch 'test/snapshots-infra' into test/snapshots-html-normalisers

63a4b4a

Merge branch 'test/snapshots-html-normalisers' into test/snapshots-md…

8fe2dab

…-rest

Base automatically changed from test/snapshots-html-normalisers to main November 5, 2026 14:22

Merge remote-tracking branch 'origin/main' into test/snapshots-md-rest

26d93f7

Mpdreamz merged commit cf20363 into main Nov 5, 2026
36 checks passed

Mpdreamz deleted the test/snapshots-md-rest branch November 5, 2026 14:43

Navigation Menu

Repository navigation

Move the rest of the Markdown tests onto Nullean.Snapshots - #4724

Move the rest of the Markdown tests onto Nullean.Snapshots#4724
Mpdreamz merged 25 commits into
mainfrom
test/snapshots-md-rest

Mpdreamz commented Nov 5, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot Nov 5, 2026

Uh oh!

Uh oh!

infra-vault-gh-plugin-prod Bot Nov 5, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot Nov 5, 2026

Uh oh!

github-actions Bot commented Nov 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Mpdreamz commented Nov 5, 2026

Found by snapshot testing

Caveats

Checked

Left as they were

Not checked

Uh oh!

infra-vault-gh-plugin-prod Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

infra-vault-gh-plugin-prod Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

infra-vault-gh-plugin-prod Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 5, 2026

Docs preview (local build)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant