Repository navigation
Move the API Explorer tests onto Nullean.Snapshots - #4729
Merged
Merged
Conversation
The ApiExplorer pilot (#4654) used Nullean.Snapshots.Html and .TUnit 0.2.0 in one project. This makes the library available to all unit and integration test projects at 0.4.0 and adds the shared pieces the next migration steps build on: - tests-shared/Elastic.Documentation.Testing: DocsHtml (page section without the fixture H1 and htmx attributes), DiagnosticsSnapshot, NavigationDump, MockFileSystemDump. A scrubber for temp and repo paths is compiled into each test project. - The snap CLI as a local tool; CI uploads received snapshots when a test run fails; the build fails on orphaned snapshots after a unit run. - AGENTS.md describes the workflow. The PR review prompt no longer claims the repo uses xUnit. The five TestDiagnosticsCollector copies stay: they differ in lifecycle (Write vs HandleItem), so merging them is a behaviour change. DiagnosticsSnapshot works on any of them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ShouldBeHtml and ShouldContainHtml (Elastic.Markdown.Tests) and ConvertsToHtml, ConvertsToContainingHtml and the per-file ToHtml and ContainsHtml (Elastic.Authoring.Tests) keep their names and call sites, and now run on BeHtml and ContainHtml with the shared DocsHtml options. That removes the second copy of the AngleSharp.Diffing diff code and the AngleSharp.Diffing package. DiffPlex stays in Authoring for the plain text assertions. One test passed without testing anything. "marks exactly two options as a fork" gave the bare text `is-fork` to ConvertsToContainingHtml. The old normaliser printed only element children, so a text-only fragment became an empty string and every page contains an empty string. It now uses ConvertsToContainingRawHtml, like the test above it. Checked by corrupting one ShouldContainHtml and one ConvertsToHtml expectation: both fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Breadcrumbs, catalog cards, code blocks, the design-system nav, the TOC, the copy buttons, parameter section headers and the description pipeline now compare the whole rendered component to a snapshot (inline when a few lines, a file otherwise) instead of a cluster of Contain/NotContain fragments. These tests became stricter: every old fragment appears in the accepted snapshots, and the NotContain checks are covered by the snapshot showing the full markup. The line-number gutter keeps its own Contain, because the HTML snapshot collapses the newlines the gutter is made of. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Each test renders one component and now compares its whole markup to a snapshot instead of asserting fragments. These tests became stricter: the old fragments (and the ordering and count checks on aria-current, border-b and the switcher) are all visible in the accepted snapshots, and the NotContain checks are covered by the complete markup. The reasons the old assertions carried are kept as comments. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The scenario panel, scenario content and response caption tests compare the rendered markup to snapshots, scoped with Remove/Select so the card bodies do not repeat in every file. These tests became stricter: each old Contain fragment is visible in the accepted snapshots. The positional checks (IndexOf ordering) are now the order of the snapshot. Three tests that rendered the same two-scenario panel and asserted different fragments (chips, more menu, heading) are one test with one snapshot, since their snapshots were identical. Structural counts on the generated-media test and the description toggle regex stay, as cheap precise assertions. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Document security, state and deprecated badges, overview row lifecycle badges, the property reference link and the schema type constraint compare rendered markup to snapshots, scoped to the badge strip or the authentication section with Select. These tests became stricter: every old fragment is in the accepted snapshots (the section checks now see the whole authentication panel, so the Bearer, Basic and Api key panels are pinned too). The where-it-sits ordering checks that need the whole page stay. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The remaining responses block tests and the parameter row render tests compare the rendered markup (scoped to the parameter list with Select where the test is about one section) to snapshots. These tests became stricter: all old fragments appear in the accepted snapshots. The button/anchor position checks are now the nesting in the snapshot. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The landing, operation and catalog markdown the generator writes, and the wrapped authored markdown page, are file snapshots instead of about forty Contain/NotContain checks. These tests became stricter: every old fragment is in the accepted files, and the NotContain checks (no applies_to, no DOCTYPE, no merged status line) are covered by the whole text. The file snapshots also pin the section order of the operation page, which the fragments did not. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The catalog page tests (combined entries, groups, declared categories, the title band) compare the page's main container, minus its inline SVG icons, to a snapshot instead of IndexOf/Split counting. These tests became stricter: all old fragments are in the accepted snapshots, and the ordering and tag-count checks are the order and content of the snapshot. The main-container class and a few absence checks of whole-page furniture stay as targeted assertions. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…dies The request and response example tests that compared a body's JSON text to a one-line literal use MatchInlineJsonSnapshot with the same literals. The literals pass unchanged, which shows the conversion keeps the expectation. Key order and number spelling stay significant (MapOrder.Significant and StrictNumbers), so the strictness is the same as the string comparison it replaces. The [Arguments] table keeps its string comparison, since an inline snapshot cannot take a per-row literal. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tructural pages The front matter writer, the HTML sanitizer and the StripHtml helper, and the authentication and servers pages, compare their whole output to an inline snapshot instead of Contain/NotContain fragments. These tests became stricter: the sanitizer tests now pin the exact output, so a tag or attribute that the old NotContain list did not name can no longer slip through. Every old fragment appears in the accepted literals. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…g pages The array response render, the array request summary, the request content type picker (heading and body) and the tag and product landing pages compare rendered markup to snapshots. These tests became stricter: the old fragments are in the accepted snapshots, and the vague checks on the tag pages (Contain 'api-overview', Contain 'Spaces') now pin the whole page content, including the description wrapper and the grouped endpoint list. The array limits and CommonMark checks stay as targeted Contains. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sixteen tests that asserted a few fragments of the generated curl command now pin the whole command with an inline snapshot. These tests became stricter: each old Contain fragment is in the accepted literal and each old NotContain fragment is absent (checked with a script against the accepted literals), and a stray or reordered header or flag now fails the test. The [Arguments] rows keep their fragment checks. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
![infra-vault-gh-plugin-prod[bot]](https://avatars.githubusercontent.com/u/6764390?s=60&v=4){kind=small}
| html.Should().Contain("A distributed search engine."); | ||
| html.Should().NotContain("Choose a product"); | ||
| // The card is the link itself and the heading sits inside it, so there is no stretched overlay to trap. | ||
| html.Should().MatchHtmlSnapshot(new HtmlSnapshotOptions().Select("#main-container").Remove("svg")); |
There was a problem hiding this comment.
Severity: P3 (Low)
This test used to guard the whole catalog page with negative checks: hub-page, data-copy-page, listing-root, listing-group-chips, api-catalog-badge, Choose a product, View docs, hub-card. The new snapshot covers only #main-container. Those markers can now appear in the header, toolbar or footer that the catalog layout renders outside that container, and the test will still pass. The other catalog tests lost the same kind of check, such as applicable- and No APIs match your filter.. So these tests are less strict than before, not more. Either keep the whole-page NotContain checks for markers the snapshot's scope can't see, or snapshot a wider scope.
Generated by Libra. Specialist: libra-complexity. React with 👍 or 👎 to give feedback on this comment.
The repository and temp folder scrubbers replaced only the root, so a Windows run printed {repo}\docs\index.md where the verified text says
{repo}/docs/index.md, and about 160 diagnostics snapshots failed there. The path that follows the root is now written with forward slashes.
The temp scrubber also swallowed everything up to the next whitespace, including the :line after a file name. A path now ends at a colon too,
and the repository root is tried before the temp root because the repository can sit under the temp folder. The doc comment claimed it
scrubbed git SHAs; it does not, and no longer says so.
Tested with a Windows-style root, which also runs on Linux and macOS. Removing the separator replacement makes that test fail.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A test that works in a Guid-named folder in the temp directory printed that name in its diagnostics, so a snapshot of them could not be stable.
The temp scrubber now replaces a run of 32 hex digits in the path with {id}.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The stack was rebased onto fixes for how snapshots scrub paths (Windows separators, random temp folder names). So that the pushed history is not rewritten, this commit has the rebased content and has both the previously pushed tip and the updated base as parents. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The stack was rebased onto fixes for how snapshots scrub paths (Windows separators, random temp folder names). So that the pushed history is not rewritten, this commit has the rebased content and has both the previously pushed tip and the updated base as parents. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ting system CI on Windows failed five of the stacked PRs for reasons a laptop never shows: - DiagnosticsSnapshot now writes paths with forward slashes and drops a leading drive letter, so `C:\docs\toc.yml`, `C:/docs/toc.yml` and `/docs/toc.yml` read the same, and `testing\req.md` inside a message reads `testing/req.md`. - ToSnapshotText(firstLineOnly: true) keeps only the first line of each message, for diagnostics that carry an exception: its stack trace has source line numbers and the exception type differs per platform. - SnapshotPathLengthTests fails when a verified snapshot's path, with the received file's longer folder and the CI checkout root, would pass Windows' 260 characters. git on the runner cannot check such a file out, so the whole job failed before any test ran. Each has a test. Lowering the length limit makes the guard fail. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
DiagnosticsSnapshot drops a leading drive letter and writes forward slashes, so on Windows D:\a\docs-builder\docs-builder\docs\index.md reaches the
{repo} scrubber as /a/docs-builder/docs-builder/docs/index.md, which it no longer recognised. Every diagnostics snapshot that names a file failed on
the Windows runner. The scrubber now also matches the root without its drive letter. Removing that variant makes the new test fail.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
Docs preview (local build)Handbook preview: https://docs-v3-preview.elastic.dev/elastic/docs-builder/pull/4729/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4719 and #4720. Only touches
tests/Elastic.ApiExplorer.Tests/, and is kept as its own PR because this area is under active development. It builds on the pilot (#4654) and the shared helpers (#4660).About 200
Contain/NotContain/IndexOfchecks became whole-output snapshots, in 11 commits by component (breadcrumbs, catalog cards, code blocks, nav, TOC, copy buttons, page actions, union options, examples panel and carousel, badges, document security, property rows, responses block, parameter rows, generated CommonMark and catalog pages, example bodies, front matter and sanitizer, tag pages, curl samples)..verified.*files in all, 25 of them from the pilot.#main-containerwith the svg removed. Other pages are scoped withSelect(...)/Remove(...).These tests are stricter; each commit says so.
Found by snapshot testing
Contain("api-overview")andContain("Spaces"); they now pin the whole page content.## unknownsection header for operations without a tag in the fixture. Not fixed here.Pitfall for anyone adding scoped snapshots
Selecttakes the inner content of the first match only; several matches are not concatenated. A first draft of the card-header test silently covered one of four cards. It now selects the carousel strip and removes the card bodies.Checked
snap statusclean,snap orphansempty.Content-Type, theRequest bodyheading, the example-body"string"placeholder, a response-card class, front mattertype: api, the sanitizer allowingscriptandjavascript:) each failed the converted tests. I did not read every failure diff.Not converted
The
[Arguments]tables, the full-page positional checks (OperationBaseUrlRenderingTests,ApiBreadcrumbLayoutRenderingTests,ApiStyleGuidePreviewTests),ExampleScenarioTests,UnionVariantTests,TopicPageTests,ApiPropertyTreeBuilderTests, the markdown-string checks (ApiMarkdownSplitTests,IntraApiLinkTests,XReqAuthTestsand others), the catalog-split and spec-download tests, and the FakeItEasy tests. The 19 privateRenderAsynchelpers that #4660 meant to replace still exist in several files.The HTML snapshot collapses whitespace inside
<pre>, so the line-number gutter keeps its ownContainfor the newlines.Not checked
Windows and Linux; the new
.verified.txtfiles have only run on macOS.🤖 Generated with Claude Code