Pre-compress the assembler preview S3 upload by Mpdreamz · Pull Request #4210 · elastic/docs-builder

Mpdreamz · 2026-09-22T16:11:26Z

Pre-compresses the assembler preview S3 upload so the wire cost drops from ~2.0 GiB to ~120 MB per preview run.

Affects: Deploys & previews, Automation

Prompt summary: The assembler preview upload in `assembler-preview.yml` sent raw uncompressed bytes to S3. The ask was to gzip the output tree in place before the sync, matching what PR #4206 does for the local docs preview.

Why

Each API operation page is ~420 KB uncompressed (89% sidebar nav). At 4,029 pages the raw upload is ~1.7 GiB. CloudFront already serves with `content-encoding: gzip` on the fly, but the origin upload still pays the full uncompressed byte count. Storing pre-compressed objects lets CloudFront forward them untouched at ~120 MB.

What

Pre-compressed upload with zip exclusion

All non-zip files are gzip-compressed in place using `find ... ! -name "*.zip"` piped to parallel `gzip`. Each file is compressed to a `mktemp`-generated temp path then atomically renamed, avoiding collisions when both `foo` and `foo.gz` exist in the tree. Zip bundles (`llm.zip`, `okf.zip`) are excluded from compression and synced in a separate pass without `content-encoding: gzip` so clients receive raw zip bytes.

Verify

Trigger the assembler preview workflow on a PR. The `Upload assembled docs to S3` step should complete in seconds rather than minutes. Verify a preview page renders correctly in a browser and that downloading `llm.zip` or `okf.zip` from the preview URL produces a valid zip archive.

Stack: 5 of 5, on top of #4205.

🤖 Generated with Claude Code

github-actions

Approved — no blocking issues found.

What is this? | From workflow: PR Review

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

github-actions

Requesting changes: the new compression step introduces a real collision risk in temp-file naming that can corrupt or fail uploads and should be fixed before merge.

What is this? | From workflow: PR Review

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

github-actions · 2026-09-24T11:31:53Z

-          aws s3 sync .artifacts/assembly/${ASSEMBLER_PREVIEW_PATH_PREFIX} "s3://elastic-docs-v3-website-preview/${ASSEMBLER_PREVIEW_PATH_PREFIX}" --delete --no-follow-symlinks
+          # Pre-compress the output tree so aws s3 sync ships ~120 MB instead of ~2 GiB.
+          find ".artifacts/assembly/${ASSEMBLER_PREVIEW_PATH_PREFIX}" -type f -print0 \
+            | xargs -0 -P "$(nproc)" -I{} sh -c 'gzip -9 -n -c "$1" > "$1.gz" && mv -f "$1.gz" "$1"' _ {}


[MEDIUM] Temporary "$1.gz" target can collide with real files

This in-place compression uses "$1.gz" as a temp path before moving it back to "$1". That temp name is not unique and can be a real file in the tree (for example when both foo and foo.gz exist), which can overwrite/delete the real foo.gz or make later workers fail with No such file depending on execution order.

Suggested change

| xargs -0 -P "$(nproc)" -I{} sh -c 'gzip -9 -n -c "$1" > "$1.gz" && mv -f "$1.gz" "$1"' _ {}

find ".artifacts/assembly/${ASSEMBLER_PREVIEW_PATH_PREFIX}" -type f -print0 \

| xargs -0 -P "$(nproc)" -I{} sh -c '

tmp="$(mktemp "${1}.tmp.XXXXXX")" || exit 1

gzip -9 -n -c "$1" > "$tmp" && mv -f "$tmp" "$1"

' _ {}

github-actions

Approved — no blocking issues found.

What is this? | From workflow: PR Review

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

…4223) * Downgrade "Invalid host" OpenAPI diagnostic to a warning Swagger 2.0 specs that use template placeholders (e.g. `{{hostname}}`) produce an "Invalid host" error from Microsoft.OpenApi because the value is not a valid URI host. The document itself still parses correctly. Treating this as a hard error blocks generation of specs like the ECE API that intentionally use placeholder values. Downgrading to a warning lets generation proceed while still surfacing the diagnostic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add test and ECE fixture for template-host warning behaviour Covers the case where a Swagger 2.0 spec uses `{{hostname}}` as the `host` value. The fixture is stripped down from the real ECE spec to a single `/account` path so it stays minimal. The test asserts that `OpenApiReader` produces zero errors and at least one warning when it encounters the placeholder — confirming the downgrade added in the previous commit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…4224) The "Invalid host" message for {{hostname}}-style placeholders in Swagger 2.0 specs is informational noise, not something a build author needs to act on. Emitting it as a hint rather than a warning keeps the warning count meaningful. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* Escape {{...}} in changelog-generated markdown to prevent substitution errors A changelog entry whose title or description contains `{{hostname}}` (or any double-brace template placeholder) triggers the substitution parser when `ParseMarkdownStringAsync` re-parses the generated markdown. Because the parser can't resolve the key, it emits an Error diagnostic. That diagnostic's column is computed from the generated string, not from the source file (`docs/release-notes.md`), so Errata crashes trying to render a source label past the end of the line. `RenderChangelogMarkdown` now replaces `{{` with `{{` before returning. The HTML entity is invisible to the substitution parser (which opens only on `{`), passes through Markdig as literal text, and the browser decodes it back to `{{`. The visual output is unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Use ChangelogPipeline to parse generated changelog markdown Changelog markdown is generated from GitHub PR/issue titles and descriptions, not authored in docs-builder source files. Running it through the full docs-builder pipeline causes substitution keys, MyST roles, and the diagnostic link validator to fire on content that was never intended to carry any of that syntax — the {{hostname}} build failure is the direct example. A dedicated ChangelogPipeline strips all docs-specific parsers: no UseSubstitution, no role parsers, no UseDiagnosticLinks. It keeps the structural elements the renderer actually uses (headings, lists, emphasis, autolinks, pipe tables, directives for dropdowns) and calls DisableHtml so raw HTML in descriptions does not bleed into the output. The earlier {{}} → { escape workaround in RenderChangelogMarkdown is removed — it is no longer needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* Port API sitemap parity audit test to main The test lived only on the `test/api-sitemap-parity` branch. Bringing it to `main` keeps the audit reproducible without a branch switch. The test skips unless `FEATURE_ASSEMBLER_API_EXPLORER=true` is set, so it never runs in CI. Set that variable and point the assembler output at `.artifacts/assembly` to compare the local build against the Bump.sh production sitemap. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Migrate ApiSitemapParityTests to TUnit The xUnit-to-TUnit migration in #4182 missed this test, so it still used `[Fact]`, `Assert.Skip*`, and `TestContext.Current.CancellationToken`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fail fast on invalid or empty sitemap in ApiSitemapParityTests Prevents a silent green pass when the sitemap endpoint returns an unexpected XML document (still HTTP 200). DownloadPageUrls now throws if the root element is not a urlset or sitemapindex in the expected namespace, and the test asserts the discovered URL set is non-empty. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add TODO noting ApiSitemapParityTests should be deleted post-migration References elastic/docs-eng-team#436. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…4235) Made with ❤️️ by updatecli Co-authored-by: elastic-observability-automation[bot] <180520183+elastic-observability-automation[bot]@users.noreply.github.com>

Same gzip-in-place + --content-encoding gzip pattern as the local docs preview upload. The assembler tree is ~2 GiB of uncompressed HTML; pre-compressing before aws s3 sync cuts wire cost to ~120 MB. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Mpdreamz · 2026-09-29T10:04:39Z

Closed to retarget to main as a standalone PR. Content preserved in new PR.

Mpdreamz requested a review from a team as a code owner September 22, 2026 16:11

Mpdreamz requested a review from theletterf September 22, 2026 16:11

Mpdreamz added the automation packaging, ci/cd. label Sep 22, 2026

Mpdreamz mentioned this pull request Sep 22, 2026

Replace Parcel with Vite for frontend builds #4205

Closed

github-actions Bot approved these changes Sep 22, 2026

View reviewed changes

Mpdreamz added this pull request to stack #4209 September 22, 2026 16:20

Mpdreamz force-pushed the fix/compress-assembler-preview branch from d3c4439 to 8d34452 Compare September 22, 2026 16:30

Mpdreamz force-pushed the fix/compress-assembler-preview branch from 8d34452 to 8caa45c Compare September 22, 2026 16:32

Mpdreamz force-pushed the fix/compress-assembler-preview branch from 8caa45c to 1c42cde Compare September 24, 2026 11:23

github-actions Bot requested changes Sep 24, 2026

View reviewed changes

Mpdreamz force-pushed the fix/compress-assembler-preview branch from 1c42cde to f22fd2f Compare September 24, 2026 12:28

Mpdreamz force-pushed the fix/compress-assembler-preview branch 2 times, most recently from 94f9bb8 to d95e908 Compare September 24, 2026 14:14

github-actions Bot approved these changes Sep 24, 2026

View reviewed changes

akira28 and others added 4 commits September 25, 2026 09:10

akira28 approved these changes Sep 28, 2026

View reviewed changes

elastic-observability-automation Bot and others added 2 commits September 29, 2026 08:30

chore: Update config/versions.yml elasticsearch-client-dotnet 9.5.3 (#…

2f6f603

…4235) Made with ❤️️ by updatecli Co-authored-by: elastic-observability-automation[bot] <180520183+elastic-observability-automation[bot]@users.noreply.github.com>

Mpdreamz force-pushed the fix/compress-assembler-preview branch from d95e908 to f6d881a Compare September 29, 2026 10:01

Mpdreamz closed this Sep 29, 2026

Mpdreamz mentioned this pull request Sep 29, 2026

Pre-compress the assembler preview S3 upload #4239

Open

github-actions Bot deployed to assembler-preview September 29, 2026 10:07

View deployment

Navigation Menu

Repository navigation

Pre-compress the assembler preview S3 upload - #4210

Pre-compress the assembler preview S3 upload#4210
Mpdreamz wants to merge 6 commits into
fix/slow-docs-previewfrom
fix/compress-assembler-preview

Mpdreamz commented Sep 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot Sep 24, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

Mpdreamz commented Sep 29, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Mpdreamz commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Pre-compressed upload with zip exclusion

Verify

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Mpdreamz commented Sep 29, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Mpdreamz commented Sep 22, 2026 •

edited

Loading