Skip to content

Pre-compress the assembler preview S3 upload #4239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mpdreamz wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Pre-compress the assembler preview S3 upload #4239

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f6d881a
CI: pre-compress assembler preview S3 upload
Mpdreamz Sep 22, 2026
ff5fe69
CI: enable pipefail before the find|xargs gzip pre-compression pipeline
Mpdreamz Sep 29, 2026
6658c74
Merge remote-tracking branch 'origin/main' into fix/compress-assemble…
akira28 Oct 27, 2026
957487c
CI: silence SC2016 for the intentional single-quoted gzip script
akira28 Oct 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .github/workflows/assembler-preview.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,11 +114,24 @@ jobs:
AWS_S3_MAX_CONCURRENT_REQUESTS: 50
AWS_S3_MAX_QUEUE_SIZE: 20000
run: |
set -o pipefail
aws configure set default.s3.preferred_transfer_client classic
aws configure set default.s3.max_concurrent_requests "${AWS_S3_MAX_CONCURRENT_REQUESTS}"
aws configure set default.s3.max_queue_size "${AWS_S3_MAX_QUEUE_SIZE}"

aws s3 sync ".artifacts/assembly/${ASSEMBLER_PREVIEW_PATH_PREFIX}" "s3://elastic-docs-v3-website-preview/${ASSEMBLER_PREVIEW_PATH_PREFIX}" --delete --no-follow-symlinks
# Pre-compress all non-zip files so aws s3 sync ships ~120 MB instead of ~2 GiB.
# Zip bundles (llm.zip, okf.zip) are already compressed; syncing them with
# content-encoding:gzip would prevent clients from decoding them correctly.
# The inner sh -c script is intentionally single-quoted so it expands per file, not here.
# shellcheck disable=SC2016
find ".artifacts/assembly/${ASSEMBLER_PREVIEW_PATH_PREFIX}" -type f ! -name "*.zip" -print0 \

@github-actions github-actions Bot Sep 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Fail the step if any pre-compression discovery fails

This pipeline can succeed even when find fails, because GitHub Actions runs bash -e but not pipefail by default. In find ... | xargs ..., the pipeline status comes from xargs; a partial find error can be masked, leaving some files uncompressed while the next sync still applies --content-encoding gzip to all non-zip files.

That can publish incorrectly encoded objects (clients receive non-gzip bytes labeled as gzip). Please enable set -o pipefail before this pipeline (or refactor to avoid a pipeline) so any find failure stops the upload.

| xargs -0 -P "$(nproc)" -I{} sh -c 'tmp=$(mktemp "$1.tmp.XXXXXX") && gzip -9 -n -c "$1" > "$tmp" && mv -f "$tmp" "$1"' _ {}

# Sync compressed files with gzip content-encoding; sync zip downloads separately.
aws s3 sync ".artifacts/assembly/${ASSEMBLER_PREVIEW_PATH_PREFIX}" "s3://elastic-docs-v3-website-preview/${ASSEMBLER_PREVIEW_PATH_PREFIX}" \
--delete --no-follow-symlinks --content-encoding gzip --exclude "*.zip"
aws s3 sync ".artifacts/assembly/${ASSEMBLER_PREVIEW_PATH_PREFIX}" "s3://elastic-docs-v3-website-preview/${ASSEMBLER_PREVIEW_PATH_PREFIX}" \
--delete --no-follow-symlinks --exclude "*" --include "*.zip"
aws cloudfront create-invalidation \
--distribution-id EKT7LT5PM8RKS \
--paths "/${ASSEMBLER_PREVIEW_PATH_PREFIX}" "/${ASSEMBLER_PREVIEW_PATH_PREFIX}/*"
Expand Down
Loading