Repository navigation
Conversation
sproctor
marked this pull request as draft
October 5, 2026 09:30
Block maps were read with BodyHandlers.ofByteArray() and inflated with readBytes(), so their size was decided by the server: a manifest blockMapSize just under the artifact size pulled most of the artifact into memory for an AppImage tail, and a small gzip or deflate bomb inflated without limit. The resulting OutOfMemoryError is an Error, so it escaped the catch (Exception) that falls back to a full download. - The manifest's blockMapSize is capped at 64 MiB (the existing bound on embedded block maps) before the tail is requested. - Fetched block maps and range reads stop at their expected size. - Inflating a block map stops at 128 MiB. - A block map with a negative offset or a non-positive block size is rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lock map URL getBlockMapUrl() appended ".blockmap" to the whole URL, so for an artifact URL carrying a query string (a download token, a cache-busting version) the suffix landed inside the last query parameter's value and the block map was never found. It now goes at the end of the path, as electron-updater does. Signed URLs whose signature covers the path still need an override, as the KDoc already says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plan only merged exactly adjacent operations, so an update with many scattered changes issued one ranged request per changed run, each paying a full round trip. DeltaPlan.coalesce() now merges two downloads that are separated only by copies totalling at most 256 KiB, fetching the gap instead of copying it from disk: a few extra bytes for far fewer requests. compute() is unchanged and still matches electron-updater's plan; the resolver applies coalesce() on top of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0% of it When nearly every block changed, a differential download still issued ranged requests for almost the entire artifact and assembled it from them. It now falls back to one streamed full download once the plan would fetch at least 90% of the artifact's bytes, where the ranged requests cannot save anything worthwhile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The differential downloader fetched changed ranges one at a time, paused a second after every 100, followed the release host's redirect again for every range, and abandoned the whole delta on the first transient error. An update touching many scattered blocks spent most of its time on round trips, and one dropped connection threw away all the progress. - Operations are written to their fixed position in the output, so copies run alongside up to six concurrent range requests, which an HTTP/2 client multiplexes over one connection. The fixed pause is gone. - The first range runs alone and resolves the redirect; later ranges go straight to the resolved URL. Credentials are only sent to the original host, not to the CDN it redirects to. - I/O errors, HTTP 429 and 5xx are retried (3 attempts, exponential backoff, Retry-After honoured and capped at 10 s). A 401/403 from the redirect target, typically an expired pre-signed URL, goes back through the original URL for a fresh one. - Range responses must start within 30 s. - A failed or cancelled assembly deletes the partial file. Because blocks now arrive out of order, the SHA-512 is computed in one pass over the assembled file instead of while writing. Progress is still reported from the caller's coroutine, so it can emit into the Flow. RangeHttpServer gains a thread pool, per-range latency, scripted failure responses and redirects, so DifferentialDownloaderTest can cover these. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A differential download needs the previous artifact, and the update cache is only filled by the updater's own downloads, so every first update on a machine downloaded the whole installer. But every NSIS install already leaves one: electron-builder's template copies the installer that ran to %LOCALAPPDATA%\<updaterCacheDirName>\installer.exe. When nothing usable is cached and the update is an .exe installer, the resolver now diffs against that seeded installer. Its block map is the one published with the installed release: the artifact name is the new one with the installed version swapped in, fetched through the provider for that version. If the old release is no longer hosted, the name does not contain the version, or no installer was seeded, the update is a full download as before. updaterCacheDirName is electron-builder's sanitized, lower-cased package name plus "-updater". The plugin derives that name from the executable name, which is also the name of the running .exe, so the updater derives the directory at run time and the plugin needs no change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The universal ZIP is assembled with ditto after the per-architecture builds, so electron-builder never sees it and no .zip.blockmap sidecar was published: every update of a universal install was a full download. build-macos-universal now generates the sidecar with electron-builder's own block map builder (app-builder-lib's buildBlockMap, as used for its own sidecars), installed from the plugin's pinned lock file with npm ci --ignore-scripts. The sidecar leaves the ZIP unchanged, so the sha512 and size in latest-mac.yml stay correct, and the ZIP is never stapled, so the map stays valid. generate-update-yml already reads its size into blockMapSize. A failure (no Node.js, a registry error) only warns, since updaters fall back to full downloads. Workflows that use the action by reference get the whole repository, so the lock file is there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stapling rewrites a notarized DMG (or PKG) after electron-builder wrote its .blockmap. The manifest's blockMapSize was already dropped, but the stale sidecar stayed in the output and was published. The updater picks the block map flavour by artifact format, not by blockMapSize, so every differential update of such a DMG fetched the stale map, failed the plan-size check, fell back to a full download, and cached the stale map as the basis for the next update. The notarize task now deletes the sidecar after stapling, so updaters go straight to a full download. ZIPs are unaffected: they are not stapled and keep their block maps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sproctor
force-pushed
the
feat/delta-download-improvements
branch
from
November 2, 2026 14:50
e415dfe to
26d7c79
Compare
sproctor
marked this pull request as ready for review
November 2, 2026 14:54
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Description
Improvements to differential (blockmap) updates, as eight self-contained commits that can be reviewed one at a time:
fix(updater): bound block map reads…blockMapSizeover 64 MiB is refused before any request; non-positive block sizes are rejected.fix(updater): insert .blockmap before a query string…UpdateProvider.getBlockMapUrl()'s default puts.blockmapat the end of the path, before?…, as electron-updater does.perf(updater): merge ranged requests separated by small unchanged gapsDeltaPlan.coalesce()merges two downloads separated by ≤ 256 KiB of copies into one request.compute()is unchanged and still matches electron-updater.perf(updater): download the whole artifact when a delta would fetch 90%…perf(updater): download delta ranges concurrently, with retriesRetry-After); an expired redirect target is re-resolved; credentials only go to the original host; 30 s response timeout; no partial file left on failure.perf(updater): make the first Windows NSIS update differential.exeupdate diffs against the installer electron-builder seeds in%LOCALAPPDATA%\<updaterCacheDirName>\installer.exe, using the installed release's published block map.ci(macos): publish a block map for the universal macOS ZIPbuild-macos-universalgenerates<zip>.blockmapwith electron-builder's ownbuildBlockMap, installed from the plugin's pinned lock file.fix(plugin): delete a DMG's block map once stapling invalidates it.blockmapafter stapling.📄 Motivation and Context
Correctness and robustness
BodyHandlers.ofByteArray()and inflated withreadBytes(), so their size was decided by the server. A manifestblockMapSizejust under the artifact size pulls most of an AppImage into memory, and a small gzip bomb inflates without limit. The resultingOutOfMemoryErroris anError, so it escapes thecatch (Exception)that falls back to a full download."$fileUrl.blockmap"put the suffix inside the last query parameter's value for any artifact URL carrying a query (token, cache-busting version), so the block map was never found.blockMapSizewas already stripped from the manifest, but the resolver keys the flavour on format, so it still fetched the stale sidecar, failed the plan-size check, fell back to a full download, and cached the stale map for the next update.Performance
Notes for review
Flow.updaterCacheDirNameis derived at run time from the running.exe: electron-builder builds it from thepackage.jsonname, which the plugin derives from the executable name (windows.packageName ?: packageNamefor jpackage,imageNamefor GraalVM). If it cannot be found, or the old release's block map is no longer hosted, the update is a full download as before.app-builder-bin, but builds block maps in JS (app-builder-lib/out/targets/blockmap/blockmap). The release workflow's universal job adds the lock file's directory to its sparse checkout; callers using the action by reference get the whole repository. A failure there only warns.🧪 How Has This Been Tested?
./gradlew :updater-runtime:checkand./gradlew :plugin:check(inplugin-build) pass. New and extended tests:BlockMapCodecTest: a 160 MiB gzip bomb is rejected by the bound while inflating; non-positive block sizes;readAtMost.DeltaResolverTest: an implausibleblockMapSizeis refused with no request made; the first NSIS update diffs against a seeded installer (byte-identical result,EXPECTED_DELTA_BYTESfetched); no delta when the old release's map is gone; the seeded installer is only used for.exetargets.DeltaPlanTest: coalescing merges, respects the bound, sums multi-copy gaps, and a coalesced plan with scattered edits still reproduces the artifact byte for byte.DifferentialUpdateE2ETest: a ~96% delta becomes a full download with no ranged requests.DifferentialDownloaderTest(new): concurrency (max in flight > 1), progress ordering, retry of 503/429, giving up after 3 attempts with no partial file, no retry for a host ignoringRangeor a 404, ranges going straight to the redirect target without the original host's credentials, and re-resolution of an expired target.SeededInstallerTest(new): directory naming, lookup, and previous artifact name.GenericProviderTest: block map URL with a query string.RangeHttpServergains a thread pool, per-range latency, scripted failures and redirects for these tests; existing tests are unaffected.Every new test except the naming/URL ones was checked to fail with the corresponding change removed. The concurrency tests passed five repeated runs. The universal-ZIP step was exercised on Linux by running its function against a sample ZIP (valid version-2 map covering the file). Not tested: a real Windows NSIS update, a real macOS universal release build, and the notarize-task deletion (it needs
xcrun).📦 Types of changes
✅ Checklist
🤖 Generated with Claude Code