Skip to content

feat(updater): faster, sturdier differential updates - #745

Open
sproctor wants to merge 8 commits into
NucleusFramework:nucleus-3.0from
sproctor:feat/delta-download-improvements
Open

sproctor wants to merge 8 commits into
NucleusFramework:nucleus-3.0from
sproctor:feat/delta-download-improvements

Conversation

@sproctor

Copy link
Copy Markdown
Contributor

🚀 Description

Improvements to differential (blockmap) updates, as eight self-contained commits that can be reviewed one at a time:

Commit Change
fix(updater): bound block map reads… Block map downloads, the AppImage tail read and inflation are capped; a manifest blockMapSize over 64 MiB is refused before any request; non-positive block sizes are rejected.
fix(updater): insert .blockmap before a query string… UpdateProvider.getBlockMapUrl()'s default puts .blockmap at the end of the path, before ?…, as electron-updater does.
perf(updater): merge ranged requests separated by small unchanged gaps DeltaPlan.coalesce() merges two downloads separated by ≤ 256 KiB of copies into one request. compute() is unchanged and still matches electron-updater.
perf(updater): download the whole artifact when a delta would fetch 90%… A plan fetching ≥ 90% of the artifact becomes one streamed full download.
perf(updater): download delta ranges concurrently, with retries Up to 6 concurrent ranges written by position; redirect resolved once; retries for I/O errors, 429 and 5xx (backoff, Retry-After); an expired redirect target is re-resolved; credentials only go to the original host; 30 s response timeout; no partial file left on failure.
perf(updater): make the first Windows NSIS update differential With nothing cached, an .exe update diffs against the installer electron-builder seeds in %LOCALAPPDATA%\<updaterCacheDirName>\installer.exe, using the installed release's published block map.
ci(macos): publish a block map for the universal macOS ZIP build-macos-universal generates <zip>.blockmap with electron-builder's own buildBlockMap, installed from the plugin's pinned lock file.
fix(plugin): delete a DMG's block map once stapling invalidates it The notarize task deletes the stale .blockmap after stapling.

📄 Motivation and Context

Correctness and robustness

  • Unbounded reads. Block maps were read with BodyHandlers.ofByteArray() and inflated with readBytes(), so their size was decided by the server. A manifest blockMapSize just under the artifact size pulls most of an AppImage into memory, and a small gzip bomb inflates without limit. The resulting OutOfMemoryError is an Error, so it escapes the catch (Exception) that falls back to a full download.
  • Query strings. "$fileUrl.blockmap" put the suffix inside the last query parameter's value for any artifact URL carrying a query (token, cache-busting version), so the block map was never found.
  • Notarized DMGs. Stapling rewrites the DMG after electron-builder wrote its block map. blockMapSize was already stripped from the manifest, but the resolver keys the flavour on format, so it still fetched the stale sidecar, failed the plan-size check, fell back to a full download, and cached the stale map for the next update.

Performance

  • Ranges were fetched one at a time, with a 1 s pause every 100 and a redirect round trip per range; one transient error abandoned the whole delta. An update touching many scattered blocks spent most of its time on round trips.
  • Every first update of a non-AppImage install was a full download, although NSIS installs already leave the previous installer on disk.
  • Universal macOS installs never got deltas: the ditto-built ZIP had no block map.

Notes for review

  • Blocks now arrive out of order, so the SHA-512 is computed in one pass over the assembled file instead of while writing. Progress is still reported from the caller's coroutine, so it can emit into the Flow.
  • updaterCacheDirName is derived at run time from the running .exe: electron-builder builds it from the package.json name, which the plugin derives from the executable name (windows.packageName ?: packageName for jpackage, imageName for GraalVM). If it cannot be found, or the old release's block map is no longer hosted, the update is a full download as before.
  • electron-builder 26 no longer ships app-builder-bin, but builds block maps in JS (app-builder-lib/out/targets/blockmap/blockmap). The release workflow's universal job adds the lock file's directory to its sparse checkout; callers using the action by reference get the whole repository. A failure there only warns.
  • For notarized DMGs I chose deleting the stale block map over regenerating it: the toolchain is removed after packaging, and macOS updates prefer the ZIP, which is never stapled and keeps its map. Regenerating would mean re-provisioning electron-builder in the notarize task or porting its Rabin chunker (plus BLAKE2b) to Kotlin; happy to do either if you prefer.

🧪 How Has This Been Tested?

./gradlew :updater-runtime:check and ./gradlew :plugin:check (in plugin-build) pass. New and extended tests:

  • BlockMapCodecTest: a 160 MiB gzip bomb is rejected by the bound while inflating; non-positive block sizes; readAtMost.
  • DeltaResolverTest: an implausible blockMapSize is refused with no request made; the first NSIS update diffs against a seeded installer (byte-identical result, EXPECTED_DELTA_BYTES fetched); no delta when the old release's map is gone; the seeded installer is only used for .exe targets.
  • DeltaPlanTest: coalescing merges, respects the bound, sums multi-copy gaps, and a coalesced plan with scattered edits still reproduces the artifact byte for byte.
  • DifferentialUpdateE2ETest: a ~96% delta becomes a full download with no ranged requests.
  • DifferentialDownloaderTest (new): concurrency (max in flight > 1), progress ordering, retry of 503/429, giving up after 3 attempts with no partial file, no retry for a host ignoring Range or a 404, ranges going straight to the redirect target without the original host's credentials, and re-resolution of an expired target.
  • SeededInstallerTest (new): directory naming, lookup, and previous artifact name.
  • GenericProviderTest: block map URL with a query string.

RangeHttpServer gains a thread pool, per-range latency, scripted failures and redirects for these tests; existing tests are unaffected.

Every new test except the naming/URL ones was checked to fail with the corresponding change removed. The concurrency tests passed five repeated runs. The universal-ZIP step was exercised on Linux by running its function against a sample ZIP (valid version-2 map covering the file). Not tested: a real Windows NSIS update, a real macOS universal release build, and the notarize-task deletion (it needs xcrun).

📦 Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

✅ Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

🤖 Generated with Claude Code

sproctor and others added 8 commits November 2, 2026 09:29
Block maps were read with BodyHandlers.ofByteArray() and inflated with
readBytes(), so their size was decided by the server: a manifest
blockMapSize just under the artifact size pulled most of the artifact
into memory for an AppImage tail, and a small gzip or deflate bomb
inflated without limit. The resulting OutOfMemoryError is an Error, so
it escaped the catch (Exception) that falls back to a full download.

- The manifest's blockMapSize is capped at 64 MiB (the existing bound on
  embedded block maps) before the tail is requested.
- Fetched block maps and range reads stop at their expected size.
- Inflating a block map stops at 128 MiB.
- A block map with a negative offset or a non-positive block size is
  rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lock map URL

getBlockMapUrl() appended ".blockmap" to the whole URL, so for an
artifact URL carrying a query string (a download token, a cache-busting
version) the suffix landed inside the last query parameter's value and
the block map was never found. It now goes at the end of the path, as
electron-updater does. Signed URLs whose signature covers the path still
need an override, as the KDoc already says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The plan only merged exactly adjacent operations, so an update with many
scattered changes issued one ranged request per changed run, each paying
a full round trip. DeltaPlan.coalesce() now merges two downloads that are
separated only by copies totalling at most 256 KiB, fetching the gap
instead of copying it from disk: a few extra bytes for far fewer
requests.

compute() is unchanged and still matches electron-updater's plan; the
resolver applies coalesce() on top of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0% of it

When nearly every block changed, a differential download still issued
ranged requests for almost the entire artifact and assembled it from
them. It now falls back to one streamed full download once the plan
would fetch at least 90% of the artifact's bytes, where the ranged
requests cannot save anything worthwhile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The differential downloader fetched changed ranges one at a time, paused
a second after every 100, followed the release host's redirect again for
every range, and abandoned the whole delta on the first transient error.
An update touching many scattered blocks spent most of its time on round
trips, and one dropped connection threw away all the progress.

- Operations are written to their fixed position in the output, so
  copies run alongside up to six concurrent range requests, which an
  HTTP/2 client multiplexes over one connection. The fixed pause is gone.
- The first range runs alone and resolves the redirect; later ranges go
  straight to the resolved URL. Credentials are only sent to the
  original host, not to the CDN it redirects to.
- I/O errors, HTTP 429 and 5xx are retried (3 attempts, exponential
  backoff, Retry-After honoured and capped at 10 s). A 401/403 from the
  redirect target, typically an expired pre-signed URL, goes back
  through the original URL for a fresh one.
- Range responses must start within 30 s.
- A failed or cancelled assembly deletes the partial file.

Because blocks now arrive out of order, the SHA-512 is computed in one
pass over the assembled file instead of while writing. Progress is still
reported from the caller's coroutine, so it can emit into the Flow.

RangeHttpServer gains a thread pool, per-range latency, scripted failure
responses and redirects, so DifferentialDownloaderTest can cover these.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A differential download needs the previous artifact, and the update
cache is only filled by the updater's own downloads, so every first
update on a machine downloaded the whole installer. But every NSIS
install already leaves one: electron-builder's template copies the
installer that ran to %LOCALAPPDATA%\<updaterCacheDirName>\installer.exe.

When nothing usable is cached and the update is an .exe installer, the
resolver now diffs against that seeded installer. Its block map is the
one published with the installed release: the artifact name is the new
one with the installed version swapped in, fetched through the provider
for that version. If the old release is no longer hosted, the name does
not contain the version, or no installer was seeded, the update is a
full download as before.

updaterCacheDirName is electron-builder's sanitized, lower-cased package
name plus "-updater". The plugin derives that name from the executable
name, which is also the name of the running .exe, so the updater derives
the directory at run time and the plugin needs no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The universal ZIP is assembled with ditto after the per-architecture
builds, so electron-builder never sees it and no .zip.blockmap sidecar
was published: every update of a universal install was a full download.

build-macos-universal now generates the sidecar with electron-builder's
own block map builder (app-builder-lib's buildBlockMap, as used for its
own sidecars), installed from the plugin's pinned lock file with
npm ci --ignore-scripts. The sidecar leaves the ZIP unchanged, so the
sha512 and size in latest-mac.yml stay correct, and the ZIP is never
stapled, so the map stays valid. generate-update-yml already reads its
size into blockMapSize.

A failure (no Node.js, a registry error) only warns, since updaters fall
back to full downloads. Workflows that use the action by reference get
the whole repository, so the lock file is there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stapling rewrites a notarized DMG (or PKG) after electron-builder wrote
its .blockmap. The manifest's blockMapSize was already dropped, but the
stale sidecar stayed in the output and was published. The updater picks
the block map flavour by artifact format, not by blockMapSize, so every
differential update of such a DMG fetched the stale map, failed the
plan-size check, fell back to a full download, and cached the stale map
as the basis for the next update.

The notarize task now deletes the sidecar after stapling, so updaters go
straight to a full download. ZIPs are unaffected: they are not stapled
and keep their block maps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sproctor
sproctor force-pushed the feat/delta-download-improvements branch from e415dfe to 26d7c79 Compare November 2, 2026 14:50
@sproctor
sproctor changed the base branch from main to nucleus-3.0 November 2, 2026 14:50
@sproctor
sproctor marked this pull request as ready for review November 2, 2026 14:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant