Skip to content

perf(updater): hash a full download while writing it - #780

Merged
kdroidFilter merged 3 commits into
NucleusFramework:nucleus-3.0from
palmoni5:perf/updater-hash-while-downloading
Oct 6, 2026
Merged

kdroidFilter merged 3 commits into
NucleusFramework:nucleus-3.0from
palmoni5:perf/updater-hash-while-downloading

Conversation

@palmoni5

@palmoni5 palmoni5 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

What

For a full (non-differential) download, downloadFully (NucleusUpdater.kt) did three things:

  1. streamed the artifact into the temp file;
  2. called ChecksumVerifier.verify, which read the whole file back to compute SHA-512;
  3. on a mismatch, called computeSha512Base64, which read it a third time to build ChecksumException.

Now the bytes are hashed as they are written, through a DigestOutputStream. That is the approach DifferentialDownloader already uses ("The digest is computed while writing"). The resulting value is used both for the comparison and for the exception.

ChecksumVerifier stays the single place that defines the algorithm and encoding. It is now just newSha512Digest() and encode(digest):

  • verify() and computeSha512Base64() lost their last callers and are removed;
  • DifferentialDownloader goes through the same two functions instead of its own SHA_512 constant and Base64 encoding.

What is unchanged:

  • the hashed bytes, which are exactly the bytes written;
  • the exact string comparison against targetFile.sha512;
  • deleting the rejected file;
  • the ChecksumException(expected, actual) values;
  • the progress events.

No public API change (apiCheck passes).

Measured

This is an end-to-end downloadUpdate() of a 128 MiB artifact with differentialDownload = false. It runs from the module's own RangeHttpServer on loopback, on Windows 11 x64 with JDK 17.0.19, using a throwaway test that was not committed. There were 2 warm-up runs, then 5 measured runs:

runs (ms) best
before 1169, 1138, 1015, 1013, 1300 1013
after 1144, 1043, 884, 933, 899 884

That is roughly 13% faster here. The SHA-512 work itself remains; it now overlaps the download. What goes away is the second sequential read of the artifact. Here that read was served from the page cache right after the write. The saving should be larger when it isn't: a cold cache, an HDD, or Windows Defender scanning the new file on read.

Tests

New DifferentialUpdateE2ETest case, a full download that does not match the manifest checksum is rejected and not staged. It runs downloadUpdate() against RangeHttpServer with an empty cache, so the download is a full one. Differential downloads stay on, so the rejected artifact could reach the cache if the code allowed it. The manifest's SHA-512 belongs to another build, and the test checks three things:

  • the ChecksumException carries the real SHA-512 of the bytes received as actual;
  • no nucleus-update-* staging directory is left behind;
  • nothing is cached.

It fails if the bytes are written without going through the digest, and if the rejected artifact is cached.

ChecksumVerifierTest is removed: every case exercised the deleted verify() / computeSha512Base64(), or only the JDK's DigestOutputStream.

./gradlew :updater-runtime:test :updater-runtime:ktlintCheck :updater-runtime:detekt :updater-runtime:apiCheck: 259 tests, 0 failures, run on Windows only. The change is plain JVM I/O.

Note: draft #745 also edits NucleusUpdater.kt. It adds a function next to downloadFully but does not touch its body, and it targets main.

`downloadFully` wrote the artifact, then `ChecksumVerifier.verify` read
the whole file back to compute its SHA-512, and a mismatch read it a
third time to build the exception.

Hash the bytes as they are written through a `DigestOutputStream`, as
`DifferentialDownloader` already does, and reuse that value for the
comparison and for `ChecksumException`. The comparison, the rejected
file's deletion and the exception are unchanged.

@kdroidFilter kdroidFilter left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, this is a better approach. A few changes:

  1. ChecksumVerifier.verify() and computeSha512Base64() no longer have callers in src/main. Please remove them.
  2. DifferentialDownloader still has its own SHA_512 constant and Base64 encoding. Please route it through newSha512Digest() / encode().
  3. The new test only checks the JDK's DigestOutputStream and never runs downloadFully. Please replace it with a downloadUpdate() test (differentialDownload = false, RangeHttpServer) that checks a wrong manifest hash throws ChecksumException with the real SHA-512 as actual and leaves nothing staged.

@palmoni5

palmoni5 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

All three done in ca7e859:

  1. verify() and computeSha512Base64() are removed. ChecksumVerifier is now just newSha512Digest() and encode().
  2. DifferentialDownloader uses both of them. Its SHA_512 constant and Base64 import are gone.
  3. ChecksumVerifierTest is removed, since all of its cases used the deleted functions. In its place, DifferentialUpdateE2ETest has a downloadUpdate() case with differentialDownload = false against RangeHttpServer and a wrong manifest hash. It asserts that ChecksumException carries the real SHA-512 as actual, that no nucleus-update-* staging directory is left, and that nothing is cached. I checked that it fails when the bytes bypass the digest.

:updater-runtime:test (259), ktlint, detekt and apiCheck pass on Windows. Description updated.

@kdroidFilter kdroidFilter left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache assertion in the new test can't fail. With differential = false, cacheForNextUpdate returns early, so nothing is ever cached, whatever the hash. I checked it by making downloadFully cache the rejected artifact: the test still passes. Without differential = false it fails as it should, and the download is still a full one since the cache is empty. Please drop that argument.

@palmoni5

palmoni5 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Right, thanks. Done in bbb432a: the test now uses the default differentialDownload. The cache is empty, so it is still a full download. I checked the same mutation, caching the rejected artifact in downloadFully: the test now fails on the cache assertion. 259 tests pass. Description updated.

@kdroidFilter
kdroidFilter merged commit d0089f4 into NucleusFramework:nucleus-3.0 Oct 6, 2026
19 of 23 checks passed
@palmoni5
palmoni5 deleted the perf/updater-hash-while-downloading branch October 6, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants