Repository navigation
perf(updater): hash a full download while writing it - #780
kdroidFilter merged 3 commits into
Conversation
`downloadFully` wrote the artifact, then `ChecksumVerifier.verify` read the whole file back to compute its SHA-512, and a mismatch read it a third time to build the exception. Hash the bytes as they are written through a `DigestOutputStream`, as `DifferentialDownloader` already does, and reuse that value for the comparison and for `ChecksumException`. The comparison, the rejected file's deletion and the exception are unchanged.
kdroidFilter
left a comment
There was a problem hiding this comment.
Right, this is a better approach. A few changes:
ChecksumVerifier.verify()andcomputeSha512Base64()no longer have callers insrc/main. Please remove them.DifferentialDownloaderstill has its ownSHA_512constant and Base64 encoding. Please route it throughnewSha512Digest()/encode().- The new test only checks the JDK's
DigestOutputStreamand never runsdownloadFully. Please replace it with adownloadUpdate()test (differentialDownload = false,RangeHttpServer) that checks a wrong manifest hash throwsChecksumExceptionwith the real SHA-512 asactualand leaves nothing staged.
…gest with DifferentialDownloader
|
All three done in ca7e859:
|
kdroidFilter
left a comment
There was a problem hiding this comment.
The cache assertion in the new test can't fail. With differential = false, cacheForNextUpdate returns early, so nothing is ever cached, whatever the hash. I checked it by making downloadFully cache the rejected artifact: the test still passes. Without differential = false it fails as it should, and the download is still a full one since the cache is empty. Please drop that argument.
|
Right, thanks. Done in bbb432a: the test now uses the default |
d0089f4
into
NucleusFramework:nucleus-3.0
What
For a full (non-differential) download,
downloadFully(NucleusUpdater.kt) did three things:ChecksumVerifier.verify, which read the whole file back to compute SHA-512;computeSha512Base64, which read it a third time to buildChecksumException.Now the bytes are hashed as they are written, through a
DigestOutputStream. That is the approachDifferentialDownloaderalready uses ("The digest is computed while writing"). The resulting value is used both for the comparison and for the exception.ChecksumVerifierstays the single place that defines the algorithm and encoding. It is now justnewSha512Digest()andencode(digest):verify()andcomputeSha512Base64()lost their last callers and are removed;DifferentialDownloadergoes through the same two functions instead of its ownSHA_512constant and Base64 encoding.What is unchanged:
targetFile.sha512;ChecksumException(expected, actual)values;No public API change (
apiCheckpasses).Measured
This is an end-to-end
downloadUpdate()of a 128 MiB artifact withdifferentialDownload = false. It runs from the module's ownRangeHttpServeron loopback, on Windows 11 x64 with JDK 17.0.19, using a throwaway test that was not committed. There were 2 warm-up runs, then 5 measured runs:That is roughly 13% faster here. The SHA-512 work itself remains; it now overlaps the download. What goes away is the second sequential read of the artifact. Here that read was served from the page cache right after the write. The saving should be larger when it isn't: a cold cache, an HDD, or Windows Defender scanning the new file on read.
Tests
New
DifferentialUpdateE2ETestcase,a full download that does not match the manifest checksum is rejected and not staged. It runsdownloadUpdate()againstRangeHttpServerwith an empty cache, so the download is a full one. Differential downloads stay on, so the rejected artifact could reach the cache if the code allowed it. The manifest's SHA-512 belongs to another build, and the test checks three things:ChecksumExceptioncarries the real SHA-512 of the bytes received asactual;nucleus-update-*staging directory is left behind;It fails if the bytes are written without going through the digest, and if the rejected artifact is cached.
ChecksumVerifierTestis removed: every case exercised the deletedverify()/computeSha512Base64(), or only the JDK'sDigestOutputStream../gradlew :updater-runtime:test :updater-runtime:ktlintCheck :updater-runtime:detekt :updater-runtime:apiCheck: 259 tests, 0 failures, run on Windows only. The change is plain JVM I/O.Note: draft #745 also edits
NucleusUpdater.kt. It adds a function next todownloadFullybut does not touch its body, and it targetsmain.