Skip to content

Hash only the bytes read from a stream in Bytes.getMD5 - #16543

Merged
EarthChen merged 3 commits into
apache:3.3from
HhH666-ux:fix-16451-md5-stream
Oct 28, 2026
Merged

EarthChen merged 3 commits into
apache:3.3from
HhH666-ux:fix-16451-md5-stream

Conversation

@HhH666-ux

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Bytes.getMD5(InputStream) returns a wrong MD5 whenever the stream length is not a multiple of the internal 8 KiB buffer, and returns the digest of an empty input for streams that report available() == 0 while data is still readable. Fixes #16451.

Brief changelog

getMD5(InputStream, int) had two defects:

  1. The read loop filled the whole buffer and then called md.update(buf), hashing the entire buffer instead of the total bytes actually read. The last chunk of any stream whose length is not a multiple of bs therefore hashed stale tail bytes from the previous chunk, producing a wrong digest. This is the normal case for FileInputStream on file systems that short-read (network mounts, encrypted volumes), which is exactly what getMD5(File) forwards to.
  2. The outer loop was gated on while (is.available() > 0). available() reports the bytes readable without blocking, not the remaining length, so a stream that momentarily reports 0 (pipelined/socket-backed/deflated streams) terminated the loop before any byte was consumed.

Both are fixed by the canonical digest loop:

for (int read = is.read(buf); read != -1; read = is.read(buf)) {
    md.update(buf, 0, read);
}

The buffer-filling inner loop becomes unnecessary once the digest consumes exactly the bytes read.

Verifying this change

Added two regressions to BytesTest, both comparing against getMD5(byte[]):

  • testMD5ForStreamWithShortFinalRead: a random 8192 + 17 byte stream (short final chunk).
  • testMD5ForStreamWithZeroAvailable: a stream whose available() always returns 0 but whose read() serves data.

Both fail on the previous implementation (verified locally) and pass with this change. mvn -f dubbo-common/pom.xml test -Dtest=BytesTest and spotless:check pass.

Follow this checklist to request gating of a pull request:

  • Make sure there is a GitHub_issue field for the change (usually above).
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Write necessary unit-test to verify your logic correction. More hard-to-mock scenarios are considered acceptable: verify them via integration tests or e2e tests.
  • ./mvnw -T 1C clean install -DskipTests=false -DskipIntegrationTests=false passes locally — not run locally in full; focused module tests and spotless pass (module-level mvn -f dubbo-common/pom.xml test and spotless:check).
  • If this contribution is large, please file an Apache Individual Contributor License Agreement — change is small.

@codecov-commenter

codecov-commenter commented Oct 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.89%. Comparing base (843b98b) to head (24b1b79).

Additional details and impacted files
@@             Coverage Diff              @@
##                3.3   #16543      +/-   ##
============================================
- Coverage     60.90%   60.89%   -0.01%     
- Complexity       15    11756   +11741     
============================================
  Files          1953     1953              
  Lines         89280    89274       -6     
  Branches      13476    13474       -2     
============================================
- Hits          54376    54365      -11     
- Misses        29323    29326       +3     
- Partials       5581     5583       +2     
Flag Coverage Δ
integration-tests-java21 32.07% <0.00%> (-0.01%) ⬇️
integration-tests-java8 32.20% <0.00%> (+0.05%) ⬆️
samples-tests-java21 32.15% <0.00%> (+0.02%) ⬆️
samples-tests-java8 29.77% <0.00%> (+<0.01%) ⬆️
unit-tests-java11 59.13% <100.00%> (+0.01%) ⬆️
unit-tests-java17 58.64% <100.00%> (+0.03%) ⬆️
unit-tests-java21 58.62% <100.00%> (-0.01%) ⬇️
unit-tests-java25 58.59% <100.00%> (-0.02%) ⬇️
unit-tests-java8 59.15% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

md5.testfile.txt is 12 bytes while getMD5(File) reads through an 8 KiB
buffer. The old getMD5 copied the whole buffer into the digest on every
iteration, so the expectation in this test was the digest of the file
followed by ~8180 zero bytes rather than the digest of the file.

Now that only the bytes actually read are hashed, the expectation has to
be the real digest ("/D/5joxqDTCH1RXARz+Gdw=="). Assert it against
getMD5(byte[]) as well, so the test states the intent instead of pinning
whatever the buffer handling happens to produce.
@HhH666-ux

Copy link
Copy Markdown
Contributor Author

Fixed the failing test — the expectation was stale, not the implementation.

md5.testfile.txt is 12 bytes while getMD5(File) reads through an 8 KiB buffer. The old loop called md.update(buf) without a length, so the single iteration hashed the whole buffer: the file content followed by ~8180 zero bytes. That is why the old expectation looked like a normal digest — it was the digest of the zero-padded buffer:

input MD5
the 12 bytes of md5.testfile.txt /D/5joxqDTCH1RXARz+Gdw== (what the new code returns)
those 12 bytes padded to 8192 iNZ+5qHafVNPLJxHwLKJ3w== (the old expectation)

a38a5b4 updates the expectation and additionally asserts it against getMD5(byte[]), so the test states the intent (digest of the file content) instead of pinning whatever the buffer handling happened to produce.

The other overloads are unaffected: getMD5(String) and getMD5(byte[]) hash the array directly via md.digest(source). Only getMD5(File) and getMD5(InputStream) went through the read loop, and testMD5ForStreamWithShortFinalRead / testMD5ForStreamWithZeroAvailable already cover the loop itself — this was the only test that had the buggy value baked in.

@EarthChen EarthChen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Confirmed the bug: the old loop called md.update(buf) with the whole 8 KiB buffer instead of the bytes actually read, so any stream whose length is not a multiple of 8192 produced a wrong MD5. The available() > 0 loop condition also returned the empty-input digest for streams that report 0 available. Reading until -1 and hashing buf[0..read) fixes both. BytesTest passes locally. In this repo only getMD5(String) is called from production code, so no compatibility impact.

Non-blocking: the thread-local MessageDigest is not reset if read throws (pre-existing); consider md.reset() in the failure path.

@EarthChen
EarthChen merged commit 16e25a5 into apache:3.3 Oct 28, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Bytes.getMD5(InputStream/File) returns incorrect MD5 for short final reads

3 participants