Skip to content

internal/json: decode in place instead of through a streaming Decoder - #1454

Open
raveesh-me wants to merge 1 commit into
modelcontextprotocol:mainfrom
raveesh-me:perf/json-unmarshal-no-stream-buffer
Open

raveesh-me wants to merge 1 commit into
modelcontextprotocol:mainfrom
raveesh-me:perf/json-unmarshal-no-stream-buffer

Conversation

@raveesh-me

@raveesh-me raveesh-me commented Oct 25, 2026 •

Copy link
Copy Markdown

internal/json.Unmarshal decoded through segmentio's streaming json.Decoder, which copies its input into a read buffer of at least 32 KiB on every call. Unmarshal runs several times per request, so these buffers made up most of the memory allocated to serve a tool call (528 KiB per call in BenchmarkStreamableServing). Details in #1453.

This change decodes in place:

  • json.Parse into a json.RawMessage with DontCopyRawMessage finds the extent of the first JSON value. It's the same parseValue the Decoder runs on its buffer.
  • That value is then decoded with the same flags as before.
  • If finding the value fails, the input is malformed and goes through a Decoder as before. Errors are therefore exactly today's: io.EOF for empty input, io.ErrUnexpectedEOF for truncated input, and the same SyntaxError and UnmarshalTypeError text.

Case-sensitive matching and the depth limit are unchanged.

One function, newDecoder, configures the Decoder, and parseFlags gives json.Parse the same rules. TestUnmarshalSameAsStream and FuzzUnmarshalSameAsStream compare the new path with the old one for Unmarshal and UnmarshalUseNumber into several target types, checking the decoded value, the error type and the error text. Locally the fuzzer ran about 5 million inputs without finding a difference. TestUnmarshalDoesNotAliasInput checks that decoded values never share memory with the input, since the value is now found in place and callers may reuse their buffers. TestUnmarshalUseNumberCaseSensitivity covers UnmarshalUseNumber, which decodes every JSON-RPC envelope since #1401; the existing case-sensitivity tests only cover Unmarshal.

name                  old time/op    new time/op    delta
StreamableServing     368.7µs ± 2%   219.3µs ± 1%   -40.51%
Unmarshal             4.015µs ± 1%   1.337µs ± 1%   -66.71%

name                  old alloc/op   new alloc/op   delta
StreamableServing     528.3KiB ± 0%  60.1KiB ± 1%   -88.62%
Unmarshal             37.41KiB ± 0%  5.39KiB ± 0%   -85.59%

Run with go test ./..., go test -race ./..., go vet ./..., staticcheck and go generate ./... on Go 1.26.6, and internal/json and mcp on Go 1.25.0.

This covers every call site, including the one #1395 changes, and doesn't conflict with #1287.

Fixes #1453

@raveesh-me
raveesh-me force-pushed the perf/json-unmarshal-no-stream-buffer branch from 83dbad6 to 68e54b1 Compare October 25, 2026 21:49
Unmarshal and UnmarshalUseNumber wrapped data in a bytes.Reader and
decoded it with a json.Decoder, which copies its input into a read
buffer of at least 32 KiB on every call. Unmarshal runs several times
per request, so these buffers made up most of the memory allocated to
serve a tool call.

Decode in place instead: find the extent of the first JSON value with
the same parse the Decoder runs on its buffer, then decode just that
value with the same flags. Malformed input, the rare case, still goes
through a Decoder, so results are unchanged: the decoded value, trailing
data ignored, io.EOF on empty input, and each error by type and text.
One function, newDecoder, configures the Decoder; parseFlags gives
json.Parse the same rules, and a differential test and fuzz target
compare the two paths, so the rules cannot drift apart. Tests also check
that decoded values never share memory with the input, and that
UnmarshalUseNumber, which decodes every JSON-RPC envelope, matches field
names case-sensitively.

On an Apple M1 Max:

	name                  old time/op    new time/op    delta
	StreamableServing     368.7µs ± 2%   219.3µs ± 1%   -40.51%
	Unmarshal             4.015µs ± 1%   1.337µs ± 1%   -66.71%

	name                  old alloc/op   new alloc/op   delta
	StreamableServing     528.3KiB ± 0%  60.1KiB ± 1%   -88.62%
	Unmarshal             37.41KiB ± 0%  5.39KiB ± 0%   -85.59%

Fixes modelcontextprotocol#1453
@raveesh-me

Copy link
Copy Markdown
Author

A note on #1287, which moves internal/json to encoding/json/v2: the two are independent. This PR keeps the current dependency and only changes how Unmarshal reads its input. If #1287 lands first, I'll rebase. The tests here no longer import segmentio, so they carry over unchanged. With v2's decoder the per-call buffer is already small, so the code change would mostly fall away and the tests would remain.

While checking that, I found that #1287 makes UnmarshalUseNumber case-insensitive on current main, which matters for the JSON-RPC envelope since #1401. I've left the details on #1287. TestUnmarshalUseNumberCaseSensitivity in this PR catches it.

raveesh-me added a commit to raveesh-me/go-sdk that referenced this pull request Oct 25, 2026
raveesh-me added a commit to raveesh-me/dedo that referenced this pull request Oct 25, 2026
The MCP Go SDK decodes every message through a JSON decoder that
allocates at least 32 KiB per call, several times per tool call. The fix
is proposed upstream (modelcontextprotocol/go-sdk#1454); until a release
has it, dedo init and dedo upgrade replace the SDK release dedo is built
with by v1.8.0-dedo.1 on github.com/raveesh-me/go-sdk: v1.8.0 plus the
fix. The replace names the release, so a service that needs a later SDK
gets the later upstream release; a comment beside it says why. A later
dedo release sets sdkForkRelease to "" and dedo upgrade removes it.

dedo leaves a replace of the SDK it did not write alone. The examples use
the same tagged replace, and the release check stops if the tag is not
published or the examples name another.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NR3ete9fMFJAMqHnnCXzdz
@vsem-azamat

Copy link
Copy Markdown

Tested this against a Streamable HTTP gateway built on the SDK (v1.8.0), where we'd profiled the same 32 KiB decoder buffer as the top source of allocations: about 13 Unmarshal calls per proxied tools/call, roughly 450–500 KiB of garbage per call and 23–26% of CPU in GC.

With an equivalent change, allocations dropped to about 63 KiB per call and throughput went up 66–76% (4 clients, 20k calls after warm-up, median of 3 runs). The gateway's full test suite also passes against this branch at 923d798. Would be great to see this land.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

internal/json: Unmarshal allocates a 32 KiB stream buffer on every call

2 participants