Repository navigation
Do less repeated work when signing an S3 request - #50
Closed
tiptenbrink wants to merge 19 commits into
Closed
tiptenbrink wants to merge 19 commits into
tiptenbrink wants to merge 19 commits into
Conversation
The canonical request, both lists of signed names and the head now walk one SignedHeaders, so the fixed headers are sorted once and no 24-slot array is copied into each walk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The encoder writes the head once with a placeholder signature, hashes the path, query and signed header values from the head's own bytes, and then writes the signature in place. A head that does not fit its buffers is no longer signed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
WireRequest keeps the head as bytes, and url and headers check just the part they return, so a host that reads the head by range checks nothing. The encoders write timestamps as bytes rather than re-checking them on every use. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A copy_from_slice of a length known only at run time calls the C library's memcpy, and an encode makes about 30 of a few bytes each. Writer::push copies up to 32 bytes as two overlapping fixed-length copies instead, inlined, with the destination sliced to the length first so the compiler drops the bounds checks. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A canonical request reaches the hash in about 30 pieces of a few bytes, each a call through the provider's function pointer. Sum now copies pieces into a 64-byte buffer and hands the provider whole chunks, and a piece of a block or more directly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sha256Provider takes three functions and the client builds HMAC-SHA256 itself, so SigningKey can keep the hash states after the key's inner and outer padding blocks and start each signature from copies. Provider states must therefore be Copy: Sha256RustCrypto keeps its own state over sha2's compression function, compressing a full block where it is, and the crypto crate drops its hmac dependency. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ResponseHead::insert compared every name, case-insensitively, with up to 25 known names in turn, so each header a head does not keep ran through all of them. It now selects candidates by length, and compares one as two or three words with the case bit of each letter set. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
validate_key split the key on slashes to compare each segment with . and .., which cost about 900 instructions for a 45-byte key. It now looks only at a dot that starts a segment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SignedHeaders::iter merged the sorted fixed headers with the metadata on every step, polling the metadata iterator even when the request carries none. It now steps through the fixed headers directly then. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AsciiSet kept four 32-bit masks and tested ASCII first, several instructions a byte of every key a URL carries. It now keeps one flag per byte, with the bytes outside ASCII set, so a test is one load. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SigV4 folds runs of spaces in a header value, so the signer scanned each value byte by byte. The content hash is always text this crate formatted, a hex digest or a fixed word, so it is now a SignedValue::Formatted that the signer takes as it is. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An S3 head writes each field as name:value after a newline from the one before, which the spans leave out, so the signed headers after authorization are the canonical request's lines. The signer hashes them in one or two pieces rather than four per header, and per header only when a value needs folding. An Azure head, which signs nothing, keeps its layout. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
validate_key walked the key three times, for control bytes, for the segment count and for a segment that ends in a dot, which was about half of encoding an Azure GET. One loop now gathers all three, and the checks keep their order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
parse_content_range searched the value three times and parsed each number with checked arithmetic. A value of start-end/total in digits, as both services write it, is now read in one forward pass, and anything else falls back to the general reading. decimal skips the overflow checks for up to 19 digits, which cannot overflow. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
hex looked up each nibble of the canonical request's hash and of the signature in turn. A table of the two digits of every byte makes each byte one lookup. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The headers this crate names have a fixed order by name, so fixed_headers now yields them in it and SignedHeaders::of appends them; only the plan's extra headers are sorted in. A debug assertion keeps the order honest. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
U64Decimal divided by ten for each digit of a range's offsets and a content length. It now divides by a hundred and takes both digits from a table, and stays out of line: the compiler unrolls the digit loop, and inlined at each place that writes a number it made write_range alone about 2,400 instructions of machine code. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tiptenbrink
force-pushed
the
encode-once
branch
from
October 4, 2026 19:05
793cfd1 to
87fd863
Compare
Collaborator
Author
|
Superseded by #52, which includes these 19 commits unchanged, followed by the templates work, as one PR. |
tiptenbrink
added a commit
that referenced
this pull request
Oct 13, 2026
These are the 19 changes of #50, which lists each. S3 signing hashes its canonical form once and keeps its key's HMAC states for the day. Request heads are written in fewer passes and copies, and response heads are read with fewer comparisons. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Removes repeated and wasted work from encoding S3 and Azure requests and reading their response heads. Each change is one commit, and no commit regresses: each was measured against the one before it (below).
S3 signing
name:valueafter a\nfrom the one before, which the spans leave out. The signed headers are then the canonical request's lines, hashed in one or two pieces instead of four per header. An S3 head takes two more bytes per signed field; an Azure head is laid out as before.Sum, so the provider gets a few large updates.Sha256Provider::newtakesstart,updateandfinish, and the client builds HMAC-SHA256 itself. TheSha256trait requiresCopyand dropshmac.Sha256RustCryptokeeps its own state oversha2::block_api::compress256and compresses blocks in place, and the crypto crate no longer depends onhmac.x-amz-content-sha256, which the crate formats itself, without scanning it for spaces..and..key segments in one pass. Hex-encode digests from a table of digit pairs.Shared
memcpy.write_rangeabout 2,400 instructions of machine code.WireRequestchecks UTF-8 only in the part thaturl()orheaders()returns.Azure
Response head
ResponseHead::insertselects candidate names by length, then compares a name as two or three words with each letter's case bit set. In machine code it shrinks from 1,402 instructions to 204.Content-Range(start-end/total) in one forward pass, and parse decimals of up to 19 digits without overflow checks. Anything else falls back to the general reading.Measured with
perf statover a loop of each step: exact user-space instructions and cycles per call on a performance core, for the workload replay's ranged GET of an SST and a real Tigris answer to it:encode_get, ranged GETfrom_headers+accept_get_head)encode_get, bearer token of 1,826 charactersEach commit was measured the same way against the one before it. Each lowers the instructions of the path it changes. On paths it does not touch, counts move by at most about 50 instructions, which is what builds of identical code show here: the CHANGELOG-only commits move them as much.
Checked:
cargo test, clippy and docs with warnings denied, on the whole workspace with all featuresno-allocatorbuildThe pinned SigV4 signature tests are unchanged, on both providers. CHANGELOG entries are under
## Unreleased.