refactor(rpc/get_account): batch storage map proofs by kkovaacs · Pull Request #2439 · 0xMiden/node

kkovaacs · 2026-08-05T08:35:39Z

Summary

GetAccount now returns a PartialSmt with all requested key-value pairs.

A request for explicit map keys always returns one partial SMT covering those keys.
Partial SMTs are scoped to one storage map at one block. Proofs from different maps or roots are never merged.
The response carries the original, unhashed StorageMapKeys. The SMT contains hashed keys, so the raw keys cannot be recovered from the tree.
Values are not duplicated outside the SMT. A client obtains a value by hashing the raw key and calling PartialSmt::get_value().
too_many_entries, all_entries, and partial_map become mutually exclusive results.
The removed entries_with_proofs field number and name are reserved. Even though compatibility is intentionally broken, reusing the field could make an old client try to decode the new length-delimited message as the old one.
The compact protobuf representation mirrors miden_crypto::merkle::smt::UniqueNodes. Do not encode PartialSmt as an opaque byte string.

Closes #617

Changelog

[[entry]]
scope       = "rpc"
impact      = "breaking"
description = "Storage map proofs for multiple keys in `AccountResponse` are now represented as a batched `PartialSmt`."

GetAccount now returns a PartialSmt with all requested key-value pairs. - A request for explicit map keys always returns one partial SMT covering those keys. - Partial SMTs are scoped to one storage map at one block. Proofs from different maps or roots are never merged. - The response carries the original, unhashed `StorageMapKey`s. The SMT contains hashed keys, so the raw keys cannot be recovered from the tree. - Values are not duplicated outside the SMT. A client obtains a value by hashing the raw key and calling `PartialSmt::get_value()`. - `too_many_entries`, `all_entries`, and `partial_map` become mutually exclusive results. - The removed `entries_with_proofs` field number and name are reserved. Even though compatibility is intentionally broken, reusing the field could make an old client try to decode the new length-delimited message as the old one. - The compact protobuf representation mirrors `miden_crypto::merkle::smt::UniqueNodes`. Do not encode `PartialSmt` as an opaque byte string.

kkovaacs · 2026-08-05T14:03:07Z

Looks like there is a bug in PartialSmt::from_unique_nodes() that might prevent us from taking full advantage of this optimization: 0xMiden/miden-vm#3470

bobbinth · 2026-08-06T00:50:32Z

Looks like there is a bug in PartialSmt::from_unique_nodes() that might prevent us from taking full advantage of this optimization: 0xMiden/miden-vm#3470

Once 0xMiden/miden-vm#3471 is merged, we should be able to do a patch release and it'll flow through to here immediately.

kkovaacs · 2026-08-06T12:05:01Z

@igamigo Do you have an estimate on how many storage keys a client usually requests from a storage map in a single GetAccount request?

As far as I understand there are two cases where the client is requesting specific keys from a storage map:

When preparing transactions involving public foreign accounts: the client fetches all keys explicitly referenced in AccountStorageRequirements when adding the foreign account(s) to the TransactionRequestBuilder.
Lazily during transaction execution: when a required storage-map witness is absent locally and from the in-memory cache.

As far as I understand in the lazy (execution) use case the client is always requesting a single key from a single storage slot. There is nothing to be gained in that case from the optimization this PR implements.

In the first case though: how much this optimization helps depends on the number of the storage map keys typically required. Do we have an estimate on how many keys we're expecting there?

CC @Mirko-von-Leipzig

igamigo · 2026-08-06T14:46:04Z

@igamigo Do you have an estimate on how many storage keys a client usually requests from a storage map in a single GetAccount request?

Not really, and as we discussed the other day, this is very transaction/smart contract-specific. It's very hard to generalize so I would not let any specific usecase inform any limits here. For example, I think Zoro is the only user who has hit the 64 key limit before, and they worked around it by sharding data in different accounts AFAIK. They were going to try locally following the foreign account as an alternative, but surely they would benefit from a higher limit.

Asset callbacks will probably be the dominant FPI usecase but at first sight I don't think you would need to fetch a large amount of keys there on average (not sure though).

Mirko-von-Leipzig · 2026-08-18T10:24:04Z

+
+        let proof = partial_smt.open(&map_key.hash().as_word()).map_err(|err| {
+            RpcError::InvalidResponse(format!(
+                "response did not track the requested storage map key: {err}"


{err} likely truncates the error

Why? err here is a MerkleError (from miden-crypto) that has a thiserror-derived Display implementation that seems to contain reasonably useful information.

I just mean that it relies on the underlying error to have a report style display implemented.

…-batched-storage-proofs

Now that the underlying bug has been fixes in `miden-crypto`.

kkovaacs · 2026-08-18T11:55:43Z

FYI: I've removed that workaround for PartialSmt reconstruction now that we're on miden-crypto == 0.29.1 that does have the bugfix.

igamigo

LGTM! Apologies for the delay in the review. In terms of whether we can merge this for 0.16, I don't see why not. This is mostly an optimization but it does change up things a bit so maybe the sooner the better. Not a strong opinion either way though.

igamigo · 2026-08-22T22:26:52Z

+    // Leaves for which only the hash is known.
+    repeated IndexedDigest value_only_leaves = 4;


nit (feel free to disregard): Would hash_only_leaves be a better name? value_only intuitively makes me think that the original value is there.

igamigo · 2026-08-22T22:37:20Z

+        let map_root = match self.forest.root_info(tree) {
+            RootInfo::LatestVersion(root) | RootInfo::HistoricalVersion(root) => root,
+            RootInfo::Missing => return None,
+        };

        let proofs = Result::from_iter(raw_keys.iter().map(|raw_key| {
            let key_hashed = raw_key.hash().into();
            self.forest.open(tree, key_hashed).map_err(Self::map_forest_error)
        }));

-        Some(proofs.map(|proofs| AccountStorageMapDetails::from_proofs(slot_name, proofs)))
+        Some(proofs.and_then(|proofs| {
+            AccountStorageMapDetails::from_proofs(slot_name, map_root, raw_keys.to_vec(), proofs)
+        }))


I guess it would be nice to have an LargeSmtForest::open_many() or something similar here. This is probably allocating a lot more than it needs to, and probably re-hashing stuff that's been validated already.

In this same line, can we pass raw_keys as a Vec<> here to avoid the to_vec()? It sounds unlikely that it would be used after having built the response, so maybe there's no need to pass a slice. With 64 keys being the limit I doubt this is bad in any way so this is definitely mostly a nit.

igamigo mentioned this pull request Aug 5, 2026

Update node to 0.16 0xMiden/rust-sdk#2360

Open

kkovaacs force-pushed the krisztian/get-account-batched-storage-proofs branch from 5ae4953 to 1a99dc6 Compare August 5, 2026 14:01

fix: formatting

946e7e2

kkovaacs requested review from Mirko-von-Leipzig, bobbinth and igamigo August 5, 2026 15:33

kkovaacs marked this pull request as ready for review August 5, 2026 15:34

Mirko-von-Leipzig approved these changes Aug 18, 2026

View reviewed changes

kkovaacs added 2 commits August 18, 2026 13:34

Merge remote-tracking branch 'origin/next' into krisztian/get-account…

49b9a28

…-batched-storage-proofs

fix(proto/merkle): remove hack around UniqueNodes reconstruction

5e21753

Now that the underlying bug has been fixes in `miden-crypto`.

Mirko-von-Leipzig approved these changes Aug 18, 2026

View reviewed changes

igamigo approved these changes Aug 22, 2026

View reviewed changes

Navigation Menu

Uh oh!

refactor(rpc/get_account): batch storage map proofs - #2439

refactor(rpc/get_account): batch storage map proofs#2439
kkovaacs wants to merge 4 commits into
nextfrom
krisztian/get-account-batched-storage-proofs

kkovaacs commented Aug 5, 2026 •

edited

Loading

Uh oh!

kkovaacs commented Aug 5, 2026 •

edited

Loading

Uh oh!

bobbinth commented Aug 6, 2026 •

edited

Loading

Uh oh!

kkovaacs commented Aug 6, 2026

Uh oh!

igamigo commented Aug 6, 2026 •

edited

Loading

Uh oh!

Mirko-von-Leipzig Aug 18, 2026

Uh oh!

kkovaacs Aug 18, 2026

Uh oh!

Mirko-von-Leipzig Aug 18, 2026

Uh oh!

kkovaacs commented Aug 18, 2026

Uh oh!

igamigo left a comment

Uh oh!

igamigo Aug 22, 2026

Uh oh!

igamigo Aug 22, 2026

Uh oh!

igamigo Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// Leaves for which only the hash is known.
		repeated IndexedDigest value_only_leaves = 4;

Uh oh!

Conversation

kkovaacs commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changelog

Uh oh!

kkovaacs commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bobbinth commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kkovaacs commented Aug 6, 2026

Uh oh!

igamigo commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mirko-von-Leipzig Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

kkovaacs Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Mirko-von-Leipzig Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

kkovaacs commented Aug 18, 2026

Uh oh!

igamigo left a comment

Choose a reason for hiding this comment

Uh oh!

igamigo Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

igamigo Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

igamigo Aug 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kkovaacs commented Aug 5, 2026 •

edited

Loading

kkovaacs commented Aug 5, 2026 •

edited

Loading

bobbinth commented Aug 6, 2026 •

edited

Loading

igamigo commented Aug 6, 2026 •

edited

Loading