Repository navigation
feat: page entities with StableBTreeMap.range instead of probing every id - #15
Merged
Merged
Conversation
…every id
load_some (and through it find, instances, rebuild_field_index) probed
Type@1..Type@max_id with get(), paying 3 storage reads per id (counter
re-read in the loop, counter re-read in __init__, the row itself) plus an
INFO log per row, and paying for deleted ids forever.
- storage.py: optional Storage.range(start, end, limit), supports_range(),
and MemoryStorage.range ordered like the canister (str keys sort
shorter-first, then bytewise, matching Basilisk's key encoding)
- db_engine.py: Database.max_id() and Database.load_page(): one range read
per digit-count bucket ("Type@10".."Type@::"), custom string ids inside a
bucket skipped and the range resumed so pages stay full; falls back to
get()-probing when the backend has no range (older templates, dict-backed
test doubles)
- entity.py: load_some via load_page with identity-map check and a shared
_hydrate() (also used by load); rebuild_field_index batches over live
entities; skip the max-id counter re-read when hydrating (_loaded=True);
per-row INFO log removed
- tests: MemoryStorage.range order, load_page buckets/gaps/custom ids/max_id
bound, load_some across three key lengths with deletions, custom ids at
page boundaries, probe-only Storage fallback
- tests/entrypoint*.sh: download the template of the installed Basilisk
release (v{version}) instead of the stale cpython-wasm-3.13.0 asset
No key format change, no migration: Type@id keys are unchanged.
Measured (MemoryStorage instrumented, 1000 ids / 250 deleted):
load_some(1,50) cold 150 reads -> 1 read + 2 range calls
load_some(501,50) cold 250 reads -> 1 read + 1 range call
load_some(1,1000)/instances 2751 reads -> 2 reads + 4 range calls
load("42") cold 2 reads -> 1 read
Closes #14
Depends on smart-social-contracts/basilisk#57 for the on-chain fast path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14
Companion to smart-social-contracts/basilisk#57 (
StableBTreeMap.range).What
Listing paths (
load_some,find,instances,rebuild_field_index) read a page of entities with one orderedrangecall per digit-count bucket instead of probingType@1..Type@max_idwith threeget()s per id. Deleted ids cost nothing; cost tracks rows returned, not the highest id ever issued.storage.pyStorage.range(start, end=None, limit);supports_range();storage_sort_key()andMemoryStorage.rangethat order keys exactly like the canister (Basiliskstrkeys sort shorter-first, then bytewise)db_engine.pyDatabase.max_id(),Database.load_page(type, from_id, count, max_id=None) -> [(id, data)]. Range path walks[Type@{lo}, Type@{':'*d})per digit count (tightened toType@{max_id+1}in the last bucket); custom string ids inside a bucket are skipped and the range resumed at the next possible sequential id so pages stay full. Probe fallback when the backend has norangeentity.pyload_some→load_page(identity map first, shared_hydrate()withload);rebuild_field_indexbatches over live entities with the same cursor contract;__init__skips the max-id counter re-read when_loaded=True; per-row INFO log removedtests/entrypoint*.shv{version}) with the-ic1rolling asset as fallback, instead of the stalecpython-wasm-3.13.0asset. Without this the on-chain tests would never exerciserangeMemoryStorage.rangeordering;load_pagebuckets/gaps/custom ids ("1x","007","abc")/other types/max_id;load_someover ids 1–150 with deletions across three key lengths, cursor walk, custom ids at page boundaries, and a probe-onlyStoragegiving identical resultsNo key format change and no migration.
Type@idkeys are untouched, so deployed canisters need nothing on upgrade.Measured (
MemoryStorageinstrumented, 1000Zones, every other id in the top half deleted)load_some(1, 50)coldload_some(501, 50)cold, 50 % deleted regionload_some(1, 1000)/instances()/find()coldload("42")coldVerification done locally
tests/run_test.shequivalent: all 18 local modules pass (16 in the script +context,hooks);./run_linters.shclean (black, isort, flake8, mypy).tests/src/main.pyon a localicpreplica with a Basilisk template built from basilisk#57:database,entity,indexed_fields,relationships,enhanced_relations,serialization,namespaces,alias_and_properties,mixins,properties,audit,example_1,example_2all return0; entity rows identical before/after an upgrade.range) — the fallback path is what CI'sTest on ICexercises until Basilisk ships the new template.Sequencing
test-all.ymlrebuild the template, cut Basilisk minor release.pip install --upgrade ic-basiliskand now fetches thev{version}template), merge, cut ic-python-db minor release viarelease.yml.