Skip to content

feat: page entities with StableBTreeMap.range instead of probing every id - #15

Merged
deucalioncodes merged 1 commit into
mainfrom
feat/range-paging
Sep 16, 2026
Merged

deucalioncodes merged 1 commit into
mainfrom
feat/range-paging

Conversation

@deucalioncodes

Copy link
Copy Markdown
Member

Closes #14
Companion to smart-social-contracts/basilisk#57 (StableBTreeMap.range).

What

Listing paths (load_some, find, instances, rebuild_field_index) read a page of entities with one ordered range call per digit-count bucket instead of probing Type@1..Type@max_id with three get()s per id. Deleted ids cost nothing; cost tracks rows returned, not the highest id ever issued.

File Change
storage.py Optional Storage.range(start, end=None, limit); supports_range(); storage_sort_key() and MemoryStorage.range that order keys exactly like the canister (Basilisk str keys sort shorter-first, then bytewise)
db_engine.py Database.max_id(), Database.load_page(type, from_id, count, max_id=None) -> [(id, data)]. Range path walks [Type@{lo}, Type@{':'*d}) per digit count (tightened to Type@{max_id+1} in the last bucket); custom string ids inside a bucket are skipped and the range resumed at the next possible sequential id so pages stay full. Probe fallback when the backend has no range
entity.py load_some → load_page (identity map first, shared _hydrate() with load); rebuild_field_index batches over live entities with the same cursor contract; __init__ skips the max-id counter re-read when _loaded=True; per-row INFO log removed
tests/entrypoint*.sh Download the template attached to the installed Basilisk release (v{version}) with the -ic1 rolling asset as fallback, instead of the stale cpython-wasm-3.13.0 asset. Without this the on-chain tests would never exercise range
tests MemoryStorage.range ordering; load_page buckets/gaps/custom ids ("1x", "007", "abc")/other types/max_id; load_some over ids 1–150 with deletions across three key lengths, cursor walk, custom ids at page boundaries, and a probe-only Storage giving identical results

No key format change and no migration. Type@id keys are untouched, so deployed canisters need nothing on upgrade.

Measured (MemoryStorage instrumented, 1000 Zones, every other id in the top half deleted)

Operation Before After
load_some(1, 50) cold 150 reads 1 read + 2 range calls (50 rows)
load_some(501, 50) cold, 50 % deleted region 250 reads 1 read + 1 range call
load_some(1, 1000) / instances() / find() cold 2 751 reads 1–2 reads + 4 range calls (750 rows)
load("42") cold 2 reads 1 read
Local CPU, 1000-row scan 28.9 ms 17.0 ms

Verification done locally

  • tests/run_test.sh equivalent: all 18 local modules pass (16 in the script + context, hooks); ./run_linters.sh clean (black, isort, flake8, mypy).
  • Deployed tests/src/main.py on a local icp replica with a Basilisk template built from basilisk#57: database, entity, indexed_fields, relationships, enhanced_relations, serialization, namespaces, alias_and_properties, mixins, properties, audit, example_1, example_2 all return 0; entity rows identical before/after an upgrade.
  • Ran the same modules on the current published template (no range) — the fallback path is what CI's Test on IC exercises until Basilisk ships the new template.

Sequencing

  1. Merge basilisk#57, let test-all.yml rebuild the template, cut Basilisk minor release.
  2. Re-run this PR's CI (its Docker IC test does pip install --upgrade ic-basilisk and now fetches the v{version} template), merge, cut ic-python-db minor release via release.yml.

…every id

load_some (and through it find, instances, rebuild_field_index) probed
Type@1..Type@max_id with get(), paying 3 storage reads per id (counter
re-read in the loop, counter re-read in __init__, the row itself) plus an
INFO log per row, and paying for deleted ids forever.

- storage.py: optional Storage.range(start, end, limit), supports_range(),
  and MemoryStorage.range ordered like the canister (str keys sort
  shorter-first, then bytewise, matching Basilisk's key encoding)
- db_engine.py: Database.max_id() and Database.load_page(): one range read
  per digit-count bucket ("Type@10".."Type@::"), custom string ids inside a
  bucket skipped and the range resumed so pages stay full; falls back to
  get()-probing when the backend has no range (older templates, dict-backed
  test doubles)
- entity.py: load_some via load_page with identity-map check and a shared
  _hydrate() (also used by load); rebuild_field_index batches over live
  entities; skip the max-id counter re-read when hydrating (_loaded=True);
  per-row INFO log removed
- tests: MemoryStorage.range order, load_page buckets/gaps/custom ids/max_id
  bound, load_some across three key lengths with deletions, custom ids at
  page boundaries, probe-only Storage fallback
- tests/entrypoint*.sh: download the template of the installed Basilisk
  release (v{version}) instead of the stale cpython-wasm-3.13.0 asset

No key format change, no migration: Type@id keys are unchanged.

Measured (MemoryStorage instrumented, 1000 ids / 250 deleted):
  load_some(1,50) cold        150 reads -> 1 read + 2 range calls
  load_some(501,50) cold      250 reads -> 1 read + 1 range call
  load_some(1,1000)/instances 2751 reads -> 2 reads + 4 range calls
  load("42") cold             2 reads -> 1 read

Closes #14
Depends on smart-social-contracts/basilisk#57 for the on-chain fast path.
@deucalioncodes
deucalioncodes marked this pull request as ready for review September 16, 2026 10:24
@deucalioncodes
deucalioncodes merged commit 8a78fdf into main Sep 16, 2026
10 checks passed
@deucalioncodes
deucalioncodes deleted the feat/range-paging branch September 16, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Paging scans probe every id 1..max_id (3 reads each): read pages with StableBTreeMap.range

1 participant