Repository navigation
[Feature]: Read a service's pages in memory close to the data's size #830
Description
Before you start
- I searched the existing issues and the roadmap and did not find a duplicate request.
What problem does this solve?
A service layer read page by page (OGC API Features since #519, ArcGIS since #823) uses far more memory than its data. Reading 100,000 parcels from a county's ArcGIS server peaked at 2.2 GB, and those parcels are 149 MB as GeoJSON. Fairfax County's 100,000 peaked at 1.1 GB, for 85 MB.
That multiple is what keeps the page cap at 100,000 (ogc::MAX_PAGED_FEATURES). It also makes a large read risky in the browser, where wasm memory tops out at 4 GB.
The likely cause is ogc::read_pages / Pages. It holds every page's features as serde_json::Value trees, so each coordinate is its own value. Then it writes one joined GeoJSON document, which layer.addDownload parses again into a dataset. That's three copies at once, one of them several times the size of the text.
Proposed solution
- Profile first. Find where the peak actually is: the
Valuetrees, the joined document, the second parse, or the dataset and drawing caches. Record the numbers in the issue. - Probably: keep each page's features as raw JSON text (
Box<serde_json::value::RawValue>; the workspace already enablesraw_value) instead ofValuetrees, and splice them into the joined document. Memory while reading would then be close to the data's own size. ArcGIS's "first features again" check can compare raw text. - Possibly, if the profile points there: build the dataset as pages arrive rather than writing and re-reading one document. That touches
layer.addDownload's bytes interface, so it's a bigger change and would only be done if step 2 isn't enough. - Then, as a separate decision, consider raising
MAX_PAGED_FEATURESfor the desktop. The download byte budgets (512 MB on the desktop, 256 MB in the browser) would remain the real limit.
It changes no data model, command, project format or dependency.
Estimated scope
Small to medium: rustgis-data (ogc.rs), plus a regression test that bounds memory, or at least the copies held, for a many-page read.
Alternatives considered
- Raising the cap without this. Memory grows about 15× the data with it, and the browser reaches its 4 GB limit at around 350,000 parcels.
- Reading large layers a view at a time ([Feature]: Read large ArcGIS layers live, a view at a time (an ArcGIS FeatureSource) #829, the ArcGIS
FeatureSourceproposal). That's the answer for layers too big to download, and it would make this matter less. But downloaded layers (small and medium ones, working offline) should still cost close to what they weigh.
I'm taking this one.
Activity
perf(data): hold paged service features as their JSON text
Mavrick-F commentedon Oct 24, 2026
Step 1, the profile. I replayed the saved 100,000-parcel layers offline as 50 ArcGIS pages of 2,000, running each step in its own process. Each figure is that step's peak resident memory above its starting point.
ogc::read_pages:Valuetrees, then the joined writeio::read_bytes: the joined GeoJSON into a datasetlayer.addDownloadThe peak was in the
Valuetrees, as suspected: about 8× the text. Step 2 (#841) keeps each feature as its raw text, which bringsread_pagesdown to +323 MB and +183 MB. A liveservice.addof Brentwood's parcels went from 1,270 MB peak to 576 MB.The peak is now in the dataset read and
layer.addDownload. Step 3 would remove only the joined text from it (about 1× the data), so I've left steps 3 and 4 as separate decisions.perf(data): hold paged service features as their JSON text (#841)
Mavrick-F commentedon Oct 24, 2026
Steps 1 and 2 are done in #841. Live,
service.addof 100,000 Brentwood parcels went from 1,270 MB peak to 576 MB, and the profile is in the comment above.layer.addDownload. If larger downloads become common, I'd first profile what adding the layer spends.MAX_PAGED_FEATURES): I've moved this to [Feature]: Read large ArcGIS layers live, a view at a time (an ArcGIS FeatureSource) #829, because the cap and the point where a layer goes live are one decision.Closing this as done.