Repository navigation
[Feature]: Read large ArcGIS layers live, a view at a time (an ArcGIS FeatureSource) #829
Description
Before you start
- I searched the existing issues and the roadmap and did not find a duplicate request.
What problem does this solve?
#764 (merged in #823) reads an ArcGIS FeatureServer or MapServer layer past its record limit by downloading it page by page, up to 100,000 features. That's enough for a small town's parcels or a county's zoning, but not for a large county. Fairfax County publishes 369,542 parcels on ArcGIS Online, and RustGIS reads the first 100,000 by object id, with a note. The map looks complete until you notice parcels missing, and they can be anywhere in the county.
Raising the cap doesn't solve it. Downloading a whole county to look at one neighbourhood is slow, costs memory (100,000 parcels peaked at 2.2 GB natively), and a browser tab's wasm memory tops out at 4 GB.
RustGIS already handles data bigger than memory: a FeatureSource answers "the features in this extent, at most this many", and a layer draws it a view at a time. GeoPackage, FlatGeobuf, GeoParquet, Shapefile, large GeoJSON files and the PostGIS, MySQL and SQL Server tables all work this way. A county parcel table in PostGIS can already be browsed in full. The same table published as an ArcGIS service can't.
Proposed solution
An ArcGIS FeatureSource in rustgis-data (source/arcgis.rs), for layers over a threshold. ArcGIS's query API already speaks the trait's language:
- A view (
read):query?geometry=<extent>&geometryType=esriGeometryEnvelope&inSR=4326&spatialRel=esriSpatialRelIntersects&resultRecordCount=<limit>&outSR=4326&f=geojson, paged byresultOffsetin object-id order as in [Feature]: Read every feature of an ArcGIS FeatureServer or MapServer layer, past its record limit #764. When the limit is hit, the trait asks for an evenly spread sample rather than the first N. ArcGIS has no direct way to ask for that, so one option is to use the layer'smaxRecordCountFactororquantizationParametersfor generalised drawing at small scales. To settle in review. - Every feature (
visit_chunks): page through the whole layer a chunk at a time, so geoprocessing runs on a large layer with bounded memory. - The browser: follow the remote sources' pattern. A read returns
remote::NEEDuntil its pages are fetched, andremote::drivefetches them asynchronously.
Questions to settle before building:
- When a layer goes live. Above a feature count (the layer's
returnCountOnly, one cheap request), always, or as the user's choice in Add From Service? - Server load. Every pan becomes a query to someone's county server. That needs a debounce, a cache of extents already read, and a limit on concurrent requests per host (the tile fetcher already has one).
- Offline and saved projects. A live layer needs the server when a project reopens, as a PostGIS layer does. Should "make a local copy" be offered, using [Feature]: Read every feature of an ArcGIS FeatureServer or MapServer layer, past its record limit #764's download?
- Selection, the attribute table and editing. These assume rows in memory. They'd behave the way they do for PostGIS layers today, whatever that is for each.
- Tokens. These come from the address today. With the secret store (Solid ground) and ArcGIS sign-in (Data at the source), the source should take its credentials from there.
This sits in the roadmap's Data at the source theme, next to the ArcGIS sign-in item and deniial00's connections work (#699, #701, #705). I'd like a maintainer's view on the questions above before writing much code.
Estimated scope
Large. rustgis-data (a new source), engine (adding a service layer as a source above the threshold), probably ui-egui (the choice in Add From Service), and tests against the shared fixture server. It adds no dependency or crate. It may affect the .rgis format, depending on how a live ArcGIS layer is saved: possibly as the existing source reference, possibly with new fields.
Alternatives considered
- Raising [Feature]: Read every feature of an ArcGIS FeatureServer or MapServer layer, past its record limit #764's 100,000 cap. This moves the limit without removing it, and every feature still has to be held.
- Joining pages with less memory. This is a separate, smaller issue ([Feature]: Read a service's pages in memory close to the data's size #830). It's worth doing for downloaded layers either way, but it still downloads everything.
- Drawing the MapServer's
exportimages. RustGIS already offers a dynamic map's layers as images (layers=show:<id>). They show the whole county but have no attributes and no analysis.
Activity
Mavrick-F commentedon Oct 24, 2026
An update on the memory figure above. #841 (from #830) keeps paged features as their JSON text instead of
serde_json::Valuetrees. The 100,000 Brentwood parcels that peaked at 2.2 GB now read in 576 MB natively (155 MB of GeoJSON,service.add, release build). A downloaded layer now costs about 3.7× its data. Most of that is the dataset and the layer, not the paging.That changes one part of the case against raising the cap. Holding every feature is now affordable well past 100,000. The download budgets (512 MB on the desktop, 256 MB in the browser) would bind at roughly 330,000 and 165,000 parcels of Brentwood's size. At 3.7× those are about 1.9 GB and 950 MB. Those peaks are estimates from the 100,000-feature ratio, not measured. The other part of the case still stands: a bigger cap still downloads the whole county to look at one neighbourhood, and the blocking
service.addstops at a minute (Fairfax took 54 s for 100,000).So I think
MAX_PAGED_FEATURESand question 1 here ("when a layer goes live") are the same decision. The cap is the largest layer RustGIS downloads, and above it the layer would go live. One way to settle both:If live layers aren't wanted, or are a long way off, a modest raise of the cap on its own would be a small separate change. I'd rather not make it before hearing which way you'd like this to go, since it settles part of this issue.