Skip to content

[Feature]: Read large ArcGIS layers live, a view at a time (an ArcGIS FeatureSource) #829

Open

Description

@Mavrick-F

Before you start

  • I searched the existing issues and the roadmap and did not find a duplicate request.

What problem does this solve?

#764 (merged in #823) reads an ArcGIS FeatureServer or MapServer layer past its record limit by downloading it page by page, up to 100,000 features. That's enough for a small town's parcels or a county's zoning, but not for a large county. Fairfax County publishes 369,542 parcels on ArcGIS Online, and RustGIS reads the first 100,000 by object id, with a note. The map looks complete until you notice parcels missing, and they can be anywhere in the county.

Raising the cap doesn't solve it. Downloading a whole county to look at one neighbourhood is slow, costs memory (100,000 parcels peaked at 2.2 GB natively), and a browser tab's wasm memory tops out at 4 GB.

RustGIS already handles data bigger than memory: a FeatureSource answers "the features in this extent, at most this many", and a layer draws it a view at a time. GeoPackage, FlatGeobuf, GeoParquet, Shapefile, large GeoJSON files and the PostGIS, MySQL and SQL Server tables all work this way. A county parcel table in PostGIS can already be browsed in full. The same table published as an ArcGIS service can't.

Proposed solution

An ArcGIS FeatureSource in rustgis-data (source/arcgis.rs), for layers over a threshold. ArcGIS's query API already speaks the trait's language:

  • A view (read): query?geometry=<extent>&geometryType=esriGeometryEnvelope&inSR=4326&spatialRel=esriSpatialRelIntersects&resultRecordCount=<limit>&outSR=4326&f=geojson, paged by resultOffset in object-id order as in [Feature]: Read every feature of an ArcGIS FeatureServer or MapServer layer, past its record limit #764. When the limit is hit, the trait asks for an evenly spread sample rather than the first N. ArcGIS has no direct way to ask for that, so one option is to use the layer's maxRecordCountFactor or quantizationParameters for generalised drawing at small scales. To settle in review.
  • Every feature (visit_chunks): page through the whole layer a chunk at a time, so geoprocessing runs on a large layer with bounded memory.
  • The browser: follow the remote sources' pattern. A read returns remote::NEED until its pages are fetched, and remote::drive fetches them asynchronously.

Questions to settle before building:

  1. When a layer goes live. Above a feature count (the layer's returnCountOnly, one cheap request), always, or as the user's choice in Add From Service?
  2. Server load. Every pan becomes a query to someone's county server. That needs a debounce, a cache of extents already read, and a limit on concurrent requests per host (the tile fetcher already has one).
  3. Offline and saved projects. A live layer needs the server when a project reopens, as a PostGIS layer does. Should "make a local copy" be offered, using [Feature]: Read every feature of an ArcGIS FeatureServer or MapServer layer, past its record limit #764's download?
  4. Selection, the attribute table and editing. These assume rows in memory. They'd behave the way they do for PostGIS layers today, whatever that is for each.
  5. Tokens. These come from the address today. With the secret store (Solid ground) and ArcGIS sign-in (Data at the source), the source should take its credentials from there.

This sits in the roadmap's Data at the source theme, next to the ArcGIS sign-in item and deniial00's connections work (#699, #701, #705). I'd like a maintainer's view on the questions above before writing much code.

Estimated scope

Large. rustgis-data (a new source), engine (adding a service layer as a source above the threshold), probably ui-egui (the choice in Add From Service), and tests against the shared fixture server. It adds no dependency or crate. It may affect the .rgis format, depending on how a live ArcGIS layer is saved: possibly as the existing source reference, possibly with new fields.

Alternatives considered

Activity

  1. Mavrick-F commented on Oct 24, 2026

    @Mavrick-F
    MemberAuthor

    An update on the memory figure above. #841 (from #830) keeps paged features as their JSON text instead of serde_json::Value trees. The 100,000 Brentwood parcels that peaked at 2.2 GB now read in 576 MB natively (155 MB of GeoJSON, service.add, release build). A downloaded layer now costs about 3.7× its data. Most of that is the dataset and the layer, not the paging.

    That changes one part of the case against raising the cap. Holding every feature is now affordable well past 100,000. The download budgets (512 MB on the desktop, 256 MB in the browser) would bind at roughly 330,000 and 165,000 parcels of Brentwood's size. At 3.7× those are about 1.9 GB and 950 MB. Those peaks are estimates from the 100,000-feature ratio, not measured. The other part of the case still stands: a bigger cap still downloads the whole county to look at one neighbourhood, and the blocking service.add stops at a minute (Fairfax took 54 s for 100,000).

    So I think MAX_PAGED_FEATURES and question 1 here ("when a layer goes live") are the same decision. The cap is the largest layer RustGIS downloads, and above it the layer would go live. One way to settle both:

    • downloaded up to the byte budget (the cap raised on the desktop to about where the budget binds, the browser unchanged or raised less);
    • live above that, as this issue proposes.

    If live layers aren't wanted, or are a long way off, a modest raise of the cap on its own would be a small separate change. I'd rather not make it before hearing which way you'd like this to go, since it settles part of this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions