Tracking: large data (layers over 2 million features, slow hillshade and GeoTIFF writing) #1111

On real-sized data RustGIS either refuses or is much slower than it needs to be. Measured on one US state (3,212,306 building footprints, 4,562 census block groups, a 9667 × 11947 DEM; rustgis-cli at d58cb8a, i7-6700, single runs, ±15%), with the time an established tool takes on the same files in brackets. This issue tracks fixing that, and has one decision to make first.

What's wrong now

• Layers over 2,000,000 features can't be exported, reprojected or used in tools. MAX_LIMIT (crates/data/src/source/mod.rs) refuses them: all 3.2M buildings can't be converted, and [Bug]: A source over the 2,000,000-feature limit is silently replaced by a sample when read without streaming: exports and tools run on half the data #1108 is worse, a silent half sample. It's a count, not memory: it refuses 3.2M small polygons that fit in about 1.2 GB, but doesn't stop 2M large ones from exhausting memory.
• Exports hold the whole layer, sometimes twice. Converting 1.79M polygons to FlatGeobuf takes 22.7 s and 1.2 GB (8.3 s, 0.26 GB). Reprojection clones the Dataset, and the FlatGeobuf writer builds the whole file in a Vec<u8>.
• Hillshade is slow. It takes 59 s (7 s). gradient() in crates/analysis/src/rasterops.rs looks up the coordinate system for every cell, 115M times here, and runs five trig functions per cell. Its output is written as Float32 although the values are whole numbers 0–255.
• Writing GeoTIFFs is slow. The COG export takes 43 s (16–22 s), nearly all of it Deflate at level 6 through miniz_oxide, about 18 MB/s (crates/data/src/raster/cogwrite.rs).
• Summarize Within compares every point with every polygon ([Bug]: Summarize Within (countPoints) tests every point against every polygon: 1.8 million points in 4,562 polygons run for over 28 minutes #1105): over 28 minutes for 1.79M points (15 s).

Needs a decision: tools on layers bigger than memory

Steps 2 and 3 below happen either way: exports stream, and the count becomes a memory budget. What should tools do past the budget?

• A. Streaming tools (large). Per-feature tools (reproject, buffer, calculate geometry) work a chunk at a time. Tools that compare two layers (count points, spatial join, clip, select by location) index the smaller layer and stream the larger. Outputs go to temporary files added as streamed layers. That means tools on any size, but every tool gets a second execution path, and temporary-file outputs touch undo, project saving and the web app, which has no temporary files.
• B. Stay in memory (small to very large). Tools keep working on a whole layer under the budget, and past it the answer is "clip or filter first". There is one execution path, but a ceiling, lower in the web app. Small changes (boxing the common one-ring polygon, reserving from the known count) might take a feature from about 355 to 250 bytes. A flat, columnar geometry model would roughly halve memory, but touches every crate.

Which fits where RustGIS is heading?

The steps also assume these defaults. Say so if any is wrong:

• hillshade writes Byte;
• threads on the desktop only, with the web app single-threaded as the renderer already is (threads_for);
• a budget of 4 GiB on the desktop and 1 GiB in the web app, adjustable;
• COGs use the floating-point predictor (TIFF Predictor 3), which GDAL, QGIS and the tiff crate read.

Plan

1. Correctness (small, independent)

• Refuse instead of sampling ([Bug]: A source over the 2,000,000-feature limit is silently replaced by a sample when read without streaming: exports and tools run on half the data #1108): an incomplete read in source::materialize becomes an error, checked from the known count before reading.
• Index Summarize Within ([Bug]: Summarize Within (countPoints) tests every point against every polygon: 1.8 million points in 4,562 polygons run for over 28 minutes #1105): the polygons in ExtentIndex, each point tested against its candidates only.

2. Stream exports and reprojection (medium)

• visit_chunks for GeoPackage, FlatGeobuf and GeoParquet. Shapefile already has it (feat(render): stream complete Shapefile layers in ordered chunks #681).
• A writer per format that takes chunks, and share.exportLayer on a streamed layer reading chunk → reproject → writer. Memory goes from 1.2 GB to about 0.3 GB, and the limit no longer applies to exports.
• FlatGeobuf in the web app: its index needs a temporary file, so write it unindexed there.

3. A memory budget instead of MAX_LIMIT (small, after 2)

• load_all counts bytes as it reads chunks and refuses past the budget, with try_reserve for the large vectors.

4. Terrain kernels (small, independent)

• Hillshade, slope and aspect as row passes: constants computed once, three row slices walked with windows(3) and zip, with no-data and edges on the existing path. A standalone test gave identical output on all 115M cells, 21 s → 4 s on one thread, in the web app too.
• Hillshade writes Byte.

5. GeoTIFF writing (small to medium, independent)

• Deflate level 1 with Predictor 3 in cogwrite.rs: 157 MB/s and 25% smaller on the DEM's tiles; the COG export goes to about 12 s.
• The untiled export through the in-house writer: tiff 0.11.3 can't write Predictor 3, and its speed today depends on zip turning on zlib-rs.

6. Threads on the desktop (medium, after 4 and 5)

• Raster kernels in row bands, COG tiles compressed in parallel: about 3× on 4 cores.

7. Tools past the budget (large or skipped, after the decision)

• A: streaming tools, Summarize Within first. B: the small memory savings.

Related

#1105 and #1108 are filed separately because they're bugs whatever is decided here. Step 2 builds on #681's chunked reading. #1065 (layers read in place are read twice on every render) is about drawing, not tools, so it stays separate.