Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

First public release (`0.2.0`). DisSCube is a declarative spatial layer for
dynamic models: it aligns heterogeneous geospatial sources onto a common grid,
derives variables with registered operators, and catalogs every result with a
`spec_hash` so a derived layer can be traced back to exactly how it was made.
### Changed
- **Slimmed `examples/` and migrated real-data cases**: Real-world datasets, bundled GIS assets (~12 MB), and TerraME parity benchmarks were moved to the dedicated [LambdaGeo/disscube-recipes](https://github.com/LambdaGeo/disscube-recipes) repository (`cases/terrame_fill`, `cases/ilha_maranhao`, `cases/prodes_br163`). The core `examples/` directory now contains strictly lightweight, offline, self-contained examples with tests for both the Python API (`01_quickstart.py`, `02_vector_drivers.py`, `03_time_series.py`) and declarative TOML pipelines (`examples/pipelines/quickstart.toml`).

### Added
- **Declarative derivations.** A `Derivation` names a source, a target grid and
Expand Down
54 changes: 21 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,63 +117,51 @@ in one TOML file, the counterpart of a TerraME `fill` script, and run from the
command line:

```bash
disscube validate examples/pipelines/itaituba_fill.toml # no downloads
disscube run examples/pipelines/itaituba_fill.toml --workspace outputs/itaituba
disscube validate examples/pipelines/quickstart.toml # no downloads
disscube run examples/pipelines/quickstart.toml --workspace outputs/quickstart
```

```toml
schema = 1
name = "Quickstart"

[grid]
name = "ilha_do_maranhao"
bbox = [-44.35, -2.62, -44.20, -2.47] # WGS84 → BDC Albers grid
name = "demo/300m"
crs = "EPSG:31983"
bbox = [570000.0, 9708000.0, 582000.0, 9720000.0]
resolution = 300

[[source]]
id = "lulc_{year}"
type = "mapbiomas"
years = [2000, 2020]
id = "landuse"
type = "file"
path = "../data/quickstart/landuse.tif"
crs = "EPSG:31983"

[[derive]]
target = "urban_pct"
source = "lulc_{year}"
target = "forest_pct"
source = "landuse"
operator = "percentage"
class_code = 24
class_code = 3
```

See [`docs/guides/pipeline_files.md`](docs/guides/pipeline_files.md) and
[`examples/pipelines/`](examples/pipelines/).

## Examples

[`examples/`](examples/) has runnable scripts; 01–06 need no downloads and run
in seconds:

- **01–03, synthetic data** — a quickstart with raster operators, vector
drivers, and time series handed off to DisSModel.
- **04–06, real data** — the three *Fill* examples shipped with TerraME
(Itaituba, Emas National Park, Brazilian Amazon), derived with DisSCube and
compared cell by cell with TerraME's own output. Data are bundled in
[`examples/data/terrame/`](examples/data/terrame/).
- **07, Brazil Data Cube** — reads the Landsat 16-day data cube over Ilha do
Maranhão through the BDC STAC catalog (only the pixels of the area are
fetched) and derives NDVI, MNDWI and open-water drivers on a 300 m grid
snapped to the BDC Albers mesh. Needs network and `pip install -e ".[bdc]"`;
with `--offline` it runs on a synthetic stand-in.
- **08, MapBiomas** — land use of Ilha do Maranhão in 2000 and 2020 (Collection
11, 30 m), read straight from the national files and aggregated on the same
grid as 07 (`majority`, urban/forest/mangrove `percentage`), then handed to
DisSModel with `to_lucc_data()`. Needs network; `--offline` runs a stand-in.
- **09, PRODES** — deforestation in the LuccME Lab15 area (south of Santarém,
Pará) at the end of 2008, 2016 and 2024: the PRODES edition is downloaded
once and cached, its legend read from the `.qml`, and the maps aggregated on
a 500 m BDC grid. Needs network; `--offline` runs a stand-in.
[`examples/`](examples/) has runnable, self-contained examples that run offline in seconds:

- **01–03, synthetic data** — a quickstart with raster operators, vector drivers, and time series handed off to DisSModel.
- **quickstart.toml** — declarative pipeline equivalent to example 01.

```bash
python examples/01_quickstart.py
python examples/04_terrame_fill_itaituba.py
disscube run examples/pipelines/quickstart.toml
```

Real data workflows, TerraME parity benchmarks, and large-scale case studies are maintained in
[**LambdaGeo/disscube-recipes**](https://github.com/LambdaGeo/disscube-recipes) (`cases/terrame_fill`, `cases/ilha_maranhao`, `cases/prodes_br163`, `cases/luccme_br`).

See [`examples/README.md`](examples/README.md) for the full list, and
[`docs/terrame_fill_correspondence.md`](docs/terrame_fill_correspondence.md)
for how DisSCube's operators relate to TerraME's *Fill*.
Expand Down
92 changes: 30 additions & 62 deletions docs/examples.md
Original file line number Diff line number Diff line change
@@ -1,90 +1,58 @@
# Examples

The [`examples/`](https://github.com/DisSModel/disscube/tree/main/examples)
folder has runnable scripts that need nothing beyond `pip install -e .`: each
one works in a temporary directory and finishes in a few seconds. Pass a
directory to keep the catalog, the inputs and the derived Zarr stores for
inspection (e.g. in QGIS):
folder contains runnable, self-contained examples demonstrating the two primary modes of using DisSCube:
the **Python API** (`CubeClient`, `Derivation`) and the **declarative TOML pipeline** (`disscube validate` / `disscube run`).

All examples run offline, generate or consume tiny synthetic data (< 50 KB), and complete in seconds without downloading external assets:

```bash
python examples/01_quickstart.py # temporary workspace
python examples/01_quickstart.py ./scratch # keep the outputs
python examples/01_quickstart.py ./scratch # keep the outputs for inspection
```

All examples are executed by the test suite (`tests/test_examples.py`), so
they stay in sync with the API.
All examples are executed by the test suite (`tests/test_examples.py`), ensuring they always stay in sync with the codebase.

## Learning the API — synthetic data
## Python API Examples (Synthetic Data)

Examples 01–03 generate their own inputs, so every number they print can be
checked by hand.
Examples 01–03 generate their own inputs locally, allowing every calculated number to be verified directly:

| Example | What it shows |
|---|---|
| [`01_quickstart.py`](https://github.com/DisSModel/disscube/blob/main/examples/01_quickstart.py) | Grid, raster sources and declarative derivations: `percentage`, `majority`, `mean`; loading results; cache hits via `spec_hash` |
| [`02_vector_drivers.py`](https://github.com/DisSModel/disscube/blob/main/examples/02_vector_drivers.py) | Drivers from points, lines and polygons: `min_distance`, `count`, `presence`, `attribute`; several variables per derivation |
| [`03_time_series.py`](https://github.com/DisSModel/disscube/blob/main/examples/03_time_series.py) | Time-stamped sources, `(time, y, x)` loading, and the hand-off to DisSModel with `to_lucc_data()` (including `period`) |

## Real data — TerraME's *Fill* examples

Examples 04–06 use the three *Fill* examples shipped with TerraME's `gis`
package, bundled in
[`examples/data/terrame/`](https://github.com/DisSModel/disscube/tree/main/examples/data/terrame)
together with the cellular spaces TerraME produced. Each derives the same
attributes with DisSCube and prints a cell-by-cell comparison with TerraME's
own output.

| Example | Cells | What it shows | Result |
|---|---|---|---|
| [`04_terrame_fill_itaituba.py`](https://github.com/DisSModel/disscube/blob/main/examples/04_terrame_fill_itaituba.py) | 620 × 5 km | TerraME's [Fill tutorial](https://github.com/TerraME/terrame/wiki/Fill): `mean`, `percentage` × `coverage_purity`, `min_distance` | averages and class coverage reproduce TerraME |
| [`05_terrame_fill_emas.py`](https://github.com/DisSModel/disscube/blob/main/examples/05_terrame_fill_emas.py) | 5 514 × 500 m | `presence` of lines, `max` / `min` of a raster; a study area defined by a limit polygon | 98.4–99.7 % of cells identical |
| [`06_terrame_fill_amazonia.py`](https://github.com/DisSModel/disscube/blob/main/examples/06_terrame_fill_amazonia.py) | 2 229 × 50 km | PRODES coverage with a declared nodata, distances to roads and ports | coverage identical wherever PRODES has data |

What the differences mean — and which TerraME operations DisSCube does not
support yet — is discussed in
[TerraME Fill Cells Correspondence](terrame_fill_correspondence.md).

## Real data — Brazil Data Cube

Example 07 reads a satellite data cube straight from the Brazil Data Cube
STAC catalog; see [BDC Integration](guides/bdc.md#reading-data-cubes-via-stac).

| Example | Cells | What it shows |
|---|---|---|
| [`07_bdc_cube.py`](https://github.com/DisSModel/disscube/blob/main/examples/07_bdc_cube.py) | 58 × 59 × 300 m | `LANDSAT-16D-1` over Ilha do Maranhão: windowed reads, dry-season median, `mean` of NDVI and MNDWI, `percentage` of open water on a grid snapped to the BDC Albers mesh |

It needs network access and `pip install -e ".[bdc]"`. Without network, or
with `--offline`, it runs on a synthetic scene with the same grid and value
ranges and prints a warning that the numbers are not BDC data.
## Declarative Pipeline Files (TOML)

## Real data — MapBiomas
Pipelines can be declared in clean, version-controlled TOML files in
[`examples/pipelines/`](https://github.com/DisSModel/disscube/tree/main/examples/pipelines):

Example 08 reads the MapBiomas annual land-cover maps straight from the
national files; see [MapBiomas](guides/mapbiomas.md).

| Example | Cells | What it shows |
| Pipeline | What it shows | Execution |
|---|---|---|
| [`08_mapbiomas_land_use.py`](https://github.com/DisSModel/disscube/blob/main/examples/08_mapbiomas_land_use.py) | 58 × 59 × 300 m, 2 years | Collection 11 (30 m) over Ilha do Maranhão in 2000 and 2020: `majority` and `percentage` (urban, forest, mangrove) on the grid of example 07, `(time, y, x)` series handed to DisSModel with `to_lucc_data()` |

It needs network access. Without it, or with `--offline`, it runs on a
synthetic scene with the same grid and class codes and says so.
| [`quickstart.toml`](https://github.com/DisSModel/disscube/blob/main/examples/pipelines/quickstart.toml) | Declarative counterpart of `01_quickstart.py`: derives `percentage`, `majority`, and `mean` on a 300 m grid | `disscube run examples/pipelines/quickstart.toml` |

## Real data — PRODES
Validate and execute:
```bash
disscube validate examples/pipelines/quickstart.toml
disscube run examples/pipelines/quickstart.toml --workspace outputs/quickstart
```

Example 09 downloads the current PRODES edition once (≈130 MB, cached) and
reads its legend from the `.qml` in the ZIP; see
[Land-cover maps: PRODES and classified maps](guides/landcover.md).
See [Pipeline files (TOML)](guides/pipeline_files.md) for full syntax and options.

| Example | Cells | What it shows |
|---|---|---|
| [`09_prodes_deforestation.py`](https://github.com/DisSModel/disscube/blob/main/examples/09_prodes_deforestation.py) | 94 × 86 × 500 m, 3 years | The LuccME Lab15 area south of Santarém at the end of 2008, 2016 and 2024: forest / deforested / other, `percentage` of deforested and forest per cell |
## Real Data & Case Studies (External Repository)

## Pipeline files (TOML)
Real-world datasets, historical reproductions and large-scale case studies are maintained in the dedicated
[**DisSCube Recipes and Case Studies**](https://github.com/LambdaGeo/disscube-recipes) repository:

The same preparations can be declared as TOML files in
[`examples/pipelines/`](https://github.com/DisSModel/disscube/tree/main/examples/pipelines)
and run with `disscube run <file>` — including TerraME's Itaituba Fill
tutorial, offline. See [Pipeline files](guides/pipeline_files.md).
- **TerraME Fill Parity (`cases/terrame_fill`)**: Cell-by-cell numerical parity against TerraME's C++ `fillCellularSpace` across three reference areas:
- `itaituba`: 620 cells (5 km) with elevation, multi-class deforestation, and proximity drivers.
- `emas`: 5 514 cells (500 m) with firebreaks, rivers, and accumulation rasters.
- `amazonia`: 2 229 cells (50 km) with PRODES deforestation and indigenous territory area fractions.
- See [TerraME Fill Cells Correspondence](terrame_fill_correspondence.md).
- **Ilha do Maranhão (`cases/ilha_maranhao`)**: Brazil Data Cube (Landsat-16D STAC) and MapBiomas land-cover series (2000, 2020) aligned on a 300 m BDC Albers grid with Python hand-off to DisSModel.
- **PRODES BR-163 (`cases/prodes_br163`)**: Multi-year deforestation monitoring in the Mojuí dos Campos / BR-163 corridor (500 m grid).
- **LUCCME-BR (`cases/luccme_br`)**: National-scale land-use change reconstruction for Brazil (BigEarth).

## Scope

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/bdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ or scale) produces a new file, a new checksum and a new product, while
re-fetching the same data is a cache hit. For a layer computed from several
assets — an index such as MNDWI — build the `Window2D` yourself and call
`disscube.sources.register_raster(cube, source_id, window, out_dir, provenance)`; see
`examples/07_bdc_cube.py`.
the `ilha_maranhao` case study in the recipes repository (`cases/ilha_maranhao`).

What the reader does for you:

Expand Down
16 changes: 6 additions & 10 deletions docs/guides/pipeline_files.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ before anything is downloaded, and recorded in the provenance of every
product.

```bash
disscube validate examples/pipelines/itaituba_fill.toml # check the file, no downloads
disscube run examples/pipelines/itaituba_fill.toml --workspace outputs/itaituba
disscube validate examples/pipelines/quickstart.toml # check the file, no downloads
disscube run examples/pipelines/quickstart.toml --workspace outputs/quickstart
```

## From `fill` to `[[derive]]`

TerraME's Fill tutorial (`examples/data/terrame/itaituba/itaituba.lua`) and
its pipeline file (`examples/pipelines/itaituba_fill.toml`), side by side:
TerraME's Fill tutorial (see `cases/terrame_fill` in the recipes repository) and
its DisSCube pipeline representation, side by side:

```lua
-- TerraME
Expand Down Expand Up @@ -177,10 +177,6 @@ file that declared it.

| File | What it does | Needs |
|---|---|---|
| `examples/pipelines/itaituba_fill.toml` | TerraME's Fill tutorial | nothing (bundled data) |
| `examples/pipelines/ilha_bdc.toml` | example 07: BDC NDVI and MNDWI | network, `.[bdc]` |
| `examples/pipelines/ilha_mapbiomas.toml` | example 08: MapBiomas 2000 and 2020 | network |
| `examples/pipelines/lab15_prodes.toml` | example 09: PRODES 2008, 2016, 2024 | network |
| `examples/pipelines/quickstart.toml` | Quickstart: synthetic land-use & elevation (`percentage`, `majority`, `mean`) | nothing (offline) |

All of them are validated by the test suite; the Itaituba one is also run and
compared with the same derivations made through the Python API.
For full real-world case studies and TerraME parity benchmarks, see the [DisSCube Recipes](https://github.com/LambdaGeo/disscube-recipes) repository (`cases/terrame_fill`, `cases/ilha_maranhao`, `cases/prodes_br163`, `cases/luccme_br`).
18 changes: 10 additions & 8 deletions docs/terrame_fill_correspondence.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ how faithful they are — but the engineering around them:
`PROJCS["unknown"]` round-trip problems.

A TerraME fill script can also be written as a DisSCube pipeline file —
see [Pipeline files (TOML)](guides/pipeline_files.md) and
`examples/pipelines/itaituba_fill.toml`, the tutorial below as TOML.
see [Pipeline files (TOML)](guides/pipeline_files.md),
the [quickstart pipeline](https://github.com/DisSModel/disscube/blob/main/examples/pipelines/quickstart.toml),
and the parity cases in the recipes repository (`cases/terrame_fill`).

## Strategy → operator correspondence

Expand Down Expand Up @@ -82,12 +83,13 @@ Amazônia — each with its input layers, the Lua script and the cellular space
TerraME produced. Those outputs are the reference: the same inputs are derived
with DisSCube on the same grid and compared cell by cell.

The data are bundled in `examples/data/terrame/` (provenance, license and
checksums in its README). The comparison runs in CI in
`tests/test_terrame_parity.py` — passing tests pin the parity below, strict
`xfail` tests record the known gaps — and examples
`04_terrame_fill_itaituba.py`, `05_terrame_fill_emas.py` and
`06_terrame_fill_amazonia.py` print the comparison for each dataset.
The full parity benchmark suite is maintained and executed on CI in the
[DisSCube Case Studies and Recipes](https://github.com/LambdaGeo/disscube-recipes)
repository (`cases/terrame_fill`), where input datasets are fetched from TerraME
with cryptographic checksum verification (`./run.sh all <dataset>`).

Below is the summary of the correspondences and cell-by-cell numerical parity
verified across the datasets.

### Itaituba — 620 cells, 5 km

Expand Down
Loading
Loading