Release 0.1.20: SDI metadata extraction, %sdi and %metadata magics (ref #306883)

README.md

@@ -36,7 +36,7 @@ pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@staging"
 ```
 
 # staging's latest release (early access) — pin to the tag the "staging" badge above shows
-pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.19-staging"
+pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.20-staging"
 ```
 
 ## Usage
@@ -174,6 +174,57 @@ exact same arguments.
 - `catalog.close()` (or `with Catalog(...) as catalog:`) — disposes the REST/Flight session(s).
   Idempotent, and covered by a process-exit/SIGTERM fallback if you forget.
 
+## SDI metadata
+
+`eea_datalakehouse.sdi` copies a dataset's ISO 19115-3 metadata record from the EEA SDI
+catalogue (GeoNetwork) into the dataset's `metadata` folder in DDS. The XML is a **document
+upload**, not an ingest: nothing becomes a Dremio table. The calling code always passes the
+SDI UUID. See `docs/sdi-integration-plan.md`.
+
+```python
+from eea_datalakehouse.sdi import SdiController
+
+with SdiController.from_env() as sdi:
+    metadata = sdi.get_xml("070d9baa-448d-4168-8514-7dadb3ad876d")   # SDI -> bytes
+    result = sdi.push_to_dds(metadata, "catalog/water_management_resources/bathing_water/bwd")
+    result.dds_path   # ".../bwd/metadata/070d9baa-448d-4168-8514-7dadb3ad876d.xml"
+    result.action     # "uploaded" | "unchanged" | "replaced"
+```
+
+- `get_xml(uuid)` checks the response is `mdb:MD_Metadata` and identifies `uuid`
+  (`NotIso19115_3`, `UuidMismatch`; `SdiNotFound` / `SdiAuthError` for 404 / 401-403).
+- `push_to_dds(metadata, dds_path, folder="metadata", force=False)` compares with the copy
+  already in DDS: same bytes is `unchanged`; an older copy is `replaced`; a copy that is newer or
+  undated (someone edited it in DDS) raises `DdsCopyConflict` unless `force=True`.
+- `resolve_series(series_uuid)` returns the one release of a series that is not superseded, or
+  raises `NotCurrentError` listing every release.
+- `metadata.provenance_tags()` gives `sdi_record_uuid` / `sdi_edition` / `sdi_date_stamp` in
+  `Catalog.setmeta2wiki`'s `tags=` shape, if you want them on the dataset's wiki.
+
+Environment: `SDI_API_URL` (empty means `https://sdi.eea.europa.eu/catalogue`), optional
+`SDI_USERNAME` / `SDI_PASSWORD` for non-public records, and for the upload `DDS_BASE_URL` plus
+`_DREMIO_USER` / `_DREMIO_PWD`, as for ingest. The DDS document endpoints
+(`dds_documents/client.py`) are still assumptions to confirm with the DDS team.
+
+In a notebook, two magics split the work: `%sdi` (`SdiSession`) reads SDI, and `%metadata`
+(`MetadataSession`) pushes metadata files to DDS. Both are built on first use from the kernel
+environment like `%catalog`'s and `%ingest`'s sessions. `%metadata push_to_dds` uploads the
+record `%sdi get_xml` fetched last, so it needs only the path (worked example:
+`debugger/sdi_session_example.ipynb`):
+
+```python
+import eea_datalakehouse.notebook   # registers %catalog/%ingest/%sdi/%metadata
+
+%sdi help
+%sdi get_xml("070d9baa-448d-4168-8514-7dadb3ad876d")
+%metadata push_to_dds("catalog/water_management_resources/bathing_water/bwd")
+```
+
+For the push, `DDS_BASE_URL` is read from the first `.env` in the notebook's folder or a parent
+when `%metadata` builds its session (the kernel environment's value only applies if no `.env`
+sets it; `%metadata dds_base_url()` shows which is in use). The Dremio identity is `_DREMIO_USER` /
+`_DREMIO_PWD` (as `%ingest`), falling back to `DREMIO_USERNAME` / `DREMIO_TOKEN` (as `%catalog`).
+
 ## Notebook facade (`%catalog` / `%ingest`)
 
 For interactive use in JupyterLab, `eea_datalakehouse.notebook` registers two line magics
@@ -281,6 +332,11 @@ context itself — raises `CatalogSessionError` if none is set yet:
 | `dds_ingestion/client.py` | thin, unit-testable HTTP client (`IngestClient`) |
 | `dds_ingestion/progress.py` | tqdm progress bar with graceful fallback |
 | `dds_ingestion/folder.py` | `FolderIngest` orchestration (scan/parallel/resume) |
+| `dds_documents/client.py` | `DocumentsClient` — put/get/list plain files in DDS folders (no ingest) |
+| `sdi/controller.py` | `SdiController` — `get_xml()`, `push_to_dds()`, `resolve_series()` |
+| `sdi/catalogue.py` | `SdiCatalogue` — read-only GeoNetwork REST client |
+| `sdi/session.py` | `SdiSession` / `MetadataSession` — what `%sdi` / `%metadata` dispatch onto |
+| `sdi/iso.py` | reads UUID, title, edition, dates and series children from ISO 19115-3 |
 | `catalog/client.py` | `Catalog` — two connections (REST + Flight), every operation as a method |
 | `catalog/operations.py` | the operations themselves (table2view, datacopy, createfolder, ...), as functions taking an executor and/or a `CatalogRestClient` |
 | `catalog/sql.py` | `SqlExecutor` protocol + REST/Flight implementations, `resolve_executor()` (transport env var) |
@@ -363,7 +419,7 @@ To pin to one specific release instead, use the exact tag the live badges under
 %pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.19"
 
 # staging's latest release (early access)
-%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.19-staging"
+%pip install "git+https://github.com/eeadata/EEALakeHouse.python.git@v0.1.20-staging"
 ```
 
 Use the `%pip` magic rather than `!pip` — it installs into the kernel the

debugger/.env.example

@@ -11,17 +11,37 @@ SDI_METADATA_DIR=
 DDS_BASE_URL=
 DREMIO_BASE_URL=
 
+# --- SDI catalogue (eea_datalakehouse.sdi) ------------------------------------
+# GeoNetwork base URL; the client appends /srv/api/... Empty means the public
+# EEA catalogue, https://sdi.eea.europa.eu/catalogue.
+SDI_API_URL=
+# Optional: an SDI account, only needed to read records that are not public.
+# Public records (all published EEA datasets) need neither.
+SDI_USERNAME=
+SDI_PASSWORD=
+
 # --- credentials — never commit -------------------------------------------
 # DREMIO_TOKEN is a Personal Access Token (Dremio → Account Settings → Personal
 # Access Tokens). In the local stack's stub auth mode the token is simply the
 # username: alice (read-write), bob (read-only), carol (no access).
+# debug_run.py copies these to _DREMIO_USER / _DREMIO_PWD, which the DDS
+# ingest and documents clients read.
 DREMIO_USER=
 DREMIO_TOKEN=
 # Your own Redmine (taskman) API key: taskman > My account > API access key
 # > Show. Read by the git hooks in .githooks/ to post commit notes to issues
 # referenced as "refs #1234" / "fixes #1234". Never leaves your machine.
 REDMINE_API_KEY=
 
+# --- Entra ID → Dremio token exchange (debug_run.py's PAT helper) -------------
+# DREMIO_HOST is the Dremio host name only, no scheme. ENTRA_CLIENT_SECRET is
+# only needed for the service-principal mode (--mode sp).
+DREMIO_HOST=
+ENTRA_TENANT_ID=
+ENTRA_CLIENT_ID=
+ENTRA_CLIENT_SECRET=
+ENTRA_SCOPE=
+
 # --- notebooks --------------------------------------------------------------
 JUPYTER_HOST_PORT=
 # Leave empty for a token-free lab on localhost; set one if the port is exposed.

debugger/debug_run.py

@@ -40,7 +40,9 @@
 from pathlib import Path
 from xmlrpc.client import Boolean
 from eea_datalakehouse.catalog import Catalog
+from eea_datalakehouse.dds_documents import DocumentsClient
 from eea_datalakehouse.dds_ingestion import DremioCreds, FolderIngest, IngestClient
+from eea_datalakehouse.sdi import SdiCatalogue, SdiController, metadata_path
 from eea_datalakehouse.dds_ingestion.common.dremio_identity import (endpoint, dds_credentials,
                                     resolve as _resolve_identity)
 
@@ -420,6 +422,24 @@ def run_catalog_bulk_close() -> None:
     print(f"catalog_bulk_close  after   a.closed={a._closed}  b.closed={b._closed}")
 
 
+def run_sdi_extract() -> None:
+    """Download the release's ISO 19115-3 XML from SDI and push it to DDS's
+    metadata folder. Extraction runs for real even in DRY_RUN (it is a public,
+    read-only SDI call); only the DDS upload is skipped."""
+    uuid = CONFIG["sdi"]["latest_record_uuid"]
+    documents = DocumentsClient(DDS_BASE_URL, dds_credentials(DREMIO_USERNAME, DREMIO_TOKEN))
+    with SdiController(SdiCatalogue.from_env(), documents) as sdi, documents:
+        metadata = sdi.get_xml(uuid)
+        print(f"sdi_extract  {metadata}")
+        print(f"  revised   {metadata.record.date_stamp}")
+        if DRY_RUN:
+            print("DRY RUN — not uploading to DDS. Set DRY_RUN = False to run this for real.")
+            print(f"  would put {metadata_path(SDI_DDS_PATH, uuid)}")
+            return
+        result = sdi.push_to_dds(metadata, SDI_DDS_PATH)
+    print(f"sdi_extract  {result.action}  {result.dds_path}")
+
+
 
 # --------------------------------------------------------------------------
 # Step 1 — get an Entra ID JWT
@@ -599,6 +619,12 @@ def create_pat(host: str, access_token: str, username: str, label: str,
     )
     TABLE2VIEW_IDEMPOTENCY_KEY = "debug-table2view-testview"
 
+    # --- sdi_extract ------------------------------------------------------------
+    # Where the dataset lives in DDS; push_to_dds adds /metadata/{uuid}.xml.
+    # Provisional until the DDS document path is agreed (docs/sdi-integration-plan.md).
+    _t = CONFIG["target"]
+    SDI_DDS_PATH = f"catalog/{_t['domain']}/{_t['subdomain']}/{_t['dataflow']}"
+
     # --- services --------------------------------------------------------------
     # Resolved most-authoritative-first by endpoint():
     #   1. the JupyterLab "Dremio Catalog" settings panel — ddsServerUrl / dremioUrl.
@@ -657,6 +683,8 @@ def create_pat(host: str, access_token: str, username: str, label: str,
     #run_createfolder()
     #run_deletefolder()
 
+    #run_sdi_extract()
+
 
 
 

debugger/sdi_session_example.ipynb

@@ -0,0 +1,179 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "# `%sdi` + `%metadata` — SDI metadata into DDS\n\nA test notebook for the `%sdi` and `%metadata` magics from `eea_datalakehouse.notebook.magics`.\n`%sdi` downloads a dataset's ISO 19115-3 metadata record from the EEA SDI catalogue;\n`%metadata` uploads it to the dataset's `metadata` folder in DDS — a **document upload**, never an ingest: nothing becomes a\nDremio table. See `docs/sdi-integration-plan.md` for the design.\n\n**Kernel environment** — read the same way `%catalog` and `%ingest` read theirs, when each magic's\nfirst call builds its session:\n\n| Variable | Needed for | Notes |\n|---|---|---|\n| `SDI_API_URL` | `%sdi get_xml`, `%sdi resolve_series` | empty or unset → `https://sdi.eea.europa.eu/catalogue` |\n| `SDI_USERNAME` / `SDI_PASSWORD` | non-public records only | optional |\n| `DDS_BASE_URL` | `%metadata push_to_dds` | read from the first `.env` in the notebook's folder or a parent (here `debugger/.env`); else the kernel environment |\n| `_DREMIO_USER` / `_DREMIO_PWD` | `%metadata push_to_dds` | as for `%ingest`; else `DREMIO_USERNAME` / `DREMIO_TOKEN`, as for `%catalog` |\n\n`%sdi` makes public, read-only calls and needs none of the DDS settings. Install the extra this\nneeds once: `pip install \"EEADataLakehouse[notebook]\"`.",
+   "id": "bc3d7aa0"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "import eea_datalakehouse.notebook  # registers %catalog/%ingest/%sdi — no %load_ext needed",
+   "id": "201451f8"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Quick reference\n\n`%sdi help` and `%metadata help` render every command as an HTML table, the same format as\n`%catalog help` and `%ingest help`. They work before any environment variable is set.",
+   "id": "4db70eeb"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "%sdi help",
+   "id": "655f890a"
+  },
+  {
+   "cell_type": "code",
+   "id": "1cb0d697",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "%metadata help"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## Settings for this test\n\nThe bathing water release used throughout the library's tests. The DDS path is\n**provisional** until the DDS document path convention is agreed; `%metadata push_to_dds` appends\n`/metadata/<uuid>.xml` to it.",
+   "id": "200b71fe"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "RELEASE_UUID = \"070d9baa-448d-4168-8514-7dadb3ad876d\"   # Bathing Water Directive, 2025 v1.0\nSERIES_UUID = \"c3858959-90da-4c1b-b9ca-492db0e514df\"    # its series\nDDS_PATH = \"catalog/water_management_resources/bathing_water/bwd\"",
+   "id": "fb073e8a"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## 1. Get the XML from SDI\n\nRuns immediately. The record is checked before it is returned: it must be ISO 19115-3\n(`mdb:MD_Metadata`) and its identifier must be `RELEASE_UUID`. Any failure prints one\n`sdi error: ...` line instead of a traceback.",
+   "id": "795e9151"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "metadata = %sdi get_xml(RELEASE_UUID)\nmetadata",
+   "id": "6269d370"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "What was parsed out of it — title, edition, scope and the metadata dates:",
+   "id": "ddf2bdb3"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "metadata.record",
+   "id": "230bfbc2"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "print(metadata.xml[:600].decode())",
+   "id": "9fce9388"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## 2. A series instead of a release\n\n`resolve_series` returns the one release of a series that is not superseded. If there are zero\nor several it refuses to guess and lists them all. The bathing water series currently has\n**two** non-superseded releases, so expect an `sdi error` listing both — the cell after it\nuses the release UUID directly, which is the normal way.",
+   "id": "4d67a127"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "%sdi resolve_series(SERIES_UUID)",
+   "id": "647cddc0"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## 3. Push it to DDS\n\n**Writes to DDS.** `%metadata push_to_dds` uploads the record `%sdi get_xml` fetched last, so\nonly the DDS path is needed. It compares with the copy already there first:\n\n| DDS already has | Result |\n|---|---|\n| nothing | `uploaded` |\n| the same bytes | `unchanged` (no upload) |\n| an older copy | `replaced` |\n| a newer, same-dated or non-ISO copy (edited in DDS) | refused, unless `force=True` |\n\nThe DDS document endpoints are still interface assumptions to confirm with the DDS team, so\nrun this against a test instance first. Set `RUN_PUSH = True` to run it.",
+   "id": "873f813c"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cbb0491a",
+   "metadata": {},
+   "source": "Check which DDS the push will go to first. `DDS_BASE_URL` is taken from the first `.env`\nfound in this notebook's folder or a parent (read once, when `%metadata` builds its session), and\nfrom the kernel environment only if no `.env` sets it:"
+  },
+  {
+   "cell_type": "code",
+   "id": "416cd8ae",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "%metadata dds_base_url()"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "RUN_PUSH = False\n\nif RUN_PUSH:\n    result = %metadata push_to_dds(DDS_PATH)\n    print(result)\nelse:\n    from eea_datalakehouse.sdi import metadata_path\n    print(\"RUN_PUSH is False — would upload to\", metadata_path(DDS_PATH, metadata.uuid))",
+   "id": "b962036b"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "Running the push a second time should report `unchanged`. To replace a copy that was edited in\nDDS, pass `force=True`; to push a record other than the last one, pass it explicitly:\n\n```\n%metadata push_to_dds(DDS_PATH, force=True)\n%metadata push_to_dds(DDS_PATH, metadata=other_metadata)\n```",
+   "id": "e4248c6b"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## 4. Optional: record the provenance on the wiki\n\n`provenance_tags()` gives `sdi_record_uuid` / `sdi_edition` / `sdi_date_stamp` in the shape\n`Catalog.setmeta2wiki`'s `tags` take (a folder's wiki \"# Meta Data\" section). Writing them is\nup to you — neither magic writes to Dremio itself:\n\n```python\nfrom eea_datalakehouse.catalog import Catalog\n\nwith Catalog(DREMIO_BASE_URL, DREMIO_TOKEN, username=DREMIO_USERNAME) as catalog:\n    catalog.setmeta2wiki(\n        \"catalog.water_management_resources.bathing_water.bwd\",\n        tags=metadata.provenance_tags(),\n        overwrite=True,   # merge with the tags already there\n        idempotency_key=f\"sdi-provenance-{metadata.uuid}\",\n    )\n```",
+   "id": "48ed678c"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "metadata.provenance_tags()",
+   "id": "16fa54c8"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": "## What errors look like\n\n```\n%sdi get_xml(\"00000000-0000-0000-0000-000000000000\")\n# -> sdi error: SDI API error 404 on GET /srv/api/records/.../formatters/xml: no SDI record with UUID '...'\n\n%metadata push_to_dds(DDS_PATH)   # in a kernel with no Dremio identity\n# -> metadata error: pushing to DDS needs a Dremio identity in the kernel environment: ...\n```",
+   "id": "217752bd"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "%sdi get_xml(\"00000000-0000-0000-0000-000000000000\")",
+   "id": "0f8edce6"
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

debugger/test_catalog_magic.ipynb

@@ -499,11 +499,29 @@
    "source": [
     "%catalog schema('bwd.draft.bw_assessment.assessments')  # should be empty now, after the delete"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "ca2ba7e1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "UUUU\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(\"UUUU\")"
+   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv (3.12.3.final.0)",
+   "display_name": ".venv (3.12.3)",
    "language": "python",
    "name": "python3"
   },