Ci/pin harvest content

.github/workflows/release.yml

@@ -519,10 +519,6 @@ jobs:
       attestations: write # write the image build-provenance attestation
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-      # Both download-artifact steps in this workflow log two warnings per run: v8 also
-      # writes an "artifact metadata storage record", which wants `artifact-metadata: write`.
-      # That permission is deliberately not granted and the record is not used — provenance
-      # here comes from the attestation below. The download itself is unaffected.
       - name: Download the released x86_64 gnu (glibc) service binary
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
         with:
@@ -610,3 +606,10 @@ jobs:
           subject-name: ${{ steps.push.outputs.image }}
           subject-digest: ${{ steps.push.outputs.digest }}
           push-to-registry: true
+          # With push-to-registry the action also tries to write an org-level "artifact
+          # metadata storage record" (POST /orgs/{org}/artifacts/metadata/storage-record),
+          # which needs `artifact-metadata: write` and, for this image, came back "no
+          # artifacts found" — two warnings on every release. The record only feeds the
+          # org's linked-artifacts page and is not used; provenance is the attestation
+          # itself, so skip it rather than widen the token.
+          create-storage-record: false

.github/workflows/scheduled.yml

@@ -224,12 +224,13 @@ jobs:
   # conformance/ga4gh-beacon-v2-default-model/, conformance/ga4gh-vrs-1.3/) from the commit
   # pinned in its VENDORED.md and asserts it is still byte-identical — turning the
   # "verbatim at commit X" claim from documentation into an enforced check. `check` also
-  # runs the `pins` assertions: the userportal's deployed ckanext-dcat/ckanext-fairdatapoint
-  # refs and gdi-metadata's declared HealthDCAT-AP release (so a federation move surfaces
-  # here as a ticket, not a silent harvest break). It is off the per-PR `ci` workflow
-  # because it needs network access and gates nothing on a PR. A red run means a vendored file
-  # was locally edited, a pin drifted, or an upstream deploy ref moved — re-vendor with
-  # `scripts/vendored.sh fetch` + bump VENDORED.md, update EXTERNAL_PINS, or revert the edit.
+  # runs the `pins` assertions: the userportal's deployed profile code, at the tags its own
+  # ckan/Dockerfile names, and the beacon-v2 release watch (so a consumer change surfaces
+  # here as a ticket, not a silent harvest break). It is off the per-PR `ci` workflow because
+  # it needs network access and gates nothing on a PR. A red run means a vendored file was
+  # locally edited, or upstream content or a pinned token changed — re-vendor with
+  # `scripts/vendored.sh fetch` + bump VENDORED.md, re-check and re-pin in scripts/vendored.sh,
+  # or revert the edit.
   # (The authored FDP shapes under conformance/shapes/fdp/ are not vendored — see
   # their PROVENANCE.md — so they are not byte-checked here.) GITHUB_TOKEN raises the
   # unauthenticated rate limit for the raw fetches.
@@ -249,7 +250,9 @@ jobs:
       # `continue-on-error`, because upstream moving is news, not a defect in this tree.
       # The policy is to follow the federation in lockstep rather than chase it, so this
       # surfaces the change as exit 3, visible in the run, without failing a job that then
-      # gets muted. The `notify` step below still reports the run.
+      # gets muted. The `notify` step below still reports the run. The gdi-metadata shapes
+      # are the exception: the conformance gate validates against them, so the `pins` job
+      # below runs `drift gdi-metadata` fatally.
       - name: Report whether upstream has moved past our pinned commits
         continue-on-error: true
         env:
@@ -495,13 +498,14 @@ jobs:
           key: gdi-corpus-${{ hashFiles('scripts/fetch-corpus.sh') }}
       - run: ./scripts/ci-local.sh corpus
 
-  # External pin freshness: the userportal deploy refs and gdi-metadata lineage this tree
-  # follows, plus the distroless runtime digest in the Dockerfile. Weekly, not per-PR,
-  # because it is a verdict about the world rather than about the commit: it goes red when
-  # the federation moves or the distroless base is rebuilt, neither of which any pull
-  # request caused, and a gate that reddens for reasons the author cannot fix is a gate
-  # people learn to ignore. The leg already downgrades an unreachable upstream to a warning
-  # rather than a failure.
+  # External pin freshness: the userportal's deployed profile code, the vendored gdi-metadata
+  # shapes against upstream, the beacon-v2 release and action tags, plus the distroless
+  # runtime digest in the Dockerfile. Weekly, not per-PR, because it is a verdict about the
+  # world rather than about the commit: it goes red when a consumer contract moves, which no
+  # pull request caused, and a gate that reddens for reasons the author cannot fix is a gate
+  # people learn to ignore. An unreachable upstream is a warning. A rebuilt distroless base
+  # is a note, not a failure: `image-scan` above already fails on a fixable CVE in the pinned
+  # one.
   pins:
     runs-on: ubuntu-26.04
     steps:
@@ -511,6 +515,7 @@ jobs:
           GITHUB_TOKEN: ${{ github.token }}
           # Fatal here, unlike a developer's working tree: this job exists to be the place
           # drift is noticed, and a warning in a weekly log nobody opens is not that.
+          # BASE_IMAGE_STRICT stays unset, per the job comment above.
           PINS_STRICT: "1"
         run: ./scripts/ci-local.sh pins
 

CONTRIBUTING.md

@@ -545,15 +545,18 @@ checks each pinned SHA against its comment upstream.
 The `pins` leg has no `ci.yml` job: it is a verdict about the world rather than about the
 commit, so it runs weekly in `scheduled.yml` as its own `pins` job (with `PINS_STRICT=1`),
 alongside `vendored-sync`.
-`scripts/vendored.sh pins` asserts that the userportal's deployed CKAN-extension refs, the
-gdi-metadata HealthDCAT-AP lineage, the GA4GH beacon-v2 latest release tag and the pinned
-GitHub Action tags all still resolve to what this tree pins, and `ci-local.sh` itself
-compares the Dockerfile's distroless runtime digest against upstream, through Docker. It
-needs network, and unauthenticated GitHub API calls are rate-limited per IP, so export
-`GITHUB_TOKEN` — sent to `api.github.com` only — to lift the budget. Real drift,
-including a pinned path that 404s, warns in `pins` and fails `pins-strict`, which `release`
-and the weekly job run. A rate-limited or unreachable host only warns, because that is a
-verdict about your connection.
+`scripts/vendored.sh pins` asserts that the userportal's deployed profile code (at the tags
+its own `ckan/Dockerfile` names, so a userportal release that moves a tag without moving the
+profile passes), the GA4GH beacon-v2 latest release tag and the pinned GitHub Action tags
+still resolve to what this tree pins; `vendored.sh drift gdi-metadata` compares the vendored
+shapes against upstream `main`; and `ci-local.sh` itself compares the Dockerfile's distroless
+runtime digest against upstream, through Docker. It needs network, and unauthenticated
+GitHub API calls are rate-limited per IP, so export `GITHUB_TOKEN` — sent to
+`api.github.com` only — to lift the budget. Real drift, including a pinned path that 404s,
+warns in `pins` and fails under `PINS_STRICT=1`, which `pins-strict`, `release` and the weekly
+job set. A rebuilt base image is a note unless `BASE_IMAGE_STRICT=1`, which only `pins-strict`
+and `release` set, since `image-scan` already fails on a fixable CVE in the pinned base. A
+rate-limited or unreachable host only warns, because that is a verdict about your connection.
 
 ### Scheduled / on-demand jobs (`scheduled.yml`)
 
@@ -586,9 +589,9 @@ inside the `release` meta-leg are marked.
   vendored conformance file is byte-identical to its pinned commit. Because that commit is
   an immutable SHA it stays green however far upstream moves, so `scripts/vendored.sh
   drift` answers the other question, comparing the same files against the upstream branch
-  named in each `VENDORED.md`. Exit 3 means upstream moved, which is news rather than a
-  defect, so the job runs it `continue-on-error` and it stays out of `all`. Local: both
-  need network.
+  named in each `VENDORED.md`. Exit 3 means upstream moved, which for Beacon and VRS is
+  news rather than a defect, so the job runs it `continue-on-error`; the `pins` leg runs
+  it for the gdi-metadata set alone, fatally. Local: both need network.
 - **`cross-native`** — macOS and Windows `gdi-dataset-tool` builds. No local leg: it needs
   those operating systems.
 - **`cross-linux-arm`** — aarch64-linux service build-verify (gnu and static musl, full
@@ -609,9 +612,10 @@ inside the `release` meta-leg are marked.
   rather than per-PR because the leg range-downloads those slices first; the corpus is
   cached
   between runs. Local: `ci-local.sh corpus`.
-- **`pins`** — external pin freshness: the userportal deploy refs and gdi-metadata lineage
-  this tree follows, plus the distroless runtime digest in the Dockerfile. Weekly because
-  it is a verdict about the world, not about the commit. Local:
+- **`pins`** — external pin freshness: the userportal's deployed profile code, the vendored
+  gdi-metadata shapes against upstream, the beacon-v2 release and action tags, plus the
+  distroless runtime digest in the Dockerfile (a note there). Weekly because it is a
+  verdict about the world, not about the commit. Local:
   `PINS_STRICT=1 ci-local.sh pins` (the job sets that so drift is fatal there; a working
   tree only warns).
 - **`notify`** — opens or updates one tracking issue on a failed weekly run. No local

Dockerfile

@@ -119,8 +119,8 @@ FROM ${BIN_SOURCE} AS binsrc
 # committing it. Re-resolve with
 #   docker buildx imagetools inspect gcr.io/distroless/cc-debian13:nonroot
 # (or the registry manifest API) and bump tag+digest together. `ci-local.sh pins`
-# checks this digest against the current upstream one on every run (WARN in `all`,
-# fail under `PINS_STRICT=1`).
+# checks this digest against the current upstream one on every run (a note in `all` and
+# in the weekly job, fatal under `BASE_IMAGE_STRICT=1`, which `release` sets).
 FROM gcr.io/distroless/cc-debian13:nonroot@sha256:e792ab3d241a468a4fd7519ddbbebe66b49b5f365771716ea688ad40b6c6f1c2
 
 # The service reads its config from GDI_CONFIG (default /etc/gdi-node-standalone/node.toml)

conformance/README.md

@@ -111,7 +111,8 @@ into `target/conformance/union.ttl`. It then runs:
   resolution. It overrides no predicate read, so the parent's predicate set is the
   deployed one. It ships in the harvester extension
   (`gdi-userportal-ckanext-fairdatapoint`), not in `ckanext-dcat`, so a harvester bump can
-  change the profile with the `ckanext-dcat` pin standing still; both refs are watched (see
+  change the profile with the `ckanext-dcat` pin standing still; the profile code of both
+  forks is watched by content, at the tags the userportal deploys (see
   [Version pins](#version-pins)).
 
   The check prefers the real `ckanext-dcat` `RDFParser` on the parent profile, since the
@@ -200,13 +201,15 @@ You can also run a check by hand once a `union.ttl` exists:
   hash-locked `requirements.lock` (regenerate both together with
   `uv pip compile --universal --generate-hashes`); bump them in a reviewed change.
   `ckanext-dcat` is pinned to `2.4.4`, the upstream base of the GDI fork the userportal
-  deploys (`gdi-userportal-ckanext-dcat @ v2.5.0`, with harvester
-  `gdi-userportal-ckanext-fairdatapoint @ v1.7.2`, per gdi-userportal-ckan-docker). The
-  profile the deployment's harvest sources name is `fairdatapoint_dcat_ap`, supplied by the
-  harvester ref as a subclass of the fork's `euro_health_dcat_ap`, so the `ckanext-dcat`
-  pin alone would not see a profile change. That is why both refs are watched below.
-  `check_ckanext.py` runs an rdflib hand-mirror of the inherited `euro_health_dcat_ap`
-  reads, because the real `RDFParser` needs full CKAN and so never loads in this venv.
+  deploys (`gdi-userportal-ckanext-dcat`, with the harvester
+  `gdi-userportal-ckanext-fairdatapoint`, at the tags gdi-userportal-ckan-docker's
+  `ckan/Dockerfile` names). The profile the deployment's harvest sources name is
+  `fairdatapoint_dcat_ap`, supplied by the harvester as a subclass of the fork's
+  `euro_health_dcat_ap`, so the `ckanext-dcat` pin alone would not see a profile change.
+  That is why both profile files are watched below by content at the deployed tag, and the
+  fork's `pyproject.toml` version against this `2.4.4` pin. `check_ckanext.py` runs an
+  rdflib hand-mirror of the inherited `euro_health_dcat_ap` reads, because the real
+  `RDFParser` needs full CKAN and so never loads in this venv.
 - **Guards on all of the above.** Which gate reaches each one differs, and the difference
   matters, so it is stated per guard rather than claimed for the list. The offline
   integrity of these files is fully covered by `scripts/ci-local.sh all`; the two guards
@@ -218,17 +221,20 @@ You can also run a check by hand once a `union.ttl` exists:
     `"required"` array from a Beacon schema makes everything downstream pass more easily.
   - `vendored.sh pins` (network, in the `pins` leg). Under `all`, both an unreachable
     network and real drift warn and pass: drift here means the federation moved, not that
-    this tree broke, so it is reported rather than fatal. `release` and `pins-strict` set
-    `PINS_STRICT=1` and fail closed on drift. Four watches (`EXTERNAL_PINS`): the
-    userportal's two deployed CKAN-extension refs, gdi-metadata's declared HealthDCAT-AP
-    release, and the latest GA4GH beacon-v2 release tag — the only watch on a standard
-    rather than on what the federation deploys. That last one reads the releases API,
+    this tree broke, so it is reported rather than fatal. The weekly job, `release` and
+    `pins-strict` set `PINS_STRICT=1` and fail closed on drift. Two watches:
+    `USERPORTAL_PROFILE_PINS`, the deployed profile files by hash and the dcat fork's
+    upstream base against this `requirements.txt`, resolved at the tags the userportal's
+    own `ckan/Dockerfile` names, so a userportal release that moves a tag but neither fact
+    passes; and `EXTERNAL_PINS`, the latest GA4GH beacon-v2 release tag — the only watch on
+    a standard rather than on what the federation deploys. That one reads the releases API,
     because a release is a tag and tags are in no file the repo contains: beacon-v2's own
     CHANGELOG tops out at `2.0.0` while the repo is tagged `v2.2.0`, so a file-based watch
     would be a dead guard. The same subcommand also runs `check_action_pins`, which is
     about this repo rather than the federation: it re-resolves every
     `uses: owner/repo@<sha> # <tag>` in the workflows and reports a SHA that no longer is
-    that tag.
+    that tag. The `pins` leg also runs `vendored.sh drift gdi-metadata` (below), with the
+    same warn-or-fail split.
   - `pip-audit` (in the `supply_chain` leg) — the advisory scan for the one corner
     `cargo deny` cannot see. It audits the `.lock` files, not the `.txt`: pip-audit resolves
     a requirements file, and the loose `.txt` can resolve a newer version than the lock pins
@@ -245,11 +251,17 @@ You can also run a check by hand once a `union.ttl` exists:
     narrow — `verify` above already proves nothing in the tree was added, deleted or
     edited — leaving only the question of whether upstream still serves those bytes at that
     commit, which is a verdict about someone else's repository.
-  - `vendored.sh drift` (network) — the question `check` cannot answer: has upstream moved?
-    Same files, compared against the upstream branch (`**Branch:**` in each `VENDORED.md`).
-    Exit 3 means upstream moved, which is news rather than a defect, since this tree is
-    pinned. It is not in `ci-local.sh all`: it makes tens of network fetches, and a gate
-    that reddens because someone else committed is one people learn to ignore.
+  - `vendored.sh drift [set]` (network) — the question `check` cannot answer: has upstream
+    moved? Same files, compared against the upstream branch (`**Branch:**` in each
+    `VENDORED.md`). Exit 3 means upstream moved (a file removed upstream counts), 2 that a
+    file could not be fetched, 1 that this tree is broken (a vendored file missing locally,
+    or a `VENDORED.md` that does not parse). For
+    the Beacon and VRS sets that is news, not a defect: the node tracks a release. The
+    gdi-metadata shapes are the contract the conformance gate validates against, so the
+    `pins` leg runs `drift gdi-metadata` for them, a warning in `all` and fatal under
+    `PINS_STRICT=1`. The all-sets form is not in `ci-local.sh all`: it makes tens of
+    network fetches, and a gate that reddens because someone else committed is one people
+    learn to ignore.
   - Both are also scheduled weekly in `.github/workflows/scheduled.yml`, which runs on
     GitHub Actions (see [`CONTRIBUTING.md`](../CONTRIBUTING.md)). Two caveats. In a fork,
     scheduled workflows run only if the fork enables Actions, and GitHub disables a

conformance/check_ckanext.py

@@ -15,8 +15,9 @@
 ``ckanext-dcat``. It adds tag validation, ``tags_translated`` sanitising and label
 resolution on top of the parent's reads and overrides no predicate, so the parent's
 predicate set below is the deployed one. A harvester bump therefore changes this profile
-without touching the ``ckanext-dcat`` pin, which is why both refs are watched by
-``vendored.sh pins`` (see ``conformance/README.md``).
+without touching the ``ckanext-dcat`` pin, which is why both forks' profile files are
+watched by content, at the tags the userportal deploys, by ``vendored.sh pins`` (see
+``conformance/README.md``).
 
 In practice the rdflib mirror (tier 2) is what runs: the real ``ckanext-dcat``
 ``RDFParser`` needs full CKAN (``ckantoolkit`` imports ``ckan``), which the isolated
@@ -177,21 +178,22 @@ def fallback_direct_parse(g: Graph) -> None:
     installed ``ckanext/dcat/profiles/euro_health_dcat_ap.py`` and ``euro_dcat_ap_3``).
 
     Pins: the deployed profile is ``fairdatapoint_dcat_ap`` from
-    ``gdi-userportal-ckanext-fairdatapoint @ v1.7.2``, a subclass that adds tag sanitising
-    and label resolution and overrides no predicate read, so this hand-mirror tracks its
-    parent in the GDI fork ``gdi-userportal-ckanext-dcat @ v2.5.0`` (what the userportal
-    deploys; ``requirements.txt`` pins the matching upstream base ``ckanext-dcat==2.4.4``
-    for the optional real-parser tier). Both refs are watched by ``vendored.sh pins``,
-    because a harvester bump can change the profile with the ``ckanext-dcat`` pin standing
-    still.
+    ``gdi-userportal-ckanext-fairdatapoint``, a subclass that adds tag sanitising and label
+    resolution and overrides no predicate read, so this hand-mirror tracks its parent in the
+    GDI fork ``gdi-userportal-ckanext-dcat`` (what the userportal deploys;
+    ``requirements.txt`` pins the fork's upstream base ``ckanext-dcat==2.4.4`` for the
+    optional real-parser tier). ``vendored.sh pins`` watches both profile files by hash at
+    the tags the userportal's ``ckan/Dockerfile`` names, because a harvester bump can change
+    the profile with the ``ckanext-dcat`` pin standing still.
 
     The parent ``euro_health_dcat_ap`` profile reads
     ``healthdcatap:{numberOfRecords,healthCategory,hdab}`` (namespace
     ``http://healthdataportal.eu/ns/health#``) and ``dpv:hasLegalBasis``
     (``https://w3id.org/dpv#``), plus the DCAT-AP core ``dct:title``, ``dct:identifier``
     and ``dcat:accessURL`` — the subset asserted below, checked against the fork's
-    ``profiles/euro_health_dcat_ap.py`` at v2.5.0. Re-check this list when bumping, so the
-    best-effort tier cannot drift from the deployed profile.
+    ``profiles/euro_health_dcat_ap.py`` at the hash ``USERPORTAL_PROFILE_PINS`` pins.
+    Re-check this list when that hash drifts, so the best-effort tier cannot drift from the
+    deployed profile.
     """
     catalogs = list(g.subjects(RDF.type, DCAT.Catalog))
     _require(