Add GCS SA support and HMAC opt-out

README.md

@@ -61,15 +61,16 @@ v1alpha1. Three drivers shipped:
 | --- | --- | --- |
 | `kadm` | Kafka-protocol brokers (Redpanda, Apache Kafka, Confluent) | Topic create/alter/delete. v1alpha1: no per-consumer SASL/SCRAM scoping. |
 | `s3` | S3-compatible (VersityGW, MinIO, AWS S3, Cloudflare R2, Hetzner, GCS interop) | Bucket create/delete. v1alpha1: all consumers receive the backend's root keys. |
-| `gcs` | Google Cloud Storage via the native JSON API | Bucket create/update/delete with location, uniform bucket-level access, versioning and lifecycle parameters. Access Secrets carry a static HMAC pair (S3-protocol data path); all consumers receive the same pair. |
+| `gcs` | Google Cloud Storage via the native JSON API | Bucket create/update/delete with location, uniform bucket-level access, versioning and lifecycle parameters. Access Secrets carry a static HMAC pair (S3-protocol data path); all consumers receive the same pair. Driver 0.2 adds opt-in per-bucket service accounts (`parameters.serviceAccount`): a bucket-scoped GCP SA whose key JSON lands in each access Secret for OAuth2 bearer-token auth (`examples/gcs/service-account/`). |
 
 e2e coverage in CI runs against Redpanda (`kadm`), VersityGW +
 MinIO (`s3`) and fake-gcs-server (`gcs`). The other listed S3
 backends share the same client library and the same e2e shape; if
 you hit a compatibility issue with one of them, please file an
 issue. For `gcs`, behaviours the emulator cannot exercise (HMAC
 auth enforcement, the 90-day window for disabling uniform
-bucket-level access) are documented rather than e2e-gated.
+bucket-level access, the per-bucket serviceAccounts IAM surface)
+are documented rather than e2e-gated.
 
 Use `gcs` (not `s3` interop) when the controller should provision
 GCS buckets: creation needs the project, and location / uniform
@@ -192,6 +193,13 @@ backends:
     # mint out of band with: gcloud storage hmac create <sa-email>
     accessKeyID:     ${GCS_HMAC_ACCESS_ID}
     secretAccessKey: ${GCS_HMAC_SECRET}
+    # Optional: per-bucket service accounts (parameters.serviceAccount).
+    # Use a DEDICATED identity project - key creation equals
+    # impersonation, so the controller's SA-admin grants must not
+    # extend to projects with unrelated identities. See
+    # examples/gcs/service-account/ for required grants.
+    # serviceAccounts:
+    #   project: my-buckety-identities
 ```
 
 Access Secrets carry the S3-interop `endpoint` (a bare host - the

SPEC.md

@@ -346,6 +346,33 @@ so `${backend.zone}` must reflect whichever backend a given
 `Buckety` resolves to. Drivers do not inspect `defaults`; the
 templating layer above does.
 
+**Parameter templates.** A driver may additionally declare
+individual `spec.parameters` keys as template-resolved (the gcs
+driver declares `serviceAccount`). Declared keys resolve with a
+RESTRICTED grammar: `${name}`, `${namespace}` and `${backend.X}`
+only - no `${label[...]}`. The restriction is load-bearing:
+`spec.name` may reference mutable labels because its resolution
+freezes into `status.backendResourceName` at first reconcile,
+but parameters re-resolve on every pass, so every input must be
+immutable for the resolved value to be stable. Resolution runs
+in admission and in the reconciler on the merged
+defaults-under-CR view, which lets the cluster operator write a
+convention once as a backend parameter default
+(`serviceAccount: ${name}-${namespace}`) with tenants declaring
+nothing. A CR overrides a default per key, and for
+`serviceAccount` the empty string is the defined per-CR opt-out.
+Undeclared keys never resolve; `${...}` in their backend
+defaults stays a startup error.
+
+Be deliberate when ADDING a backend parameter default: defaults
+merge into EXISTING resources' effective view on the next
+reconcile after the controller rollout, without passing
+admission - there is no recreate gate on this route, and a
+default that resolves invalid for some existing resource (an
+over-long namespace against the SA ID's 30-character cap, say)
+freezes that resource's reconcile rather than failing its
+creation. Audit the fleet before adding one.
+
 ## Driver versioning
 
 Each driver carries a SemVer (`major.minor.patch`) advertised by
@@ -751,8 +778,12 @@ data:
   bucket:          <base64>    # tenant1-orders                (resource-type key)
   project:         <base64>    # the backend's GCP project
   region:          <base64>    # SigV4 signing region, derived with the endpoint (absent for multi-regions)
-  accessKeyID:     <base64>
+  accessKeyID:     <base64>    # static backend-wide HMAC pair; omitted when parameters.hmac="false"
   secretAccessKey: <base64>
+  # With parameters.serviceAccount (driver >= 0.2, backend opt-in):
+  serviceAccountKey:   <base64>    # SA key JSON (client_email, private_key, ...) for OAuth2 bearer-token auth
+  serviceAccountEmail: <base64>    # <shortname>@<identity-project>.iam.gserviceaccount.com
+  serviceAccountKeyId: <base64>    # private_key_id, for audit and rotation tooling
 ```
 
 Unlike the s3 driver, whose `endpoint` passes the configured URL
@@ -763,12 +794,74 @@ config can override both fields for emulators.
 
 The access keys are the backend's static HMAC pair (minted out of
 band via `gcloud storage hmac create`, copied identically to
-every `BucketyAccess`). Driver-minted per-access credentials are
-deliberately deferred to the v1alpha2 scoping design: GrantAccess
-runs on every reconcile and its result rewrites the Secret, and a
-GCS HMAC secret is only retrievable at creation, so per-access
-minting cannot be idempotent until grant-once semantics exist.
-Key names stay the same when scoped credentials land.
+every `BucketyAccess`). `parameters.hmac="false"` (driver >= 0.2,
+mutable, per CR or as a backend parameter default) omits the pair
+- typically together with `serviceAccount`, so consumers hold
+ONLY the bucket-scoped identity instead of having its
+blast-radius win undone by the backend-wide pair riding along;
+without `serviceAccount` it yields a coordinates-only Secret for
+ambient-credential consumers. The DRIVER default stays
+pair-included: it is the incumbent, family-portable contract, and
+a minor bump must not remove Secret keys - a backend chooses the
+opt-in posture by declaring `hmac: "false"` in its parameter
+defaults.
+
+**Per-bucket service accounts (gcs driver 0.2, opt-in).** A gcs
+backend that sets `serviceAccounts.project` in its config lets a
+`Buckety` declare `parameters.serviceAccount` (a template-resolved
+SA short name, immutable post-create). The driver then maintains a
+GCP service account with `roles/storage.objectAdmin` on that
+bucket only, and each `BucketyAccess` Secret additionally carries
+one user-managed key for it — native GCS auth with bucket-scoped
+blast radius, alongside the (still backend-wide) HMAC pair. The
+create-only-retrievable-key problem that deferred per-access
+minting is solved by `GrantRequest.ExistingSecretData`: GrantAccess
+returns the Secret's current key unchanged while it still verifies
+against `keys.list`, and mints only when the key is absent, revoked
+out of band, or expired — which makes server-side key deletion the
+manual rotation runbook. `status.principal` is the key's full
+resource name and `RevokeAccess` deletes it, so BucketyAccess
+deletion performs real revocation for these Secrets.
+
+The `serviceAccounts.project` SHOULD be a dedicated identity
+project, separate from the bucket project: key creation equals
+impersonation, and the project boundary is what confines the
+controller's `roles/iam.serviceAccountAdmin` +
+`roles/iam.serviceAccountKeyAdmin` grants to identities that exist
+only to hold buckety-granted bucket bindings (the bucket project
+additionally needs `storage.buckets.getIamPolicy/setIamPolicy`).
+Ownership is stamped as JSON into each SA's description and
+verified before every bind/mint/delete, so a tenant naming a
+foreign SA in `parameters.serviceAccount` is refused rather than
+handed its keys. (The marker is trusted as written: anyone
+holding serviceAccountAdmin on the identity project can rewrite
+descriptions, which is one more reason that project must be
+dedicated to buckety-minted identities.) Role scoping is still
+NOT implemented: all accesses share the bucket-scoped SA
+regardless of role, and `ScopingNotImplemented` continues to
+surface for non-ReadWrite roles. Scheduled key rotation is
+deliberately deferred (see Non-goals); the reuse check plus
+one-key-per-access keeps within GCP's 10-user-managed-keys-per-SA
+limit, bounding a bucket at ~10 CONCURRENT accesses until
+rotation lands - not cumulative lifecycle churn, because
+credentials never outlive their access (next paragraph).
+
+**Retention semantics.** `retentionPolicy=Retain` retains the
+backend data unit - the bucket, its IAM policy, and its (by then
+keyless) service account - and NEVER credentials: keys are
+revoked with each `BucketyAccess` under every policy, replaced
+keys are revoked as soon as their successor is written, and the
+implicit access is revoked before the `Buckety` itself lets go.
+Keeping the SA is deliberate: the retained bucket's policy still
+references it (deleting it would leave a dangling
+`deleted:serviceAccount:` binding), and recreate-with-adoption -
+the flow Retain exists for - finds a marker-matching keyless SA
+instead of hitting GCP's 30-day tombstone on the reserved name.
+Permanent teardowns that must reclaim SA quota delete the SA out
+of band; the ownership marker identifies buckety's. If a leak
+ever wedges an SA at the key cap, surplus `USER_MANAGED` keys on
+a marker-verified SA are safe to delete server-side and the fleet
+re-mints within one reconcile.
 
 ## Adoption
 
@@ -836,7 +929,12 @@ deletion more conservative.
 
 - Both kinds carry a finalizer `buckety.yolean.se/cleanup`.
 - `BucketyAccess` deletion blocks on `RevokeAccess` succeeding
-  (in v1alpha1 the no-op revoke completes immediately).
+  (in v1alpha1 the no-op revoke completes immediately). When the
+  backend is missing from config, deletion blocks only for
+  principals the driver stamped revocable
+  (`status.principalRevocable`, from `GrantResult.Revocable` -
+  gcs SA keys); static shared principals release as before, so
+  backend renames do not wedge access teardown.
 - `Buckety` deletion blocks on (a) all referencing
   `BucketyAccess` being gone — controller does NOT cascade-delete
   them; it surfaces a `BlockedByAccesses` condition with the
@@ -1034,7 +1132,7 @@ Required CI matrix for v1alpha1:
 | --- | --- |
 | `kadm` | redpanda |
 | `s3`   | versitygw, minio |
-| `gcs`  | fakegcs (fake-gcs-server; covers the JSON-API control plane — real-GCS-only behaviours like HMAC auth enforcement and the 90-day UBLA disable window are documented, not e2e-gated) |
+| `gcs`  | fakegcs (fake-gcs-server; covers the JSON-API control plane — real-GCS-only behaviours like HMAC auth enforcement, the 90-day UBLA disable window and the serviceAccounts IAM surface (no iam.googleapis.com or bucket-IAM emulation; unit-tested against an httptest fake instead) are documented, not e2e-gated) |
 
 Adding an implementation later (e.g. AWS S3 once the project has
 credentials and a budget) requires no example or harness changes,
@@ -1071,15 +1169,25 @@ and the corresponding GHA secret.
 - MySQL driver.
 - Per-consumer credential scoping (SASL/SCRAM for kafka, IAM
   users for S3). All `BucketyAccess` instances for the same
-  `Buckety` receive identical credentials.
+  `Buckety` receive identical credentials. Partial exception
+  since gcs driver 0.2: opt-in per-BUCKET service accounts give
+  each access its own key for a bucket-scoped identity (see
+  Secret output > gcs driver), but role scoping remains
+  unimplemented.
 - Cross-namespace `bucketyRef`.
 - Adopting backing resources that already exist outside Buckety.
 - Quota enforcement.
 - Hot-reload of `buckety-controller.yaml` (envsubst is
   startup-only; rotating credentials requires re-rolling the
   controller Pod).
 - Runtime credential rotation in issued Secrets without
-  `BucketyAccess` recreate.
+  `BucketyAccess` recreate. Deferred by intent, not omission:
+  for gcs per-bucket service accounts the reuse-or-mint grant
+  machinery is already rotation-shaped, and scheduled key
+  rotation (mint new, overlap one period, garbage-collect the
+  previous key) is the planned follow-up; until then rotation is
+  operator-driven - delete the key server-side and the next
+  reconcile re-mints (`gcloud iam service-accounts keys delete`).
 - Multi-cluster federation.
 - Admission webhook for cross-resource invariants. Per-resource
   parameter validation (against per-driver schemas) and

deploy/kustomize/crd/bucketyaccess.yaml

@@ -91,6 +91,14 @@ spec:
                   Backend-side identity granted access. In v1alpha1
                   with no per-consumer scoping this is typically
                   the backend's root principal.
+              principalRevocable:
+                type: boolean
+                description: |
+                  Whether principal names a credential the driver
+                  minted for this access and must revoke
+                  backend-side (a gcs SA key). Deletion with the
+                  backend missing from config blocks only for
+                  revocable principals.
               conditions:
                 type: array
                 items:

deploy/kustomize/release/kustomization.yaml

@@ -7,7 +7,7 @@ kind: Kustomization
 resources:
 - ../base
 images:
-- digest: sha256:85ec8d4746b37fae88387cc280631ca16269404fb60fcc83e6af5d59aa13c863
+- digest: sha256:1fd39caed0cb3b17d827ccff4e95c6e80c432a0317e9c70d7cd12839c21ccbc4
   name: ghcr.io/yolean/buckety-controller
   newName: ghcr.io/yolean/buckety-controller
-  newTag: 20260717T064548Z
+  newTag: 20260729T082750Z

examples/gcs/service-account/README.md

@@ -0,0 +1,78 @@
+# gcs per-bucket service account (opt-in, driver >= 0.2)
+
+A `Buckety` that declares `parameters.serviceAccount` gets a
+dedicated GCP service account with `roles/storage.objectAdmin` on
+its bucket only, and every access Secret additionally carries:
+
+| key | value |
+| --- | --- |
+| `serviceAccountKey` | SA key JSON — mount it and point `GOOGLE_APPLICATION_CREDENTIALS` at it for OAuth2 bearer-token auth with native GCS clients and V4 signed URLs |
+| `serviceAccountEmail` | `orders-<namespace>@<identity-project>.iam.gserviceaccount.com` |
+| `serviceAccountKeyId` | the key's `private_key_id`, for audit and rotation tooling |
+
+The static HMAC pair stays in the Secret unchanged (additive keys
+per SPEC §Secret output), so S3-interop consumers keep working;
+the SA credential is what shrinks blast radius from
+"every bucket on the backend" to "this bucket". Once no consumer
+of a bucket needs the S3-interop path, add `hmac: "false"`
+(mutable) and its Secrets drop the backend-wide pair entirely -
+per CR, or as a backend parameter default to make HMAC opt-in
+across the backend. A CR can conversely opt out of a
+backend-default SA with `serviceAccount: ""`.
+
+## Backend prerequisites
+
+This example has no `assert.sh` deliberately: fake-gcs-server
+implements neither `iam.googleapis.com` nor bucket IAM policies,
+so the feature is unit-tested (`pkg/drivers/gcs/serviceaccount_test.go`)
+and exercised against real GCS. The backend needs:
+
+```yaml
+backends:
+- name: gcs
+  driver: gcs
+  config:
+    project: my-bucket-project
+    accessKeyID: ${GCS_HMAC_ID}
+    secretAccessKey: ${GCS_HMAC_SECRET}
+    serviceAccounts:
+      # STRONGLY RECOMMENDED: a dedicated identity project. Key
+      # creation equals impersonation, so the controller's IAM
+      # grants must be confined to a project whose only identities
+      # are the ones buckety mints. Cross-project bucket bindings
+      # make the split free.
+      project: my-buckety-identities
+  # Or impose the naming convention for all buckets, letting CRs
+  # omit the parameter:
+  # parameters:
+  #   serviceAccount: ${name}-${namespace}
+```
+
+Controller credential grants:
+
+- on the identity project: a custom role with
+  `iam.serviceAccounts.{create,get,delete}` and
+  `iam.serviceAccountKeys.{create,list,delete}` (the predefined
+  `roles/iam.serviceAccountAdmin` + `roles/iam.serviceAccountKeyAdmin`
+  work but carry more than needed)
+- on the bucket project: `storage.buckets.getIamPolicy` +
+  `storage.buckets.setIamPolicy` on top of the existing bucket
+  CRUD grant
+
+Orgs that set `constraints/iam.disableServiceAccountKeyCreation`
+block this feature by design; the grant fails with an actionable
+`GrantFailed` condition.
+
+## Rotation (manual, until scheduled rotation lands)
+
+```
+gcloud iam service-accounts keys delete <serviceAccountKeyId> \
+  --iam-account=<serviceAccountEmail>
+```
+
+The next reconcile (within the requeue cadence) detects the
+revoked key via `keys.list` and mints a fresh one into the Secret
+in place. Deleting the `BucketyAccess` revokes its key
+(`status.principal` is the key's resource name); deleting the
+`Buckety` with `retentionPolicy=Delete` removes the service
+account with the bucket.

examples/gcs/service-account/buckety.yaml

@@ -0,0 +1,20 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/Yolean/buckety-controller/main/schema/buckety-gcs.schema.json
+apiVersion: buckety.yolean.se/v1alpha1
+kind: Buckety
+metadata:
+  name: orders
+spec:
+  backend: gcs
+  retentionPolicy: Delete
+  parameters:
+    uniformBucketLevelAccess: "true"
+    # Per-bucket GCP service account (backend must enable
+    # serviceAccounts in its config). Template-resolved; the
+    # -${namespace} suffix is the uniqueness convention since SA
+    # IDs are unique per project. Alternatively the cluster
+    # operator declares this once as a backend parameter default
+    # and CRs omit it entirely. Immutable post-create.
+    serviceAccount: ${name}-${namespace}
+  defaultAccess:
+    role: ReadWrite
+    credentialsSecretName: orders-bucket

examples/gcs/service-account/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- buckety.yaml

go.mod

@@ -2,6 +2,8 @@ module github.com/Yolean/buckety-controller
 
 go 1.26.1
 
+toolchain go1.26.5
+
 require (
 	cloud.google.com/go/storage v1.63.1
 	github.com/Yolean/y-cluster v0.4.6

pkg/api/v1alpha1/types.go

@@ -240,6 +240,14 @@ type BucketyAccessStatus struct {
 	// per-consumer scoping is not implemented.
 	Principal string `json:"principal,omitempty"`
 
+	// PrincipalRevocable is whether Principal names a credential
+	// the driver minted for this access and must revoke
+	// backend-side (a gcs SA key), as opposed to a shared static
+	// principal whose revoke is a no-op. Deletion with the backend
+	// missing from config blocks only for revocable principals;
+	// static ones release as in v1alpha1.
+	PrincipalRevocable bool `json:"principalRevocable,omitempty"`
+
 	Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
 }