Skip to content

Commit fde127d

Browse files
bokelleyclaude
andauthored
feat(reporting): add GCS, S3 and Azure Blob row providers (#1468)
* feat(reporting): add GCS, S3 and Azure Blob row providers Adds the cloud providers for object row bindings (adcontextprotocol/adcp#7996, section 3.4) as optional extras adcp[reporting-gcs], adcp[reporting-s3] and adcp[reporting-azure]. Each takes official clients the host builds with its own credentials and never imports the client library itself. - GCS pins the object generation and writes with if_generation_match=0 and a declared MD5. The probe requires uniform bucket-level access and enforced public access prevention and refuses lifecycle Delete rules that can match live objects under the binding prefix. - S3 and S3-compatible stores need bucket versioning: the native version is the write-unique VersionId, never the ETag, deletes confirm the exact version, and the probe proves create-only behaviour empirically. - Azure needs blob versioning: the native version is version_id, the probe refuses public containers and stores that replace blobs. - Every provider passes the shared provider conformance suite including write-unique-version; S3 runs against moto with the real boto3 client. Part of #1457. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(reporting): carry the call budget into every provider SDK call Threads the budget into every GCS and Azure call and refuses S3 clients without finite socket timeouts, since wait_for cannot stop a worker thread. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent 83f62c9 commit fde127d

9 files changed

Lines changed: 2501 additions & 0 deletions

‎docs/reporting-row-storage.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,3 +158,82 @@ await store.create_schema()
158158
- Custom providers implement `ReportingRowObjectProviderV1`. Run
159159
`run_reporting_row_object_provider_conformance_v1(provider, location=..., prefix=...)` against a dedicated test
160160
prefix before using one.
161+
162+
### Cloud providers
163+
164+
Cloud providers wrap official clients that the host constructs with its own credentials, so the extras only need to
165+
be installed where those clients are created: `pip install 'adcp[reporting-gcs]'`, `'adcp[reporting-s3]'` or
166+
`'adcp[reporting-azure]'`. A binding's `location` holds only non-secret coordinates, and its optional
167+
`credential_ref` names one of the clients you injected; an unknown reference fails `UNSAFE_BINDING`. Provider errors
168+
surface only as stable `ReportingRowStoreError` codes. Row buckets are separate from Managed Delivery buckets. A
169+
native version always identifies exactly one write, and every read and delete is pinned to it. An ETag is never
170+
enough, because an identical upload re-created at the same key has the same ETag and a delayed delete would remove
171+
the replacement.
172+
173+
**Google Cloud Storage** (`adcp.reporting.ledger.row_storage_gcs`):
174+
175+
```python
176+
from google.cloud import storage
177+
from adcp.reporting.ledger.row_storage_gcs import create_gcs_reporting_row_object_provider_v1
178+
179+
gcs = create_gcs_reporting_row_object_provider_v1(
180+
storage=storage.Client(), # keyless credentials: workload identity or ADC
181+
clients={"tenant-eu": eu_client}, # optional, selected by credential_ref
182+
)
183+
# ReportingObjectRowBindingV1(provider="gcs", location={"bucket": "acme-adcp-rows"}, prefix="adcp-rows")
184+
# providers={"gcs": gcs}
185+
```
186+
187+
- Writes are single-request uploads with `if_generation_match=0` and a declared MD5. The recorded native version is
188+
the object generation. Reads request raw stored bytes, so GCS never transcodes them.
189+
- The probe requires uniform bucket-level access and enforced public access prevention. Versioning, soft delete and
190+
retention policies are allowed. A lifecycle `Delete` rule that can match live objects under the binding prefix
191+
(no `matchesPrefix`, or one that overlaps it) is refused; rules limited to noncurrent versions are allowed. Keep
192+
retention-driven deletion in the ledger, not in bucket lifecycle.
193+
- The client needs `storage.buckets.get` for the probe plus object create, get and delete.
194+
195+
**Amazon S3 and S3-compatible stores** (`adcp.reporting.ledger.row_storage_s3`):
196+
197+
```python
198+
import boto3
199+
from adcp.reporting.ledger.row_storage_s3 import create_s3_reporting_row_object_provider_v1
200+
201+
s3 = create_s3_reporting_row_object_provider_v1(client=boto3.client("s3", region_name="us-east-1"))
202+
# ReportingObjectRowBindingV1(provider="s3", location={"bucket": "acme-adcp-rows", "region": "us-east-1"}, ...)
203+
```
204+
205+
- **Bucket versioning is required.** Writes use `put_object` with `IfNoneMatch="*"` (a 409 is retried once). The
206+
recorded native version is the write-unique `VersionId`; reads pin it, and a delete confirms the exact version
207+
with `head_object` first, then deletes that version, which removes it permanently without a delete marker. The
208+
probe refuses any bucket whose versioning status is not `Enabled`, and a write that returns no `VersionId`.
209+
- Endpoint, region, credentials and path style come only from the client you inject; `location` accepts just `bucket`
210+
and an optional `region`, which the probe checks against the client.
211+
- **Bound the client's own timeouts.** botocore takes timeouts from the client's `Config`, not per call, and the
212+
call budget cannot stop a thread already blocked on a socket. The probe therefore refuses a client whose
213+
`connect_timeout` or `read_timeout` is `None` or non-positive. botocore's defaults (60 seconds each) pass; set
214+
tighter values with `botocore.config.Config(connect_timeout=..., read_timeout=...)`. GCS and Azure carry the budget
215+
into each call as `timeout=` instead.
216+
- S3-compatible stores may ignore `If-None-Match`, so the probe writes a probe key under the binding prefix twice and
217+
requires the second write to fail with 412, then deletes it. It also refuses missing or inaccessible buckets, a
218+
bucket policy reported as public, and enabled lifecycle `Expiration` rules that can match the prefix.
219+
- The client needs `s3:ListBucket`, `s3:GetLifecycleConfiguration`, `s3:GetBucketVersioning`, object put and get,
220+
`s3:GetObjectVersion` and `s3:DeleteObjectVersion`. `s3:GetBucketPolicyStatus` is optional.
221+
222+
**Azure Blob Storage** (`adcp.reporting.ledger.row_storage_azure`):
223+
224+
```python
225+
from azure.identity import DefaultAzureCredential
226+
from azure.storage.blob import BlobServiceClient
227+
from adcp.reporting.ledger.row_storage_azure import create_azure_blob_reporting_row_object_provider_v1
228+
229+
azure = create_azure_blob_reporting_row_object_provider_v1(
230+
client=BlobServiceClient("https://acmerows.blob.core.windows.net", credential=DefaultAzureCredential())
231+
)
232+
# ReportingObjectRowBindingV1(provider="azure_blob", location={"account": "acmerows", "container": "rows"}, ...)
233+
```
234+
235+
- **Blob versioning is required** on the storage account. Writes are block-blob uploads with `If-None-Match: *`;
236+
the recorded native version is the write-unique `version_id`, and a delete removes the base blob when it is still
237+
that version, then the version itself. The probe refuses containers that allow anonymous read access or that do
238+
not enforce create-only writes, and a write that returns no `version_id`.
239+
- The client you inject must belong to the binding's `account`.

‎pyproject.toml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,12 @@ dev = [
175175
# In-memory span exporter used by the optional observability tests. The
176176
# library itself imports only opentelemetry-api and never configures an SDK.
177177
"opentelemetry-sdk>=1.25,<2",
178+
# Cloud row-storage providers are tested against the real client libraries;
179+
# moto emulates S3 including versioning and conditional writes.
180+
"google-cloud-storage>=2.18",
181+
"boto3>=1.35",
182+
"azure-storage-blob>=12.20",
183+
"moto[s3]>=5.0",
178184
]
179185
docs = [
180186
"pdoc3>=0.10.0",
@@ -187,6 +193,19 @@ pg = [
187193
"psycopg[binary]>=3.1.0",
188194
"psycopg-pool>=3.2.0",
189195
]
196+
reporting-gcs = [
197+
# Row storage on Google Cloud Storage. The provider takes a client you build,
198+
# with keyless credentials; it never imports the library itself.
199+
"google-cloud-storage>=2.18",
200+
]
201+
reporting-s3 = [
202+
# Row storage on Amazon S3 or an S3-compatible store (needs bucket versioning).
203+
"boto3>=1.35",
204+
]
205+
reporting-azure = [
206+
# Row storage on Azure Blob Storage (needs blob versioning on the account).
207+
"azure-storage-blob>=12.20",
208+
]
190209
observability = [
191210
# API-only library instrumentation: without an application-configured SDK
192211
# provider/exporter, OpenTelemetry remains a non-recording no-op.

0 commit comments

Comments
 (0)