Skip to content

feat(storage): presigned PUT uploads for stores without POST (Cloudflare R2) - #248

Merged
houko merged 5 commits into
mainfrom
feat/presigned-put-uploads
Oct 28, 2026
Merged

houko merged 5 commits into
mainfrom
feat/presigned-put-uploads

Conversation

@houko

@houko houko commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

No ticket: first unit of the owner's 2026-10-28 decision to keep forma's files on Cloudflare R2 like the other yldm apps.

What & why

Was: every upload (respondent files, logos, editor images) was a presigned POST form, which R2 does not support.

Now: with STORAGE_UPLOAD_METHOD=put the server also signs a presigned PUT of the exact key, type and size, and the survey runtime and admin app upload with it. The default stays post, so existing installs answer byte for byte as before.

  • Requests gain an optional fileSize; over the element's (or the 5 MB public) limit it is refused with 400 and maxFileSize before anything is signed.
  • Answers keep signedUrl/presignedFields and add uploadMethod: "put", uploadUrl, uploadHeaders only in put mode and only when fileSize was sent, so older clients still get a POST they understand.
  • Reads already go through the /storage route (server proxy), so R2 needs no ACL or public bucket. CSP connect-src already allows any https: endpoint (loopback endpoints outside production), so it is unchanged.

Where to look

Breaking changes

  • This PR contains breaking changes

None: fileSize and the three answer fields are optional and additive, and the new variable defaults to today's behaviour. New contract exchanges pin both modes; no existing exchange changed.

Migrations & env

  • New STORAGE_UPLOAD_METHOD (post default, put), documented in .env.example. The R2 bucket's CORS rule must allow PUT with Content-Type from the app origin and, for respondents on customer sites, any origin.
  • Production switches in two steps, never together, because a client that sends no fileSize gets the POST alone and R2 refuses POST:
    1. Deploy this image, then set STORAGE_UPLOAD_METHOD=put while the bucket is still MinIO (minio-critical after P6), which takes both POST and PUT, so old and new clients both upload.
    2. No sooner than two hours later, move the S3_* settings to R2. /js/surveys.umd.cjs is served with max-age=3600, stale-while-revalidate=3600, so a browser can run the runtime from before step 1 for up to two hours; the Reloader rolling restart also leaves pods on post for a while during step 1.
    • Before step 2, run the canary against the R2 bucket with its own settings (see Open gaps). After step 2, a survey or admin tab opened before step 1 and never reloaded fails its uploads until reloaded.
  • For the R2 cutover release note: management and client API integrations must send fileSize and use uploadMethod/uploadUrl/uploadHeaders when present; without fileSize they get the POST alone, which R2 refuses. In put mode the server logs Upload signed as a presigned POST only ... sent no fileSize with the workspaceId for each such request, so those integrations can be found.

How this was tested

Rerun: cd apps/server && go test ./internal/storage/ ./internal/clientapi/ ./internal/mgmtapi/ ./internal/platform/config/

Coverage

Behaviour How Outcome
PUT signs key, type, length, 120 s expiry; store refuses larger, smaller or retyped body unit (red on main) TestSignPutStoresExactlyTheSignedFile, TestSignPutEmptyFile RustFS answers 403 to each, stores only the exact body
fileSize validated, over-limit refused, PUT fields only in put mode with a size unit (red on main) TestServiceSignUploadPut, TestStorageUploadFileSize, TestStorageUploadPut, TestV1StorageFileSize as described
Client and management answers in both modes unit (red on main) contract cases *-put, *-file-size, *-file-too-large POST-mode answers unchanged
Runtime and admin app pick PUT or POST, map maxFileSize unit (red on main) api-client.test.ts, upload.test.ts PUT body is the file, headers as signed
Store refuses ±1 byte, another type, an expired URL; URL-encoded key stored; object private; CORS when origins are given unit TestPutStoreCheck (RustFS), TestPutCanary (any store, env-gated) RustFS refuses each, stores the encoded key
Name needing URL encoding through SignUpload → PUT → Service.Open unit TestServiceSignUploadPutEncodedName 請求書 100% a+b&c=d.pdf read back through its fileUrl
Warning for a size-less request in put mode only unit (red before the change) TestServiceSignUploadPutWarnsWithoutSize one WARN with workspaceId, none in post mode or with a size
Respondent upload on /s/…, workspace logo, editor image, oversized PUT refused e2e @upload specs, local stack against MinIO RELEASE.2025-09-07 (production's image) pass with put and with post

Open gaps

  • Not run against real R2 yet: there is no forma R2 bucket or bucket-scoped token yet. The size limit of put rests on the store refusing a body whose length is not the signed Content-Length, which Cloudflare documents for Content-Type but not for Content-Length. Before step 2 above, run TestPutCanary with the bucket's production settings; it signs through the same Bucket code, checks that a body 1 byte longer, 1 byte shorter, another Content-Type and an expired URL are refused with nothing stored, that a key needing URL encoding (請求書 100% a+b&c=d--fid--….pdf) is stored, that the object is not readable unsigned, and that the CORS preflight and the PUT are allowed for each origin, then deletes its object:

    cd apps/server
    PUT_CANARY_ORIGINS=https://forma.yldm.ai,https://customer.example S3_ENDPOINT_URL=https://<account id>.r2.cloudflarestorage.com S3_REGION=auto S3_BUCKET_NAME=<bucket> S3_ACCESS_KEY=<key id> S3_SECRET_KEY=<secret> go test ./internal/storage/ -run '^TestPutCanary$' -v

    Record the result here or in the yldm-platform R2 PR. If R2 does not refuse a body of another length, put needs another size guard before production uses it.

Risks

  • A management or client API caller that never sends fileSize keeps getting POST only, which fails on R2; the warning log above finds them, and the cutover release note must tell them to send fileSize.

houko added 4 commits October 28, 2026 11:37
Cloudflare R2 takes presigned PUT, GET, HEAD and DELETE but no POST form uploads. STORAGE_UPLOAD_METHOD=put (default post, so existing S3, MinIO and RustFS installs are unchanged) signs every upload whose request carries the new optional fileSize as a presigned PUT of that object, with Content-Type and the exact Content-Length signed and the same two-minute expiry, and answers uploadMethod, uploadUrl and uploadHeaders beside the unchanged POST fields. A request without fileSize gets the POST alone. A fileSize over the upload's limit (the element's for the client API, 5 MB for the management storage route) is refused with 400 and the limit in the details before anything is signed, in either mode.

Files stay private and are read through the /storage route, so nothing depends on a public-read bucket.
The survey runtime and the admin app send the file's size with the signing request and PUT the file to uploadUrl with uploadHeaders when the answer says uploadMethod "put"; otherwise, and with any server that predates it, they POST the form as before. A 400 carrying maxFileSize reads as a file that is too large.
Adds a respondent's file upload on a public survey page and a workspace logo upload, tags the upload specs @Upload, and runs them a second time in CI against a server started with STORAGE_UPLOAD_METHOD=put, including a check that the store refuses a PUT body larger than the size it was signed for.
…T store check

putStoreCheck signs through the Bucket as the server does and checks that the store refuses a body one byte longer or shorter than signed, another Content-Type and an expired URL with nothing stored, takes a key that needs URL encoding (請求書 100% a+b&c=d), keeps the object private, and, given origins, answers the CORS preflight and the PUT for them. It runs against RustFS in the suite and, as TestPutCanary, against a real store such as the R2 bucket when PUT_CANARY_ORIGINS and its S3_* settings are set.

In put mode the server now logs a warning for each upload request without fileSize, which can only be answered with a POST, so the callers that still send no size can be found. .env.example describes the two-step switch: put while the bucket still takes POST, the new store no sooner than two hours later.
@houko
houko force-pushed the feat/presigned-put-uploads branch from 75dc180 to 441f12b Compare October 28, 2026 02:37
…loads

# Conflicts:
#	.github/workflows/e2e.yml
#	apps/server/internal/clientapi/contract_test.go
#	deploy/docker/docker-compose.yml
@houko
houko merged commit 6913709 into main Oct 28, 2026
21 checks passed
@houko
houko deleted the feat/presigned-put-uploads branch October 28, 2026 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant