Skip to content

feat: attachments through a feedback-uploads service - #38

Merged
mattdjenkinson merged 7 commits into
mainfrom
feat/crd-uploads
Nov 18, 2026
Merged

mattdjenkinson merged 7 commits into
mainfrom
feat/crd-uploads

Conversation

@mattdjenkinson

Copy link
Copy Markdown
Contributor

Part of #8. Stacked on #37.

CRDs can't take file uploads, so this adds the feedback-uploads service from the design. Submissions and security reports can now carry attachments.

feedback uploads

A new command and Deployment. Clients send the same bearer token they use for Milo.

POST /users/{user}/uploads/{submissions|securityreports}?name=shot.png   body: the file, with its Content-Type
GET  /users/{user}/{resource}/{name}/attachments/{ref}                   the owner
GET  /{resource}/{name}/attachments/{ref}                                staff
  • Who the caller is: a Milo TokenReview, made as system:feedback-uploads@feedback.miloapis.com.
  • What they may do: a SubjectAccessReview for create (upload) or get (download) on that resource. For the user paths it carries the same parent extras Milo adds in a user's own context.
    • An access review can only say whether the user may get submissions in their own context, not whether this one is theirs. So in a user's context, the service also reads the object and checks spec.userRef.
    • Refused requests answer 404, so they don't reveal whether something exists.
  • What's accepted: PNG, JPEG, GIF, WebP, PDF, text and JSON. Binary types must match their content and text must be UTF-8. The size limit is 10 MiB by default (--attachment-max-size). Each user gets a per-minute limit (--uploads-per-minute, --uploads-burst; in memory, per replica). Unverified emails are refused.
  • Downloads are served with Content-Disposition: attachment, nosniff, a sandboxing CSP, and no-store.

Admission and the operator

  • attachments mutating webhook, on creates of submissions and security reports:
    • every uploadRef must be the caller's own upload for that purpose, in their own user context;
    • name and content type come from the upload, not from the client;
    • the operator's own creates pass (they carry files it has already copied, for reports moved from submissions);
    • with no bucket configured, attachments are refused.
  • Submission reconciler: it copies attached uploads under submissions/<name>/ and adds the feedback.miloapis.com/attachments finalizer. On delete it removes the files, then the finalizer. Downloads fall back to the original upload until the copy exists.
  • Bucket lifecycle: the environment's rule should delete uploads/ after a day.

Config

  • config/base/uploads: Deployment, Service and ServiceAccount.
  • feedback-config ConfigMap: holds attachments-bucket, shared with the operator (mem:// in base).
  • milo-integration: mounts the uploads client certificate, trust bundle and Milo kubeconfig.
  • Milo RBAC: the operator gets update/patch on submissions, for the finalizer only.

The public route, the GCS bucket, its lifecycle rule and workload identity are deployment work (#13).

Not covered

  • Staff logging a report for an external reporter can't attach files yet, since uploads are per user.
  • Security report attachments are copied by the security reports reconciler in the next PR.

Tests

  • Store against mem://:
    • the type, sniffing, size and name checks;
    • uploads are kept separate per owner and per purpose;
    • attaching twice is harmless, and someone else's upload can't be attached;
    • downloads fall back to the upload, copying to a report works, and deleting a parent removes its files.
  • Webhook: the caller's own upload gets its name and type from the store. Refused: someone else's upload, a made-up ref, an upload for the other purpose, no bucket, and staff in the platform context. The operator passes.
  • HTTP service with fake Milo:
    • 401 with no token or a bad token, and 404 in someone else's context or for a non-feedback resource;
    • HTML is refused, and oversize files get 413;
    • uploads succeed, and downloads come with the right headers;
    • 404 for another user's submission, an unlisted ref, a missing submission, or a platform-context download without staff access;
    • staff downloads succeed.
  • Envtest: an attachment is copied with the finalizer added, and deleting the submission deletes the file, then the object.
  • task test, go vet ./..., task generate with no diff, and kustomize build of base and overlay

CRDs can't take file uploads, so feedback-uploads accepts them with the
caller's Milo token: a TokenReview says who they are and a
SubjectAccessReview whether they may create (or get) the resource in the
context they name. Files are type-checked and stored per user and
purpose. The attachments webhook checks each uploadRef is the owner's
upload and sets its name and type; the operator copies attached files
under the submission, with a finalizer that deletes them with it.
Register gocloud's s3blob driver, so --attachments-bucket accepts
s3://BUCKET URLs, with endpoint, region and use_path_style for stores
outside AWS such as Cloudflare R2 or MinIO.

Add the s3-attachments kustomize component, which gives the operator and
feedback-uploads credentials from the feedback-attachments-s3 Secret.

Run the attachment store tests against an in-process S3-compatible fake
as well as memory, and against FEEDBACK_TEST_S3_URL when it is set.

Copy link
Copy Markdown

Please address before exposing the public route: downloads load the whole file into memory, and http.TimeoutHandler buffers the response again. With a 256 MiB container limit and no download concurrency limit, parallel downloads of large attachments could exhaust a replica's memory.

Stream downloads instead, use timeouts that do not buffer the response, and cap concurrent transfers. This is a production-readiness follow-up to the approval, not a cross-user access issue.

…tHandler

Downloads now stream from the bucket to the client instead of reading the
whole file into memory, and http.TimeoutHandler, which buffered every reply
again, is replaced by connection deadlines: ReadHeaderTimeout, ReadTimeout,
WriteTimeout and IdleTimeout on the server, extended per transfer with
http.ResponseController to a deadline sized for the file (30s plus the file
at 64 KiB/s), with a matching context deadline on the bucket read or write.

Uploads and downloads each take a slot from a per-replica semaphore
(--max-concurrent-uploads, default 4; --max-concurrent-downloads, default
32) and get 503 with Retry-After when none is free. Uploads still buffer,
up to the size limit via MaxBytesReader, because Check needs the bytes.
The Deployment sets GOMEMLIMIT under its 256Mi limit.
feedback-uploads never bound the zap flags, so --zap-encoder=json from
its Deployment made it exit with "unknown flag" and crash-loop. Bind
them as the operator does, and test that every Deployment's args parse
with its command.
# Conflicts:
#	internal/webhook/submission.go
@mattdjenkinson

Copy link
Copy Markdown
Contributor Author

Fixed in 289fbd5, before the public route:

  • Streaming: downloads stream from the bucket instead of loading into memory.
  • No buffering: http.TimeoutHandler is gone. The server sets read-header, write and idle timeouts, and each transfer has its own deadline.
  • Concurrency caps: concurrent transfers are capped per replica: --max-concurrent-uploads (default 4) and --max-concurrent-downloads (default 32). Requests over a cap get 503 with Retry-After.
  • Memory: upload bodies are bounded with MaxBytesReader, and GOMEMLIMIT is 220MiB, under the 256 MiB container limit.

154ef89 fixes a crash loop the kind tests in #42 found: the uploads command didn't bind the zap flags its Deployment passes. A new test parses every Deployment's args against its command.

scotwells
scotwells previously approved these changes Nov 18, 2026
Base automatically changed from feat/crd-submissions to main November 18, 2026 15:27
@mattdjenkinson
mattdjenkinson dismissed scotwells’s stale review November 18, 2026 15:27

The base branch was changed.

@mattdjenkinson
mattdjenkinson requested a review from a team as a code owner November 18, 2026 15:27
@scotwells

Copy link
Copy Markdown

The memory-exhaustion and startup issues are fixed. One low-severity hardening item remains: transfer limits are shared across each replica, so one signed-in user can occupy all four upload slots with slow requests, temporarily giving other users 503 “busy” responses.

Please add a per-user concurrent-transfer limit alongside the replica-wide cap, with a test showing that one user cannot consume all slots. This was previously considered a non-blocking follow-up, not a critical bug; it is the remaining reason the stricter automated review has not renewed approval.

The replica-wide transfer caps were shared by everyone, so one signed-in user
could hold all four upload slots with slow requests and give every other user
503s until their deadlines ran out.

Each user may now hold at most --max-concurrent-uploads-per-user (default 2)
and --max-concurrent-downloads-per-user (default 8) transfers on a replica.
Past that they get 429 with Retry-After, before taking a replica slot or
spending rate-limit quota. The service refuses to start with a per-user cap
that isn't below the replica's, so one user can never take every slot.

TestOneUserCannotTakeEverySlot holds two of one user's uploads open mid-body
and checks that user's third gets 429 while another user's upload succeeds.
@mattdjenkinson

Copy link
Copy Markdown
Contributor Author

Thanks, added in d8d8c50.

  • Per-user cap: each user may hold at most 2 of a replica's 4 upload slots and 8 of its 32 download slots (--max-concurrent-uploads-per-user, --max-concurrent-downloads-per-user). Past that, they get 429 with Retry-After. The check comes before the replica-wide cap and the rate limiter, so a refused request holds no slot and spends no quota. It's keyed on the authenticated caller, not the user in the path.
  • Can't be configured away: the service refuses to start if a per-user cap isn't below the replica's.
  • Test: TestOneUserCannotTakeEverySlot holds two of one user's uploads open mid-body, as slow clients would. It checks that user's third upload gets 429 while another user's upload succeeds. TestUserDownloadCap covers downloads. Both fail with the cap removed.

@mattdjenkinson
mattdjenkinson merged commit 3e7b65d into main Nov 18, 2026
6 checks passed
@mattdjenkinson
mattdjenkinson deleted the feat/crd-uploads branch November 18, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants