Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 26 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ webhooks.
## Status

The CRDs are defined. The operator sets owners and reviewers in admission,
and reconciles votes, follows and submission reviews. Attachments, triage,
security reports and digests are tracked as sub-issues of the epic.
reconciles votes, follows and submission reviews, and stores attachments.
Triage, security reports and digests are tracked as sub-issues of the epic.

| Resource | Who writes it | Who reads it |
| --- | --- | --- |
Expand Down Expand Up @@ -68,6 +68,25 @@ decided on a `SubmissionReview`.
- `Approved` waits until feedback-sync creates the GitHub issue and syncs a post annotated `feedback.miloapis.com/submission: <name>`. It then links the submission to that post and has the submitter follow (`reason: submitted`) and vote.
- The `Applied` condition and `status.applied` record when the decision has been fully carried out.

## Attachments

CRDs can't take file uploads, so `feedback uploads` (the `feedback-uploads` Deployment) handles them. Clients use the same bearer token they use for Milo.

```
POST /users/{user}/uploads/{submissions|securityreports}?name=shot.png (body: the file, with its Content-Type)
GET /users/{user}/{submissions|securityreports}/{name}/attachments/{ref} (the owner)
GET /{submissions|securityreports}/{name}/attachments/{ref} (staff)
```

- **Checking the caller.** A Milo `TokenReview` identifies the caller. A `SubjectAccessReview` checks that they may create (to upload) or get (to download) that resource in the context the path names. In a user's context, the service also checks that the object is theirs.
- **Files accepted.** Files are type-checked against their content: PNG, JPEG, GIF, WebP, PDF, text and JSON only. The size limit is 10 MiB by default, and each user gets a per-minute upload limit.
- **Storage.** Files land under `uploads/<purpose>/<user>/` in the bucket.
- **Limits per replica.** Downloads stream from the bucket. Each upload is buffered, up to the size limit, to check its type. `--max-concurrent-uploads` (default 4) and `--max-concurrent-downloads` (default 32) cap transfers on one replica; past them it answers 503 with `Retry-After`. One user may hold at most `--max-concurrent-uploads-per-user` (default 2) and `--max-concurrent-downloads-per-user` (default 8) of them, so slow requests from one user can't keep everyone else out; past that they get 429 with `Retry-After`. Each transfer gets a deadline sized for its file, at 64 KiB/s plus 30 seconds.
- **Attaching.** The `attachments` mutating webhook checks every `uploadRef` on a new submission or report is the owner's upload for that purpose, and sets its name and type from the upload.
- **Keeping and deleting.** The operator copies attached files under `submissions/<name>/` as soon as the submission exists, and adds the `feedback.miloapis.com/attachments` finalizer. Deleting the submission deletes its files.
- **Lifecycle rule.** The bucket's lifecycle rule should delete `uploads/` after a day.
- **Configuration.** Set the bucket in the `feedback-config` ConfigMap (`attachments-bucket`). The operator and the upload service share it.

## Roles

| Role | For |
Expand All @@ -84,15 +103,18 @@ client certificates and get Kubernetes RBAC in Milo (`config/milo/rbac`).
## Layout

```
cmd/feedback/ operator and version commands
cmd/feedback/ operator, uploads and version commands
internal/attachments/ the attachments bucket
internal/controller/ reconcilers
internal/identity/ who a request comes from, from Milo's user info
internal/index/ cache field indexes shared by reconcilers and webhooks
internal/milo/ voter lookups in Milo
internal/ratelimit/ per-user token bucket
internal/uploads/ the feedback-uploads HTTP service
internal/webhook/ admission webhooks Milo calls
pkg/apis/feedback/ feedback.miloapis.com/v1alpha1 types (CRDs)
config/crd/ generated CRDs, installed in Milo
config/base/ operator Deployment, Services, leader election RBAC
config/base/ operator and uploads Deployments, Services, leader election RBAC
config/components/ milo-integration: Milo kubeconfig, client and webhook certificates
config/milo/ applied to Milo: IAM (iam/), RBAC for the service's
components (rbac/), webhook config (webhook/), and
Expand Down
68 changes: 68 additions & 0 deletions cmd/feedback/flags_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
package main

import (
"os"
"path/filepath"
"regexp"
"testing"

appsv1 "k8s.io/api/apps/v1"
"sigs.k8s.io/yaml"
)

// TestDeploymentArgsParse checks every Deployment in config/base passes only
// flags its command accepts, so a renamed or unbound flag fails here rather
// than as a crash loop in a cluster.
func TestDeploymentArgsParse(t *testing.T) {
files, err := filepath.Glob("../../config/base/*/deployment.yaml")
if err != nil || len(files) == 0 {
t.Fatalf("no deployments found: %v", err)
}
envRef := regexp.MustCompile(`\$\([A-Z_]+\)`)
for _, f := range files {
data, err := os.ReadFile(f)
if err != nil {
t.Fatal(err)
}
var d appsv1.Deployment
if err := yaml.Unmarshal(data, &d); err != nil {
t.Fatalf("%s: %v", f, err)
}
for _, c := range d.Spec.Template.Spec.Containers {
if len(c.Command) < 2 || filepath.Base(c.Command[0]) != "feedback" {
continue
}
t.Run(d.Name+"/"+c.Name, func(t *testing.T) {
root := newRootCommand()
cmd, _, err := root.Find(c.Command[1:])
if err != nil || cmd == root {
t.Fatalf("command %v not found: %v", c.Command[1:], err)
}
args := make([]string, len(c.Args))
for i, a := range c.Args {
args[i] = envRef.ReplaceAllString(a, "x")
}
if err := cmd.ParseFlags(args); err != nil {
t.Errorf("%s args %q: %v", filepath.Base(f), c.Args, err)
}
})
}
}
}

func TestCheckUserCap(t *testing.T) {
for _, c := range []struct {
perUser, replica int
ok bool
}{
{2, 4, true},
{4, 4, false},
{5, 4, false},
{0, 4, true},
{4, 0, true},
} {
if err := checkUserCap("uploads", c.perUser, c.replica); (err == nil) != c.ok {
t.Errorf("per user %d, replica %d: %v", c.perUser, c.replica, err)
}
}
}
2 changes: 1 addition & 1 deletion cmd/feedback/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ func newRootCommand() *cobra.Command {
roadmap posts synced from GitHub, votes, submissions and security reports.`,
SilenceUsage: true,
}
cmd.AddCommand(newOperatorCommand(), newVersionCommand())
cmd.AddCommand(newOperatorCommand(), newUploadsCommand(), newVersionCommand())
return cmd
}

Expand Down
16 changes: 14 additions & 2 deletions cmd/feedback/operator.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
ctrlwebhook "sigs.k8s.io/controller-runtime/pkg/webhook"

"go.miloapis.com/feedback/internal/attachments"
"go.miloapis.com/feedback/internal/controller"
"go.miloapis.com/feedback/internal/index"
"go.miloapis.com/feedback/internal/milo"
Expand All @@ -42,6 +43,7 @@ type operatorOptions struct {
webhookCertDir string
submissionLimit int
submissionWindow time.Duration
attachmentsBucket string
}

func newOperatorCommand() *cobra.Command {
Expand All @@ -65,6 +67,7 @@ func newOperatorCommand() *cobra.Command {
f.StringVar(&o.webhookCertDir, "webhook-cert-dir", "/etc/kubernetes/pki/webhook", "Directory holding tls.crt and tls.key for the webhook server.")
f.IntVar(&o.submissionLimit, "submission-limit", 5, "Submissions a user may send per --submission-window; 0 disables the limit.")
f.DurationVar(&o.submissionWindow, "submission-window", 10*time.Minute, "Window for --submission-limit.")
f.StringVar(&o.attachmentsBucket, "attachments-bucket", "", "Bucket URL for attachments, shared with feedback-uploads. Empty refuses attachments.")
goFlags := flag.NewFlagSet("zap", flag.ExitOnError)
zapOpts.BindFlags(goFlags)
f.AddGoFlagSet(goFlags)
Expand Down Expand Up @@ -105,13 +108,22 @@ func runOperator(o operatorOptions) error {
if err := index.Add(ctx, mgr); err != nil {
return fmt.Errorf("adding indexes: %w", err)
}
webhook.Register(mgr, webhook.Options{SubmissionLimit: o.submissionLimit, SubmissionWindow: o.submissionWindow})
var files *attachments.Store
if o.attachmentsBucket != "" {
if files, err = attachments.Open(ctx, o.attachmentsBucket); err != nil {
return err
}
defer files.Close()
} else {
log.Info("No --attachments-bucket; submissions and reports with attachments will be refused")
}
webhook.Register(mgr, webhook.Options{SubmissionLimit: o.submissionLimit, SubmissionWindow: o.submissionWindow, Files: files})

users, err := milo.NewClient(miloConfig)
if err != nil {
return fmt.Errorf("creating Milo user client: %w", err)
}
if err := controller.Setup(ctx, mgr, controller.Options{Users: users}); err != nil {
if err := controller.Setup(ctx, mgr, controller.Options{Users: users, Files: files}); err != nil {
return fmt.Errorf("setting up controllers: %w", err)
}

Expand Down
122 changes: 122 additions & 0 deletions cmd/feedback/uploads.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
package main

import (
"context"
"errors"
"flag"
"fmt"
"net/http"
"time"

"github.com/spf13/cobra"
"k8s.io/client-go/kubernetes"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/log/zap"

"go.miloapis.com/feedback/internal/attachments"
"go.miloapis.com/feedback/internal/ratelimit"
"go.miloapis.com/feedback/internal/uploads"
)

type uploadsOptions struct {
miloKubeconfig string
addr string
bucket string
maxSize int
perMinute float64
burst int
maxUploads int
maxDownloads int
maxUserUploads int
maxUserDownloads int
}

func newUploadsCommand() *cobra.Command {
o := uploadsOptions{}
zapOpts := zap.Options{}
cmd := &cobra.Command{
Use: "uploads",
Short: "Serve attachment uploads and downloads",
RunE: func(cmd *cobra.Command, _ []string) error {
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&zapOpts)))
return runUploads(ctrl.SetupSignalHandler(), o)
},
}
f := cmd.Flags()
f.StringVar(&o.miloKubeconfig, "milo-kubeconfig", "", "Kubeconfig for Milo. Empty uses the in-cluster config.")
f.StringVar(&o.addr, "listen-address", ":8080", "Address to serve HTTP on. TLS is terminated in front of the service.")
f.StringVar(&o.bucket, "attachments-bucket", "", "Bucket URL for attachments: gs://BUCKET, s3://BUCKET?region=REGION, or s3://BUCKET?endpoint=URL&region=REGION&use_path_style=true for an S3-compatible store. file:///PATH for local development.")
f.IntVar(&o.maxSize, "attachment-max-size", attachments.DefaultMaxSize, "Largest file accepted, in bytes.")
f.Float64Var(&o.perMinute, "uploads-per-minute", 2, "Uploads a user may make per minute, on average; 0 disables the limit.")
f.IntVar(&o.burst, "uploads-burst", 10, "Uploads a user may make at once.")
f.IntVar(&o.maxUploads, "max-concurrent-uploads", uploads.DefaultMaxUploads, "Uploads one replica handles at once; more get 503 with Retry-After. Each can hold up to twice --attachment-max-size, plus about 16 MiB of bucket client buffers, so lower this if you raise that. 0 disables the cap.")
f.IntVar(&o.maxDownloads, "max-concurrent-downloads", uploads.DefaultMaxDownloads, "Downloads one replica streams at once; more get 503 with Retry-After. 0 disables the cap.")
f.IntVar(&o.maxUserUploads, "max-concurrent-uploads-per-user", uploads.DefaultMaxUploadsPerUser, "Uploads one user may run at once on one replica; more get 429 with Retry-After. Must be below --max-concurrent-uploads, so one user can't take every slot. 0 disables the cap.")
f.IntVar(&o.maxUserDownloads, "max-concurrent-downloads-per-user", uploads.DefaultMaxDownloadsPerUser, "Downloads one user may run at once on one replica; more get 429 with Retry-After. Must be below --max-concurrent-downloads. 0 disables the cap.")
goFlags := flag.NewFlagSet("zap", flag.ExitOnError)
zapOpts.BindFlags(goFlags)
f.AddGoFlagSet(goFlags)
return cmd
}

func runUploads(ctx context.Context, o uploadsOptions) error {
log := ctrl.Log.WithName("uploads")
if o.bucket == "" {
return errors.New("--attachments-bucket is required")
}
if err := checkUserCap("uploads", o.maxUserUploads, o.maxUploads); err != nil {
return err
}
if err := checkUserCap("downloads", o.maxUserDownloads, o.maxDownloads); err != nil {
return err
}
files, err := attachments.Open(ctx, o.bucket)
if err != nil {
return err
}
defer files.Close()
files.MaxSize = o.maxSize

cfg, err := restConfig(o.miloKubeconfig)
if err != nil {
return fmt.Errorf("loading Milo config: %w", err)
}
kube, err := kubernetes.NewForConfig(cfg)
if err != nil {
return err
}
objects, err := client.New(cfg, client.Options{Scheme: scheme})
if err != nil {
return err
}
milo := uploads.Milo{Client: kube}
s := &uploads.Server{
Files: files, Authn: milo, Authz: milo, Objects: objects,
Limiter: ratelimit.NewLimiter(o.perMinute, o.burst),
Uploads: uploads.NewSlots(o.maxUploads), Downloads: uploads.NewSlots(o.maxDownloads),
UploadsPerUser: uploads.NewUserSlots(o.maxUserUploads),
DownloadsPerUser: uploads.NewUserSlots(o.maxUserDownloads),
}
srv := uploads.NewHTTPServer(o.addr, s.Handler())
go func() {
<-ctx.Done()
shutdown, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_ = srv.Shutdown(shutdown)
}()
log.Info("Serving uploads", "address", o.addr)
if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
return err
}
return nil
}

// checkUserCap refuses a per-user cap that would let one user take every one
// of the replica's slots.
func checkUserCap(kind string, perUser, replica int) error {
if perUser > 0 && replica > 0 && perUser >= replica {
return fmt.Errorf("--max-concurrent-%s-per-user (%d) must be below --max-concurrent-%s (%d), so one user can't take every slot", kind, perUser, kind, replica)
}
return nil
}
10 changes: 10 additions & 0 deletions config/base/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Settings the environment overrides.
apiVersion: v1
kind: ConfigMap
metadata:
name: feedback-config
data:
# Bucket URL for attachments, shared by the operator and feedback-uploads:
# gs://BUCKET, or s3://BUCKET?... for S3 and S3-compatible stores (see the
# s3-attachments component).
attachments-bucket: mem://
2 changes: 2 additions & 0 deletions config/base/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- config.yaml
- operator
- uploads

images:
- name: ghcr.io/milo-os/feedback
Expand Down
6 changes: 6 additions & 0 deletions config/base/operator/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,14 @@ spec:
- --leader-elect
- --leader-election-namespace=$(POD_NAMESPACE)
- --webhook-cert-dir=/etc/kubernetes/pki/webhook
- --attachments-bucket=$(ATTACHMENTS_BUCKET)
- --zap-encoder=json
env:
- name: ATTACHMENTS_BUCKET
valueFrom:
configMapKeyRef:
name: feedback-config
key: attachments-bucket
- name: POD_NAMESPACE
valueFrom:
fieldRef:
Expand Down
Loading
Loading