Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,9 @@ temp/
.tmp
# Personal workflow artifacts (specs/plans/runbooks from superpowers skill)
docs/superpowers/

# test-infra checkout and kubeconfig, and remote Taskfile cache, for task e2e:*
.test-infra
.task
# test-infra installs kind here when the installed version differs from its pin
/kind
113 changes: 109 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ webhooks.

The CRDs are defined. The operator sets owners and reviewers in admission,
reconciles votes, follows, submission reviews and security reports, and
stores attachments. Triage, notifications and digests, audit and search
config are tracked as sub-issues of the epic.
stores attachments. Activity and search are configured. Triage,
notifications and digests are tracked as sub-issues of the epic.

| Resource | Who writes it | Who reads it |
| --- | --- | --- |
Expand Down Expand Up @@ -107,6 +107,42 @@ Milo audits feedback writes like any other CRD. Its audit policy (in `datum-clou

`config/milo/activity` holds an `ActivityPolicy` per kind, so people's actions read as sentences in activity timelines: voting, following, sending feedback, staff decisions, hiding posts and editing sources. The service's own components are `system:` users and are left out. A delete's audit event carries a `Status`, not the deleted object, so a `ValidatingAdmissionPolicy` (`post-ref.yaml`) adds the post's name to vote and subscription deletes as the audit annotation `post-ref.feedback.miloapis.com/post`, and the unvote and unfollow rules read it there. Security reports and their triage have no `ActivityPolicy`: the activity service shows every entry to anyone who can read the platform timeline, with no per-entry access check, so an entry would reveal that a report exists. `test/config` compiles every rule in the activity service's CEL environment, checks that rules never read bodies a metadata-only kind or a delete won't have, renders delete events written by an envtest API server, and checks no policy covers security reports.

## Search

`config/milo/search` has a `ResourceIndexPolicy` for `Post`, `Release` and `ChangelogEntry`.

- **What's searchable:** title, summary or body, categories, labels and state. Derived fields (`votes`, `trendingScore`, `delivery`) aren't indexed, since they change as side effects and would go stale.
- **Hidden posts:** the policy's condition leaves them out, and hiding a post removes it from the index. The condition reads `spec.hidden`, and the indexer can't evaluate it on an object with no `spec` (it logs an error and treats the post as not matching, which also deletes it from the index), so the `Post` CRD defaults `spec` to `{}`.
- **Never indexed:** submissions, reviews, security reports, votes and follows.
- **What it relies on:** the indexer reads objects from Milo's audit log, so it needs Milo to log feedback writes at `RequestResponse` (see Audit and activity).
- **Measuring lag:** `hack/search-lag.sh <post>` measures how long a post change takes to reach the index. Run it against staging with a platform-root kubeconfig; it hides a visible post briefly and always puts back its original `spec.hidden`.

### Who can search posts

The search service filters every `ResourceSearchQuery` by the caller's Milo context, taken from the `iam.miloapis.com/parent-type` and `parent-name` user extras (`internal/registry/resourcesearchquery` in `milo-os/search`):

- **Platform root** (no parent context): no filter. The query sees every indexed document from every tenant.
- **User, organization or project control plane**: only documents indexed from that same user, organization or project.

feedback-sync and the operator write posts, releases and changelog entries at the platform root, so they're indexed as platform documents. **Only platform-root queries find them.** A query in a user, organization or project context returns no posts and no error. That rules out the cloud portal's user-context calls and `datumctl` in an org or project context.

Similar posts and keyword search therefore run on a server, at the platform root:

| Caller | How it searches posts |
| --- | --- |
| datum.net | its server's service account, at the platform root |
| feedback-operator (submission triage) | its own identity, at the platform root |
| cloud portal | its server, with a platform service account like the one behind its post cache; not with the user's token in the user context |
| `datumctl`, Patch | can't query with the user's credentials; they need a server-side call (a feedback endpoint or `feedback-mcp`), not built yet |

The caller needs `search.miloapis.com/resourcesearchqueries.create` at the platform root (search's `search.miloapis.com-searcher` role). That isn't limited to feedback: search doesn't check access per object or per kind, and a platform-root query with no `targetResources` searches every index across every tenant. So:

- Never grant it to end users.
- Always set `targetResources` to the feedback kinds, and never pass through targets or tenant filters a client supplies.
- Don't filter on the result's tenant. Milo marks root requests as scope `global`, so posts indexed from the audit log carry tenant type `Global` while a reindex stores `platform`.

Nothing in `config/milo` grants this yet. `hack/search-scope.sh` checks the behavior above against a real environment.

## Roles

| Role | For |
Expand Down Expand Up @@ -138,10 +174,14 @@ config/base/ operator and uploads Deployments, Services, leader ele
config/components/ milo-integration: Milo kubeconfig, client and webhook certificates
config/milo/ applied to Milo: IAM (iam/), RBAC for the service's
components (rbac/), webhook config (webhook/),
activity rules (activity/), and optional role
bindings (bindings/)
activity rules (activity/), search index policies
(search/), and optional role bindings (bindings/)
config/samples/ an example of every resource
hack/search-lag.sh measures how long a post change takes to reach search
hack/search-scope.sh checks which Milo contexts can find a post in search
test/config/ checks config/ against the CRDs
test/e2e/ Chainsaw end-to-end tests in kind (see End-to-end tests)
hack/e2e/ deploys the end-to-end environment
```

## Development
Expand All @@ -153,6 +193,71 @@ task generate # after changing types in pkg/apis: deepcopy and CRDs
task docker-build
```

### End-to-end tests

`task e2e:up` builds a kind cluster from
[datum-cloud/test-infra](https://github.com/datum-cloud/test-infra) (cert-manager,
Envoy Gateway, Flux), deploys Milo into it from a Milo checkout, and installs
feedback in that Milo. `task e2e:test` runs the [Chainsaw](https://kyverno.github.io/chainsaw/)
tests in `test/e2e/` against it. Docker, kind, kubectl, kustomize, jq and yq
must be installed.

```bash
task e2e:up MILO_DIR=../milo # MILO_DIR defaults to a sibling ../milo
task e2e:test # or: task e2e:test -- test/e2e/votes
task e2e:redeploy # rebuild feedback and restart it
task e2e:down # deletes the kind cluster
```

`e2e:up` runs `e2e:cluster`, `e2e:milo` (Milo's own `dev:build`, `dev:load`
and `dev:deploy`, then `e2e:milo-identities`), `e2e:build` and `e2e:deploy`;
each can be run on its own. Milo answers on https://localhost:30443.

What runs where:

- **In Milo:** the CRDs, `config/milo/rbac` and the admission webhooks
(with the webhook CA as `caBundle`, since cert-manager can't inject into
Milo), and the `post-ref` admission policy from `config/milo/activity`.
`config/milo/iam` is checked with a server-side dry run only.
- **In the cluster** (`config/overlays/e2e`): the operator and
`feedback-uploads` (one replica each), a self-signed webhook certificate
from cert-manager, MinIO as the attachments bucket through the
`s3-attachments` component, and in `milo-system` a stand-in for Milo's
audit log processor that logs the audit events Milo sends it.
- **Identities:** `hack/e2e/milo-identities.sh` adds static tokens to Milo's
token file (`config/overlays/e2e/milo/tokens.csv`) for
`system:feedback-operator@feedback.miloapis.com`,
`system:feedback-uploads@feedback.miloapis.com` and
`system:feedback-sync@feedback.miloapis.com` (production uses client
certificates for the same usernames), and for `other-user`, `limit-user`,
`staff-user` and `security-user` besides Milo's `test-user`.

test-infra's Milo authorizes with Kubernetes RBAC, not Milo IAM. The tests
grant the IAM roles' permissions as ClusterRoles instead
(`hack/e2e/iam-rbac.sh`): `feedback.miloapis.com-user` to every signed-in
user, `-reviewer` to `staff-user` and `-security-team` to `security-user`.
RBAC can't scope a grant to a user's own context, so the tests don't cover
what Milo IAM decides by context: for example, that a user can't act in
another user's context, or read a submission's attachments in the platform
context.

| Test | What it checks |
| --- | --- |
| `votes` | owner webhook, `<post>.<user>` names, owner filtering, follows, vote totals, post named in delete audit events |
| `hidden-posts` | hidden posts invisible to users, visible to staff, no votes on them |
| `merge-chain` | votes moved along merge chains; a hidden target keeps them, with a condition and no hot loop |
| `submissions` | review created, reviewer recorded, Rejected and Approved carried out |
| `rate-limit` | 429 after the submission limit (`--submission-window=30s` here) |
| `attachments` | upload, attach, copy, download and delete through `feedback-uploads` and MinIO, called from inside the cluster |
| `security` | MoveToSecurity leaves an unrelated report with the target name alone, then moves once it's gone |
| `operator-readiness` | the operator isn't ready before its webhook server serves TLS |

Not covered: `feedback-sync` and GitHub (the tests write posts with sync's
identity), Milo IAM, activity and search (their services don't run in
test-infra), notifications and digests.

### Running the operator by hand

Run the operator against a Milo you can reach, in a container:

```bash
Expand Down
119 changes: 119 additions & 0 deletions Taskfile.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,23 @@ vars:
CONTROLLER_TOOLS_VERSION: "v0.22.0"
ENVTEST_VERSION: "release-0.25"
ENVTEST_K8S_VERSION: "1.37.x"
CHAINSAW_VERSION: "v0.2.15"
# End-to-end tests run in a kind cluster from datum-cloud/test-infra, with
# Milo built from MILO_DIR (a checkout of milo-os/milo, by default a
# sibling of this one). Keep the test-infra ref in step with Milo's.
TEST_INFRA_CLUSTER_NAME: "test-infra"
TEST_INFRA_REPO_REF: "v0.5.0"
MILO_DIR: '{{.MILO_DIR | default (printf "%s/../milo" .ROOT_DIR)}}'
E2E_IMAGE_TAG: "e2e"
E2E_OVERLAY: "config/overlays/e2e"

includes:
# Must set TASK_X_REMOTE_TASKFILES=1 to use this feature.
test-infra:
taskfile: https://raw.githubusercontent.com/datum-cloud/test-infra/{{.TEST_INFRA_REPO_REF}}/Taskfile.yml
checksum: 4995ce0e9bc4290b3a0175dee77af2570d715b435d5f3ff65f9cb6c491837bcb
vars:
REPO_REF: "{{.TEST_INFRA_REPO_REF}}"

tasks:
default:
Expand Down Expand Up @@ -90,3 +107,105 @@ tasks:
cmds:
- kustomize build config/base > /dev/null
- kustomize build config/overlays/core-control-plane > /dev/null

# End-to-end tests. See "End-to-end tests" in README.md.
e2e:up:
desc: Create the kind cluster, deploy Milo from MILO_DIR and deploy feedback into it
cmds:
- task: e2e:cluster
- task: e2e:milo
- task: e2e:build
- task: e2e:deploy

e2e:cluster:
desc: Create the test-infra kind cluster, or reuse it
cmds:
- task: test-infra:ensure-repo
# cluster-up only writes the kubeconfig when it creates the cluster.
- |
if kind get clusters | grep -qx "{{.TEST_INFRA_CLUSTER_NAME}}"; then
kind get kubeconfig --name "{{.TEST_INFRA_CLUSTER_NAME}}" > .test-infra/kubeconfig
fi
- task: test-infra:cluster-up

e2e:milo:
desc: Build Milo from MILO_DIR, load it into the cluster and deploy it
cmds:
- |
set -e
MILO_DIR="{{.MILO_DIR}}"
if [ ! -f "$MILO_DIR/Taskfile.yaml" ]; then
echo "No Milo checkout at MILO_DIR=$MILO_DIR; set MILO_DIR to one" >&2
exit 1
fi
# Milo's tasks find the cluster through its own .test-infra.
[ -e "$MILO_DIR/.test-infra" ] || ln -s "{{.ROOT_DIR}}/.test-infra" "$MILO_DIR/.test-infra"
kind get kubeconfig --name "{{.TEST_INFRA_CLUSTER_NAME}}" > "$MILO_DIR/.test-infra/kubeconfig"
cd "$MILO_DIR"
echo "Deploying Milo from $MILO_DIR at $(git rev-parse --short HEAD)"
TASK_X_REMOTE_TASKFILES=1 task dev:build dev:load dev:deploy
# dev:deploy doesn't restart Milo when only the image changed.
task test-infra:kubectl -- -n milo-system rollout restart deployment/milo-apiserver deployment/milo-controller-manager
task test-infra:kubectl -- -n milo-system rollout status deployment/milo-apiserver deployment/milo-controller-manager --timeout=300s
- task: e2e:milo-identities

e2e:milo-identities:
desc: Add feedback's service identities and test users to Milo's static tokens
cmds:
- hack/e2e/milo-identities.sh

e2e:build:
desc: Build the feedback image and load it into the cluster
cmds:
- task: docker-build
vars:
IMAGE_TAG: "{{.E2E_IMAGE_TAG}}"
- kind load docker-image "{{.IMAGE_NAME}}:{{.E2E_IMAGE_TAG}}" --name "{{.TEST_INFRA_CLUSTER_NAME}}"

e2e:deploy:
desc: Install feedback in Milo and run the operator, uploads and MinIO in the cluster
cmds:
- hack/e2e/deploy.sh
env:
OVERLAY: "{{.E2E_OVERLAY}}"
IMAGE: "{{.IMAGE_NAME}}:{{.E2E_IMAGE_TAG}}"

e2e:redeploy:
desc: Rebuild the feedback image and restart the operator and uploads
cmds:
- task: e2e:build
- task: e2e:deploy
- task test-infra:kubectl -- -n feedback-system rollout restart deployment/feedback-operator deployment/feedback-uploads
- task test-infra:kubectl -- -n feedback-system rollout status deployment/feedback-operator deployment/feedback-uploads --timeout=180s

e2e:test:
desc: Run the chainsaw end-to-end tests (pass test directories after --)
deps:
- install-chainsaw
cmds:
- |
set -e
if ! curl -skf -o /dev/null -H "Authorization: Bearer test-admin-token" https://localhost:30443/readyz; then
echo "Milo isn't answering on https://localhost:30443; run task e2e:up first" >&2
exit 1
fi
KUBECONFIG="{{.ROOT_DIR}}/test/e2e/kubeconfig-milo" \
INFRA_KUBECONFIG="{{.ROOT_DIR}}/.test-infra/kubeconfig" \
"{{.TOOL_DIR}}/chainsaw" test --config test/e2e/.chainsaw.yaml \
--cluster infra="{{.ROOT_DIR}}/.test-infra/kubeconfig" \
{{if .CLI_ARGS}}{{.CLI_ARGS}}{{else}}test/e2e/{{end}}

e2e:down:
desc: Delete the kind cluster (only when run explicitly)
cmds:
- task: test-infra:cluster-down

install-chainsaw:
desc: Install chainsaw into bin/
status:
- test -x {{.TOOL_DIR}}/chainsaw-{{.CHAINSAW_VERSION}}
cmds:
- mkdir -p {{.TOOL_DIR}}
- GOBIN={{.TOOL_DIR}} go install github.com/kyverno/chainsaw@{{.CHAINSAW_VERSION}}
- mv {{.TOOL_DIR}}/chainsaw {{.TOOL_DIR}}/chainsaw-{{.CHAINSAW_VERSION}}
- ln -sf {{.TOOL_DIR}}/chainsaw-{{.CHAINSAW_VERSION}} {{.TOOL_DIR}}/chainsaw
5 changes: 4 additions & 1 deletion config/crd/bases/feedback.miloapis.com_posts.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,10 @@ spec:
metadata:
type: object
spec:
description: PostSpec holds the fields staff control.
default: {}
description: |-
Spec holds the fields staff control. It defaults to {} so every post
has one, which the search index condition on spec.hidden needs.
properties:
hidden:
description: Hidden removes the post from every client and from search.
Expand Down
4 changes: 3 additions & 1 deletion config/milo/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# What the service registers in Milo: ProtectedResources and Roles (iam/),
# Kubernetes RBAC for the service's own components (rbac/), the admission
# webhooks Milo calls (webhook/) and activity timeline rules (activity/). The CRDs are in config/crd. bindings/ is
# webhooks Milo calls (webhook/), activity timeline rules (activity/) and
# search index policies (search/). The CRDs are in config/crd. bindings/ is
# separate and optional: who holds the roles is the deployment's choice.
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
Expand All @@ -10,3 +11,4 @@ components:
- rbac/
- webhook/
- activity/
- search/
18 changes: 18 additions & 0 deletions config/milo/search/changelogentries.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: search.miloapis.com/v1alpha1
kind: ResourceIndexPolicy
metadata:
name: feedback-changelogentries
spec:
targetResource:
group: feedback.miloapis.com
version: v1alpha1
kind: ChangelogEntry
fields:
- path: .metadata.name
- path: .status.title
searchable: true
- path: .status.bodyHTML
searchable: true
- path: .status.categories[*]
searchable: true
- path: .status.publishedAt
14 changes: 14 additions & 0 deletions config/milo/search/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Search index policies for Milo's search service, for keyword search and
# similar posts while someone writes feedback. Everything here is indexed as
# a platform document, so only queries at Milo's platform root find it:
# servers search on clients' behalf (see "Who can search posts" in
# README.md). The indexer reads objects from the audit log, so these need
# Milo to log feedback writes at RequestResponse. Submissions, reviews,
# security reports, votes and follows are never indexed.
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
- posts.yaml
- releases.yaml
- changelogentries.yaml
42 changes: 42 additions & 0 deletions config/milo/search/posts.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Posts are found by their GitHub text, categories, labels and state. The
# indexer keeps the whole object, but only these fields are searchable;
# votes, trendingScore and delivery change as side effects and would go
# stale, so they aren't. Hidden posts are left out, and hiding a post
# removes it from the index.
#
# The indexer binds metadata, spec and status only when the object has
# them, and a condition that can't be evaluated counts as no match. has()
# can't test for a missing spec, so the Post CRD defaults spec to {}.
#
# Posts are written at Milo's platform root, so they're indexed as platform
# documents and only platform-root ResourceSearchQueries find them; user,
# organization and project contexts get nothing. See "Who can search posts"
# in README.md.
apiVersion: search.miloapis.com/v1alpha1
kind: ResourceIndexPolicy
metadata:
name: feedback-posts
spec:
targetResource:
group: feedback.miloapis.com
version: v1alpha1
kind: Post
conditions:
- name: visible
expression: "!has(spec.hidden) || !spec.hidden"
fields:
- path: .metadata.name
- path: .status.title
searchable: true
- path: .status.summary
searchable: true
- path: .status.bodyHTML
searchable: true
- path: .status.categories[*]
searchable: true
- path: .status.labels[*]
searchable: true
- path: .status.state
searchable: true
- path: .status.github.org
- path: .status.github.repository
Loading
Loading