Skip to content

test: end-to-end tests in kind against a real Milo - #42

Merged
mattdjenkinson merged 2 commits into
feat/crd-searchfrom
feat/e2e-kind
Nov 18, 2026
Merged

mattdjenkinson merged 2 commits into
feat/crd-searchfrom
feat/e2e-kind

Conversation

@mattdjenkinson

@mattdjenkinson mattdjenkinson commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #41.

End-to-end tests for the CRD stack, in a kind cluster against a real Milo, using datum-cloud/test-infra the way datum-cloud/compute does.

  • task e2e:up creates the test-infra kind cluster, builds and deploys Milo from a checkout (MILO_DIR, default ../milo), then installs feedback in it. task e2e:test runs the Chainsaw tests in test/e2e/. e2e:redeploy and e2e:down are there too.
  • In Milo: the CRDs, RBAC, admission webhooks (with the webhook CA as caBundle) and the post-ref admission policy.
  • In the cluster (config/overlays/e2e): the operator and feedback-uploads, a cert-manager webhook certificate, MinIO as the attachments bucket through the s3-attachments component, and a stand-in audit sink that logs what Milo sends.
  • Identities: static Milo tokens for the operator, uploads and sync usernames, and for test users. test-infra's Milo authorizes with RBAC, not Milo IAM, so hack/e2e/iam-rbac.sh grants the IAM roles' permissions as ClusterRoles.
Test What it checks
votes owner webhook, <post>.<user> names, owner filtering, follows, vote totals, post named in delete audit events
hidden-posts hidden posts invisible to users, visible to staff, no votes on them
merge-chain votes moved along merge chains; a hidden target keeps them, with a condition and no hot loop
submissions review created, reviewer recorded, Rejected and Approved carried out
rate-limit 429 after the submission limit
attachments upload, attach, copy, download and delete through feedback-uploads and MinIO
security MoveToSecurity leaves an unrelated report with the target name alone, then moves once it's gone
operator-readiness the operator isn't ready before its webhook server serves TLS

Not covered: feedback-sync and GitHub, Milo IAM's context checks, activity and search (their services don't run in test-infra), notifications and digests.

What it found

Writing these turned up two bugs, fixed in the PRs they belong to, each with a unit test:

Tests

Add task e2e:up, e2e:test and e2e:down, following milo-os/billing: a
test-infra kind cluster, Milo built and deployed from MILO_DIR, and
feedback installed in that Milo with the operator, feedback-uploads and
MinIO in the cluster (config/overlays/e2e).

The operator and feedback-uploads authenticate to Milo with static tokens
for their production usernames, so the admission webhooks' identity checks
and config/milo/rbac are exercised as they are. test-infra's Milo uses
RBAC, so the IAM roles are granted as equivalent ClusterRoles
(hack/e2e/iam-rbac.sh).

Chainsaw tests cover votes and owner rules, hidden posts, merge chains,
submissions and review decisions, the submission rate limit, attachments
through feedback-uploads and MinIO, moving submissions to security, and
operator readiness.

Copy link
Copy Markdown

No blocking findings in this test PR. One small follow-up: gone() in test/e2e/lib.sh treats every failed kubectl get as proof that an object was deleted. I confirmed that a connection failure also passes this check. Please distinguish NotFound from connection/auth errors so the deletion tests cannot pass for the wrong reason.

I checked the Chainsaw schemas, shell syntax, rendered overlays and focused config/command tests; those passed. I did not rerun the full kind suite. The documented IAM and search coverage gaps still need separate integration testing.

@mattdjenkinson
mattdjenkinson merged commit 43c0e78 into feat/crd-search Nov 18, 2026
6 checks passed
@mattdjenkinson
mattdjenkinson deleted the feat/e2e-kind branch November 18, 2026 03:43
mattdjenkinson added a commit to milo-os/milo that referenced this pull request Nov 18, 2026
## Summary

Closes #1020. Follow-up to #1017: a wrong or not-yet-loaded access rule
turned protection off and only logged it. Access rules now fail closed.

- **Rules come from the CRD informer cache at request time**, the same
cache the CRD handler serves from, so a custom resource is never served
before its rule.
- **Invalid declarations are refused.** A CRD's `iam.miloapis.com/`
annotations are invalid when they are:
  - blank or unknown;
  - a field that isn't a plain selectable path;
  - an owner field under `status` or a scale path;
  - `hidden-subresource` without `hidden-field`.

Requests for that resource answer 503 outside `system:masters`, from
both the filter and `AccessRuleOwnerEnforcement`.
- **CRD writes that introduce a problem are rejected (422).** This
ratchets: a CRD that's already invalid can still be updated as long as
the update doesn't add a new problem. Changing a rule returns a warning.
- **Before the cache syncs**, every resource request outside a project
answers 503 outside `system:masters`, in every context. Before,
platform-context requests were served.
- **Staleness check:** compares each CRD's resource version in storage
with the cache. While a CRD's cached copy lags, only that CRD's
resources are refused; everything else is served. Readiness fails only
before the rules first sync, so a lag shared by every replica can't take
them all out of rotation.
- **Hidden fields:** a hidden field that can't be read hides the object.
A missing one shows it, matching storage's field selector for lists.
- **Alerts:** `milo_access_rule_invalid` and `milo_access_rules_stale`
(the number of stale CRDs) gauges, with `MiloAccessRuleInvalid` and
`MiloAccessRulesStale` alerts in `config/telemetry/alerts/access-rules`.
- **Wiring:** the control plane, APIExtensions and aggregator servers
share one list of Milo's admission initializers, which includes the
access rules.

Docs: `docs/architecture/access-rules.md`.

## Test plan

- [x] `go test ./pkg/server/access/...
./internal/apiserver/admission/plugin/accessowner/...
./cmd/milo/apiserver/...`:
  - declaration parsing for each invalid case;
- 503 for invalid rules and before sync, with the `system:masters`
bypass;
  - ratcheting CRD validation;
  - a stale CRD refused while others are served and readiness passes;
  - the shared initializers give `AccessRuleOwnerEnforcement` its rules;
  - a service account in an organization context.
- [x] Prometheus rule tests (promtool 3.13.2).
- [x] `task test:end-to-end -- access-rules` against a local test-infra
Milo, now including invalid declarations refused.
- [x] milo-os/feedback's kind end-to-end suite (milo-os/feedback#42)
against this build: 8 of 8 pass, with the feedback CRDs installed under
the stricter checks.
@mattdjenkinson mattdjenkinson mentioned this pull request Nov 18, 2026
4 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants