Repository navigation
test: end-to-end tests in kind against a real Milo - #42
Merged
Merged
Conversation
Add task e2e:up, e2e:test and e2e:down, following milo-os/billing: a test-infra kind cluster, Milo built and deployed from MILO_DIR, and feedback installed in that Milo with the operator, feedback-uploads and MinIO in the cluster (config/overlays/e2e). The operator and feedback-uploads authenticate to Milo with static tokens for their production usernames, so the admission webhooks' identity checks and config/milo/rbac are exercised as they are. test-infra's Milo uses RBAC, so the IAM roles are granted as equivalent ClusterRoles (hack/e2e/iam-rbac.sh). Chainsaw tests cover votes and owner rules, hidden posts, merge chains, submissions and review decisions, the submission rate limit, attachments through feedback-uploads and MinIO, moving submissions to security, and operator readiness.
This was referenced Nov 18, 2026
scotwells
approved these changes
Nov 18, 2026
|
No blocking findings in this test PR. One small follow-up: I checked the Chainsaw schemas, shell syntax, rendered overlays and focused config/command tests; those passed. I did not rerun the full kind suite. The documented IAM and search coverage gaps still need separate integration testing. |
2 of 3 tasks
mattdjenkinson
added a commit
to milo-os/milo
that referenced
this pull request
Nov 18, 2026
## Summary Closes #1020. Follow-up to #1017: a wrong or not-yet-loaded access rule turned protection off and only logged it. Access rules now fail closed. - **Rules come from the CRD informer cache at request time**, the same cache the CRD handler serves from, so a custom resource is never served before its rule. - **Invalid declarations are refused.** A CRD's `iam.miloapis.com/` annotations are invalid when they are: - blank or unknown; - a field that isn't a plain selectable path; - an owner field under `status` or a scale path; - `hidden-subresource` without `hidden-field`. Requests for that resource answer 503 outside `system:masters`, from both the filter and `AccessRuleOwnerEnforcement`. - **CRD writes that introduce a problem are rejected (422).** This ratchets: a CRD that's already invalid can still be updated as long as the update doesn't add a new problem. Changing a rule returns a warning. - **Before the cache syncs**, every resource request outside a project answers 503 outside `system:masters`, in every context. Before, platform-context requests were served. - **Staleness check:** compares each CRD's resource version in storage with the cache. While a CRD's cached copy lags, only that CRD's resources are refused; everything else is served. Readiness fails only before the rules first sync, so a lag shared by every replica can't take them all out of rotation. - **Hidden fields:** a hidden field that can't be read hides the object. A missing one shows it, matching storage's field selector for lists. - **Alerts:** `milo_access_rule_invalid` and `milo_access_rules_stale` (the number of stale CRDs) gauges, with `MiloAccessRuleInvalid` and `MiloAccessRulesStale` alerts in `config/telemetry/alerts/access-rules`. - **Wiring:** the control plane, APIExtensions and aggregator servers share one list of Milo's admission initializers, which includes the access rules. Docs: `docs/architecture/access-rules.md`. ## Test plan - [x] `go test ./pkg/server/access/... ./internal/apiserver/admission/plugin/accessowner/... ./cmd/milo/apiserver/...`: - declaration parsing for each invalid case; - 503 for invalid rules and before sync, with the `system:masters` bypass; - ratcheting CRD validation; - a stale CRD refused while others are served and readiness passes; - the shared initializers give `AccessRuleOwnerEnforcement` its rules; - a service account in an organization context. - [x] Prometheus rule tests (promtool 3.13.2). - [x] `task test:end-to-end -- access-rules` against a local test-infra Milo, now including invalid declarations refused. - [x] milo-os/feedback's kind end-to-end suite (milo-os/feedback#42) against this build: 8 of 8 pass, with the feedback CRDs installed under the stricter checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #41.
End-to-end tests for the CRD stack, in a kind cluster against a real Milo, using datum-cloud/test-infra the way
datum-cloud/computedoes.task e2e:upcreates the test-infra kind cluster, builds and deploys Milo from a checkout (MILO_DIR, default../milo), then installs feedback in it.task e2e:testruns the Chainsaw tests intest/e2e/.e2e:redeployande2e:downare there too.caBundle) and thepost-refadmission policy.config/overlays/e2e): the operator andfeedback-uploads, a cert-manager webhook certificate, MinIO as the attachments bucket through thes3-attachmentscomponent, and a stand-in audit sink that logs what Milo sends.hack/e2e/iam-rbac.shgrants the IAM roles' permissions as ClusterRoles.votes<post>.<user>names, owner filtering, follows, vote totals, post named in delete audit eventshidden-postsmerge-chainsubmissionsrate-limitattachmentsfeedback-uploadsand MinIOsecurityoperator-readinessNot covered:
feedback-syncand GitHub, Milo IAM's context checks, activity and search (their services don't run in test-infra), notifications and digests.What it found
Writing these turned up two bugs, fixed in the PRs they belong to, each with a unit test:
feedback-uploadscrash-looped: it didn't bind the zap flags its Deployment passes (feat: attachments through a feedback-uploads service #38).cmd/feedback/flags_test.gonow parses every Deployment's args against its command.Tests
task e2e:test: 8 passed, 0 failed, on this branch merged with every PR below it. That's whatmainwill hold once the stack merges. Milo was main plus fix: fail closed when access rules are invalid or not loaded milo#1021.task testpasses.