Skip to content

test: Test ActivityPolicies with activitytool - #1022

Open
JoseSzycho wants to merge 1 commit into
mainfrom
test/activitytool-policy-tests
Open

JoseSzycho wants to merge 1 commit into
mainfrom
test/activitytool-policy-tests

Conversation

@JoseSzycho

Copy link
Copy Markdown
Contributor

Adds activitytool unit tests for every Milo ActivityPolicy (milo-os/activity#343), and fixes the two policy bugs they found.

Changes

  • test/activity-policies/: 184 cases covering all 109 rules in the 24 policies. Every rule is reached, with the exact summary. Reads, controller writes and status updates produce no activity.
  • Fix: create summaries. For groups, roles, service accounts, organizations and projects, has(audit.responseObject.metadata.name) errored when the response had no metadata, so the event was dead-lettered. Each level is now checked.
  • Fix: UserInvitation. The accepted-with-org, declined-with-org and delete-with-org rules errored when the response had no status.organization, so the accepted, declined and delete fallbacks could never run.
  • task test-activity-policies: runs the suite in ghcr.io/milo-os/activitytool:v0.10.0, offline and read-only, and fails on any untested rule.
  • CI: .github/workflows/test-activity-policies.yaml calls datum-cloud/actions/…/test-activity-policies.yaml@v1.29.0 when policies or tests change.

Not in this PR

  • Most write rules don't check for a 2xx response or skip dry runs (audit.requestURI.contains('dryRun=')), so rejected and dry-run changes still reach the timeline.
  • config/services/identity/policies/serviceaccount-policy.yaml is an unreferenced copy of the IAM service account policy and could be deleted.

Testing

  • task test-activity-policies: 184/184 passed, 109/109 rules matched.
  • go test ./config/services/activity/policies/ (owner labels) passes.

🤖 Generated with Claude Code

Add activitytool unit tests for every Milo ActivityPolicy and fix the two
bugs they found.

- test/activity-policies: 184 cases covering all 109 rules in the 24
  policies
- Fix create summaries for groups, roles, service accounts,
  organizations and projects, which errored (and dead-lettered the
  event) when the response had no metadata
- Fix the UserInvitation accepted, declined and delete rules: the
  -with-org rules errored when the response had no
  status.organization, so their fallbacks were unreachable
- task test-activity-policies runs the suite in the activitytool image;
  CI runs it through datum-cloud/actions' reusable workflow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JoseSzycho
JoseSzycho requested a review from a team as a code owner November 18, 2026 21:00

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

The six create-summary guards and the three UserInvitation guards are valid CEL and only add conditions that were already true for events that worked before, so summaries and matching for those events don't change.

  • I ran the pinned activitytool image against the head: 184/184 pass, 24 policies, 109/109 rules matched. Reverting the metadata guard fails 5 tests (group, role, serviceaccount, organization, project, "create without a response body"); reverting the UserInvitation guards fails 3 ("accepted/declined/cancel without the organization"). The tests do guard both fixes.
  • The workflow pins activitytool by tag and digest (v0.10.0@sha256:8846...) with a renovate marker, and the Taskfile carries the same pin. The shared workflow defaults allow-empty to false and fail-on-untested: true is set, so empty input or an untested policy fails.
  • Events that errored before (no metadata, or no status.organization) now take the fallback rule instead of being dead-lettered; that is the intended behavior change, and it is customer-visible only as a summary appearing where there was none.

Follow-ups:

  • The shared workflow is referenced by tag @v1.29.0, not a commit SHA.
  • The Taskfile pin and the workflow pin must be kept in sync by hand; one Renovate rule covering both would remove that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants