Repository navigation
feat: activitytool for offline ActivityPolicy tests - #358
Merged
Merged
Conversation
- policy-author agent can now run activitytool (scoped Bash) and writes and runs tests instead of only suggesting them - skill resolves real release tags instead of placeholders, defines an install fallback order, and forbids copying actual output into expectations - document known parity limits, test-file rules, realistic MCP-sourced inputs, and CI pitfalls (Docker mount scope, kustomize patches) - SessionStart hook no longer recommends go install @latest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tooling-only follow-up to the #343 review. Production code is unchanged; the evaluator refactor and event-link fix come in a separate parity PR. - Evaluate through the real PolicyCache routing: policies in name order, first match wins; an explicit `policy:` must match the input's resource; report shadowing, skipped rules and guessed kinds as warnings - Reproduce production's current event-link output and its rule-skip semantics, guarded by drift tests against the live code paths - Fail on duplicate policy names, no longer drop schema errors, read every `---` document in a test file, reject exp_no_match with other assertions - Validate through the API server's ValidateActivityPolicySpec; warn on misspelled kinds; skip kustomize patch files; render kustomize root-only and offline - Add resource_kinds, exp_policy, --fail-on-untested-rules, per-rule coverage, and structured failureDetails/warnings in JSON output - Dockerfile: digest-pinned bases, distroless nonroot, -trimpath; add task test:policies:docker - Docs and test-policy skill: other repos run the Docker image only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the digest pins and the static-debian12 switch so the activitytool image uses the same base images as the root Dockerfile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ecv
approved these changes
Nov 18, 2026
ecv
left a comment
Contributor
There was a problem hiding this comment.
Approve.
The tool reuses the processor's real compile, audit-rule and event-match code, mirrors only the loops production does not expose, and guards the mirror with drift tests. check and test exit non-zero on zero policies, and the publish and hook wiring is safe.
- Routing:
pkg/policytest/evaluator.gocallsPolicyCache,EvaluateCompiledAuditRules,CompiledRule.EvaluateEventMatchandcel.EvaluateEventSummarydirectly. The event and policy loops are copies, withinternal/processor/policytest_drift_test.gorunning both sides on the same inputs. - Empty input: I built it and ran
checkon an empty dir and onresources: [], andteston the empty kustomization. All exit 1 ("no ActivityPolicy resources found"); only--allow-emptyexits 0 with a warning. - Publish:
publish-activitytool-container-imageuses the same pinneddatum-cloud/actionspublish-docker@v1.28.0 and permissions as the other images, and it triggers onpush/release only, so a fork PR never reaches it. The PR's own run published it; itsbuild-and-pushjob passed. - Hook:
claude-plugin/hooks/hooks.jsonhas one SessionStart command,check-tools.sh. It tests for policy directories,rgs forkind: ActivityPolicyand prints guidance. It runs nothing else, writes nothing and fetches nothing. - Production change:
EvaluateEventRulesand the live event path now sharenormalizeEventMap, which also normalizes an event whoseregardingis present but not a map. Before, any presentregardingkey, even nil, was left alone. That is a small live behavior change, likely the right one, and it is covered byevent_activity_parity_test.go. - Tamper test: reversing the audit policy order in the evaluator fails five
pkg/policytesttests, so routing order is guarded there. The processor drift tests did not catch it.
Follow-ups:
- Tamper test: changing the event match in
evaluator.goto evaluaterawEventinstead of the normalized event passes all ofinternal/processor,pkg/policytestandcmd/activitytool. Add a drift case whose event rule matches onevent.regarding.*for a legacyinvolvedObjectevent (the existinginvolvedObjectcase only matches on reason). - The evaluator keeps its own
normalizeEventcopy beside the now-exported-for-testnormalizeEventMap; export one function frominternal/processorso the tool cannot drift on normalization. productionEventLinksis a stated mirror with aTODO(parity-followup); file an issue so it does not outlive the live-path change it waits on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #343
What
activitytoolvalidatesActivityPolicymanifests and runs*.test.yamlunit tests offline, likepromtool test rules.activitytool check <files|dirs|kustomization>: schema, unique names, CEL compiles. Reports every problem in one run.activitytool test: runs inputs through the same routing as the processor (first matching policy wins) and asserts policy, rule, summary, links, actor and so on.-o jsonfor agents, and--fail-on-untested/--fail-on-untested-rulesfor coverage.ghcr.io/milo-os/activitytool. Other repos run their tests in this image.task test:policies(and:docker) here, wired into CI.test-policyClaude skill;author-policyandpolicy-authornow write and run tests.docs/guides/testing-policies.mdScope
Tooling only. Production code is unchanged. Where the tool and the processor differ today (event links, rule-error handling), the tool copies production, and drift tests fail if the two diverge. Aligning production comes in a follow-up PR.
Next
After this merges and is released,
datum-cloud/actionsgets a reusable workflow that runs the image against a repo's kustomization.Testing
go test ./internal/... ./pkg/... ./cmd/...passestask test:policies: 11/11 pass🤖 Generated with Claude Code