Skip to content

feat: activitytool for offline ActivityPolicy tests - #358

Merged
JoseSzycho merged 5 commits into
mainfrom
343-test-tooling-for-activitypolicy
Nov 18, 2026
Merged

JoseSzycho merged 5 commits into
mainfrom
343-test-tooling-for-activitypolicy

Conversation

@JoseSzycho

Copy link
Copy Markdown
Contributor

Closes #343

What

activitytool validates ActivityPolicy manifests and runs *.test.yaml unit tests offline, like promtool test rules.

  • activitytool check <files|dirs|kustomization>: schema, unique names, CEL compiles. Reports every problem in one run.
  • activitytool test: runs inputs through the same routing as the processor (first matching policy wins) and asserts policy, rule, summary, links, actor and so on.
  • -o json for agents, and --fail-on-untested / --fail-on-untested-rules for coverage.
  • Published as ghcr.io/milo-os/activitytool. Other repos run their tests in this image.
  • task test:policies (and :docker) here, wired into CI.
  • test-policy Claude skill; author-policy and policy-author now write and run tests.
  • Guide: docs/guides/testing-policies.md

Scope

Tooling only. Production code is unchanged. Where the tool and the processor differ today (event links, rule-error handling), the tool copies production, and drift tests fail if the two diverge. Aligning production comes in a follow-up PR.

Next

After this merges and is released, datum-cloud/actions gets a reusable workflow that runs the image against a repo's kustomization.

Testing

  • go test ./internal/... ./pkg/... ./cmd/... passes
  • task test:policies: 11/11 pass

🤖 Generated with Claude Code

JoseSzycho and others added 3 commits November 18, 2026 11:26
- policy-author agent can now run activitytool (scoped Bash) and writes
  and runs tests instead of only suggesting them
- skill resolves real release tags instead of placeholders, defines an
  install fallback order, and forbids copying actual output into
  expectations
- document known parity limits, test-file rules, realistic MCP-sourced
  inputs, and CI pitfalls (Docker mount scope, kustomize patches)
- SessionStart hook no longer recommends go install @latest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tooling-only follow-up to the #343 review. Production code is unchanged;
the evaluator refactor and event-link fix come in a separate parity PR.

- Evaluate through the real PolicyCache routing: policies in name order,
  first match wins; an explicit `policy:` must match the input's resource;
  report shadowing, skipped rules and guessed kinds as warnings
- Reproduce production's current event-link output and its rule-skip
  semantics, guarded by drift tests against the live code paths
- Fail on duplicate policy names, no longer drop schema errors, read every
  `---` document in a test file, reject exp_no_match with other assertions
- Validate through the API server's ValidateActivityPolicySpec; warn on
  misspelled kinds; skip kustomize patch files; render kustomize root-only
  and offline
- Add resource_kinds, exp_policy, --fail-on-untested-rules, per-rule
  coverage, and structured failureDetails/warnings in JSON output
- Dockerfile: digest-pinned bases, distroless nonroot, -trimpath; add
  task test:policies:docker
- Docs and test-policy skill: other repos run the Docker image only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JoseSzycho
JoseSzycho requested a review from a team as a code owner November 18, 2026 15:33
@JoseSzycho JoseSzycho linked an issue Nov 18, 2026 that may be closed by this pull request
JoseSzycho and others added 2 commits November 18, 2026 12:36
Drop the digest pins and the static-debian12 switch so the activitytool
image uses the same base images as the root Dockerfile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

The tool reuses the processor's real compile, audit-rule and event-match code, mirrors only the loops production does not expose, and guards the mirror with drift tests. check and test exit non-zero on zero policies, and the publish and hook wiring is safe.

  • Routing: pkg/policytest/evaluator.go calls PolicyCache, EvaluateCompiledAuditRules, CompiledRule.EvaluateEventMatch and cel.EvaluateEventSummary directly. The event and policy loops are copies, with internal/processor/policytest_drift_test.go running both sides on the same inputs.
  • Empty input: I built it and ran check on an empty dir and on resources: [], and test on the empty kustomization. All exit 1 ("no ActivityPolicy resources found"); only --allow-empty exits 0 with a warning.
  • Publish: publish-activitytool-container-image uses the same pinned datum-cloud/actions publish-docker@v1.28.0 and permissions as the other images, and it triggers on push/release only, so a fork PR never reaches it. The PR's own run published it; its build-and-push job passed.
  • Hook: claude-plugin/hooks/hooks.json has one SessionStart command, check-tools.sh. It tests for policy directories, rgs for kind: ActivityPolicy and prints guidance. It runs nothing else, writes nothing and fetches nothing.
  • Production change: EvaluateEventRules and the live event path now share normalizeEventMap, which also normalizes an event whose regarding is present but not a map. Before, any present regarding key, even nil, was left alone. That is a small live behavior change, likely the right one, and it is covered by event_activity_parity_test.go.
  • Tamper test: reversing the audit policy order in the evaluator fails five pkg/policytest tests, so routing order is guarded there. The processor drift tests did not catch it.

Follow-ups:

  • Tamper test: changing the event match in evaluator.go to evaluate rawEvent instead of the normalized event passes all of internal/processor, pkg/policytest and cmd/activitytool. Add a drift case whose event rule matches on event.regarding.* for a legacy involvedObject event (the existing involvedObject case only matches on reason).
  • The evaluator keeps its own normalizeEvent copy beside the now-exported-for-test normalizeEventMap; export one function from internal/processor so the tool cannot drift on normalization.
  • productionEventLinks is a stated mirror with a TODO(parity-followup); file an issue so it does not outlive the live-path change it waits on.

@JoseSzycho
JoseSzycho merged commit afebe87 into main Nov 18, 2026
13 checks passed
@JoseSzycho
JoseSzycho deleted the 343-test-tooling-for-activitypolicy branch November 18, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test tooling for ActivityPolicy

2 participants