Skip to content

feat: index posts, releases and changelog entries in search - #41

Open
mattdjenkinson wants to merge 8 commits into
feat/crd-activityfrom
feat/crd-search
Open

mattdjenkinson wants to merge 8 commits into
feat/crd-activityfrom
feat/crd-search

Conversation

@mattdjenkinson

@mattdjenkinson mattdjenkinson commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Part of #12. Stacked on #40. The issue stays open until the index lag is measured in staging.

Ports the search index policies from the closed Postgres PR (#31). With CRDs they matter more: there's no PostQuery, so clients run a ResourceSearchQuery both for keyword search and for "similar posts" while someone writes feedback.

  • Policies: config/milo/search has a ResourceIndexPolicy for Post, Release and ChangelogEntry.
    • Searchable: title, summary or body, categories, labels and state.
    • Not indexed: derived fields (votes, trendingScore, delivery), since they'd go stale.
    • Left out: hidden posts, via the policy's condition. Hiding a post removes it from the index.
    • Never indexed: submissions, reviews, security reports, votes and follows.
  • Dependency: the indexer builds documents from audit events, so it needs datum-cloud/infra#9437, which logs feedback writes at RequestResponse. Without it, nothing is indexed.
  • hack/search-lag.sh <post>: hides a visible post, waits for search to drop it, then unhides it and times how long until search finds it again. Run it against staging (Deploy to staging #13), then close Search integration #12 with the numbers.

Tests

  • test/config/search_test.go, ported:
    • only posts, releases and changelog entries are targeted;
    • every field path resolves on the Go types;
    • no derived field is indexed;
    • the hidden-post condition, evaluated with CEL as the indexer does, indexes visible posts and skips hidden ones.
  • kustomize build of the Milo components renders 3 policies; bash -n passes on the lag script.
  • Lag measured in staging.

Also includes #42

#42 was merged into this branch, so this PR also brings in the kind end-to-end tests (task e2e:up, task e2e:test, test/e2e/). They pass 8 of 8 on this branch.

@mattdjenkinson mattdjenkinson self-assigned this Nov 17, 2026
ResourceIndexPolicies for Post, Release and ChangelogEntry make their
text searchable through Milo's search service, which clients use for
keyword search and similar posts. Hidden posts are left out, derived
fields aren't indexed, and private kinds never are. hack/search-lag.sh
measures how long a change takes to reach the index.

Copy link
Copy Markdown

Please address before merging:

  • The lag script always unhides the post on exit, even if it started hidden or the initial search check failed. Save and restore its original visibility.
  • The index condition fails when a Post has no spec key. Default spec to an empty object or safely handle its absence, and test a raw object without it.
  • Please verify the end-to-end user search path: posts are indexed at platform scope, while user-context searches are tenant-filtered. This appears likely to return no similar posts unless clients can query the correct scope. Document the required access and prove it with an integration test.

…nt search scope

search-lag.sh records the post's original spec.hidden and restores exactly
that on every exit once it has changed it; it refuses hidden or unindexed
posts and tells failed queries apart from misses. The Post CRD defaults
spec to {} because search's indexer can't evaluate spec.hidden on a post
with no spec. Posts are indexed as platform documents, so only
platform-root queries find them; the README says who can search and with
what access, and hack/search-scope.sh checks it against an environment.
Add task e2e:up, e2e:test and e2e:down, following milo-os/billing: a
test-infra kind cluster, Milo built and deployed from MILO_DIR, and
feedback installed in that Milo with the operator, feedback-uploads and
MinIO in the cluster (config/overlays/e2e).

The operator and feedback-uploads authenticate to Milo with static tokens
for their production usernames, so the admission webhooks' identity checks
and config/milo/rbac are exercised as they are. test-infra's Milo uses
RBAC, so the IAM roles are granted as equivalent ClusterRoles
(hack/e2e/iam-rbac.sh).

Chainsaw tests cover votes and owner rules, hidden posts, merge chains,
submissions and review decisions, the submission rate limit, attachments
through feedback-uploads and MinIO, moving submissions to security, and
operator readiness.
@mattdjenkinson

Copy link
Copy Markdown
Contributor Author
  • Lag script (05b8a15): it saves the post's original visibility and restores it on exit, including when the first check fails.
  • Missing spec: Post.spec defaults to {}, and the index condition handles a missing spec. There's a test with a raw object that has none.
  • User search path: you're right, and it's a real gap. Posts are indexed at platform scope, and user-, org- and project-context queries return nothing. The README now has a "Who can search posts" section that explains this and lists who can search how. hack/search-scope.sh checks it against a real environment, since search doesn't run in test-infra. For now the similar-posts lookup has to run server-side at the platform root (datum.net, the operator, and the portal's server). How datumctl and Patch should search is still an open design question; I'll follow up with the options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants