Skip to content

feat: serve Post, Release and ChangelogEntry - #23

Closed
mattdjenkinson wants to merge 1 commit into
feat/postgres-storagefrom
feat/post-release-changelog
Closed

mattdjenkinson wants to merge 1 commit into
feat/postgres-storagefrom
feat/post-release-changelog

Conversation

@mattdjenkinson

Copy link
Copy Markdown
Contributor

Closes #4. Stacked on #22, which is stacked on #16. Merge in order.

Serves posts, releases and changelogentries, each with a status subresource, from Postgres, and enforces the design's rules about who writes what.

Who writes what

Milo's IAM decides who may use each verb at all. Rules about individual fields can't be expressed there, so the server checks the caller's username against --sync-users and --controller-users. Both default to service accounts in feedback-system; #10 settles the real Milo identities.

Posts

Caller Create, delete Main resource update Status update
feedback-sync yes spec and metadata everything except controller-owned and server-owned fields
feedback-controller no spec and metadata only delivery.stage, highestNotified, history and work.partShipped
anyone else (staff) Forbidden spec.hidden and metadata Forbidden
  • Ignored fields: fields a caller doesn't own are dropped rather than rejected, the same way a status update ignores spec. This keeps server-side apply working for sync and the controller.
  • Server-computed votes: votes.datum and votes.total are recomputed on every write, from the votes table plus GitHub 👍 reactions. Matching 👍 to linked GitHub accounts and the trending score come with vote storage in Vote and Subscription #7.
  • On create: controller-owned delivery fields are dropped; the controller fills them later.
  • Name check: the post name must equal <org>.<repo>.<number> from status.github, lower-cased. v1alpha1.PostName builds it.

Hidden posts. They are NotFound outside the platform context. Platform callers (staff, the website's service account, the service's own components) still see them, and can filter with spec.hidden=false.

Releases and changelog entries. Only feedback-sync writes them, through either endpoint.

Derived and side-effect fields

  • Derived on read: Release.status.posts is the visible posts whose status.release names the release. ChangelogEntry.status.posts is the visible posts among status.issueRefs. Whatever sync writes in those fields is dropped, so they can't drift from the posts table.
  • Embeddings: a title or body change deletes the post's embedding, and the embedder in PostQuery #5 regenerates missing ones.
  • Vote retention: deleting a post records it in deleted_posts (migration 00003). Its votes and subscriptions are kept for --vote-retention (default 30 days) and then purged by an hourly loop in the apiserver. If the post is created again first, it keeps them.
    • The purge writes to Postgres directly rather than through the API, like the trending refresh will. It only removes rows for posts that no longer exist.

Store changes

pg.Kind gains four hooks:

  • Before and after write, both run inside the request's transaction.
  • A read filter, an extra SQL condition applied to reads, updates and deletes.
  • Decorate, which fills derived fields on returned objects.

Integration tests now share setup through a pgtest package.

Testing

go test ./... passes.

  • Post integration tests (8), against Postgres:
    • only sync creates and deletes;
    • vote counts are computed, and recomputed when 👍 changes;
    • staff can hide a post, and their status writes are refused;
    • sync and the controller each keep only their own fields across alternating writes;
    • hidden posts are invisible to users and filterable for staff;
    • field selectors work;
    • a title change drops the embedding but a state change doesn't;
    • vote retention keeps one post's vote, purges the expired one, and the kept post returns with its vote.
  • Unit tests for the per-caller merge and validation.
  • Release and changelog tests: derived posts with hidden ones excluded, sync-written posts dropped, writes from anyone else refused, invalid dates and refs rejected.
  • kind cluster, with the sync service account impersonated through RBAC standing in for Milo IAM:
    • the APIService was Available with all four resources served;
    • staff create was refused, and sync created a post, a release and a changelog entry;
    • the kubectl table columns work, and votes were recomputed;
    • a staff patch set hidden but its status edit was ignored, and a staff status patch was refused;
    • the release stopped listing the post once it was hidden;
    • server-side apply of status worked for sync;
    • deleting and recreating the post cleared its retention row.

Postgres-backed storage for the three GitHub-synced kinds, with the field
ownership rules from the design.

- Posts: only feedback-sync creates and deletes. Status is merged per
  caller: sync owns GitHub-derived fields and delivery work items, the
  controller owns stage, history, highestNotified and part-shipped, the
  server computes vote counts. Staff can only change spec.hidden; their
  status writes are Forbidden
- Hidden posts are NotFound outside the platform context
- Release.status.posts and ChangelogEntry.status.posts are derived from
  the posts table on read, excluding hidden posts
- A title or body change drops the post's embedding for re-embedding
- Votes on deleted posts are kept for --vote-retention (30 days), then
  purged hourly; a returning post keeps its votes
- Store hooks: before/after write in the request transaction, a read
  filter, and decoration of returned objects
- --sync-users and --controller-users flags; pgtest package shared by
  integration tests

Closes #4.
@mattdjenkinson

Copy link
Copy Markdown
Contributor Author

Closing: the design moved from an aggregated apiserver with Postgres to CRDs in Milo with an operator (datum-cloud/enhancements#925). The CRD version starts in #32.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant