Repository navigation
feat: Postgres storage layer - #22
Closed
mattdjenkinson wants to merge 1 commit into
Closed
mattdjenkinson wants to merge 1 commit into
mattdjenkinson wants to merge 1 commit into
Conversation
A generic Postgres-backed rest.Storage, configured per kind with pg.Kind, and the schema for every Postgres-stored resource. - One table per kind with common columns (scope, name, uid, resource_version from a global sequence, labels, object JSON) plus kind-specific columns copied from the object for field selectors and sorting. Also post_embeddings (dimension left open until the model is chosen) and digest_events - Get, list, create, update (so patch and apply) and delete, with optimistic concurrency, no-op updates that keep the resourceVersion, delete preconditions, dry run and a status subresource store - Label selectors in SQL (Gt/Lt applied in Go), field selectors mapped to columns, keyset paging with continue tokens - Scopes: platform-scoped kinds, and user-owned kinds keyed by user UID that staff can list across users from the platform context - Transactions carried on the context; nested store calls join as savepoints, so side effects commit or roll back with the request - Integration tests against pgvector Postgres via testcontainers; they skip without Docker Closes #3.
mattdjenkinson
requested review from
0xmc,
JoseSzycho,
kevwilliams,
savme and
scotwells
November 11, 2026 10:26
This was referenced Nov 11, 2026
Closed
Contributor
Author
|
Closing: the design moved from an aggregated apiserver with Postgres to CRDs in Milo with an operator (datum-cloud/enhancements#925). The CRD version starts in #32. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3. Stacked on #16, which is stacked on #15. Merge in order.
Adds the shared Postgres storage that every resource except
FeedbackSourcewill use, and the schema for all of them. No resource is wired into the server yet; #4 onwards each add apg.Kindand install it.Schema (
migrations/00002_objects.sql)There is one table per kind:
posts,votes,subscriptions,submissions,security_reports,releasesandchangelog_entries.Common columns:
scope_typeandscope_name:Platform, orUserplus the user's UID.nameanduid.resource_version, taken from a global sequence.created_atandlabels.object, which holds the full object as JSON and is the source of truth.Kind-specific columns are copied out of the object on every write, for field selectors, sorting and joins. Posts, for example, get
state,github_org,votes_total,trending_score,delivery_stageanddelivery_changed_at, with indexes for eachPostQuerysort.Other tables:
post_embeddings: the vector dimension and approximate-nearest-neighbour index wait until the embedding model is chosen (PostQuery #5).digest_events: the queue for the daily digest, with a partial index on unsent rows.Votes outlive their post. Vote rows have no foreign key to posts, so they survive a post that disappears and comes back, as the design requires.
Unique names. Submissions and security reports have unique names across all users, because staff fetch them by name from the platform view.
Generic store (
internal/storage/pg)pg.NewStore(db, pg.Kind{...})returnsrest.Storagedriven by the same create, update and delete strategies the etcd registry uses. It supports:generateNameand creating through update or apply.resourceVersionreturns Conflict. Unconditional updates are allowed only where the strategy allows them.resourceVersionand isn't written.>and<, which are applied in Go. Field selectors map to columns throughFieldColumns; unknown fields return BadRequest.List.pg.NewStatusStoreserves get and update with the status strategy, and passes the reset fields through so server-side apply works.pg.PlatformScopedis for posts, releases and changelog entries.pg.UserOwnedis for user-owned kinds. In the user context it scopes to the caller. In the platform context it covers every user: staff can list and get, a name that exists for two users returns BadRequest, and creating returns Forbidden. Other contexts return Forbidden.db.InTx(ctx, fn)puts a transaction on the context. Store calls made with that context join it as savepoints, so aVoteand itsSubscriptioncommit or roll back together.Not supported yet: watch and deletecollection. The controller will need watch on submissions later.
Testing
go test ./...passes. The integration tests startpgvector/pgvector:pg17through testcontainers, apply the real migrations, and run 10 tests. They cover:Without Docker the tests skip. GitHub's
ubuntu-latestrunners have Docker, so CI runs them.