Skip to content

feat: Postgres storage layer - #22

Closed
mattdjenkinson wants to merge 1 commit into
feat/api-typesfrom
feat/postgres-storage
Closed

mattdjenkinson wants to merge 1 commit into
feat/api-typesfrom
feat/postgres-storage

Conversation

@mattdjenkinson

Copy link
Copy Markdown
Contributor

Closes #3. Stacked on #16, which is stacked on #15. Merge in order.

Adds the shared Postgres storage that every resource except FeedbackSource will use, and the schema for all of them. No resource is wired into the server yet; #4 onwards each add a pg.Kind and install it.

Schema (migrations/00002_objects.sql)

There is one table per kind: posts, votes, subscriptions, submissions, security_reports, releases and changelog_entries.

Common columns:

  • scope_type and scope_name: Platform, or User plus the user's UID.
  • name and uid.
  • resource_version, taken from a global sequence.
  • created_at and labels.
  • object, which holds the full object as JSON and is the source of truth.

Kind-specific columns are copied out of the object on every write, for field selectors, sorting and joins. Posts, for example, get state, github_org, votes_total, trending_score, delivery_stage and delivery_changed_at, with indexes for each PostQuery sort.

Other tables:

  • post_embeddings: the vector dimension and approximate-nearest-neighbour index wait until the embedding model is chosen (PostQuery #5).
  • digest_events: the queue for the daily digest, with a partial index on unsent rows.

Votes outlive their post. Vote rows have no foreign key to posts, so they survive a post that disappears and comes back, as the design requires.

Unique names. Submissions and security reports have unique names across all users, because staff fetch them by name from the platform view.

Generic store (internal/storage/pg)

pg.NewStore(db, pg.Kind{...}) returns rest.Storage driven by the same create, update and delete strategies the etcd registry uses. It supports:

  • Get, list, create, update, delete. Update also covers patch and server-side apply.
    • System fields are filled the way the generic registry fills them, including generateName and creating through update or apply.
    • Optimistic concurrency: a stale resourceVersion returns Conflict. Unconditional updates are allowed only where the strategy allows them.
    • An update that changes nothing keeps the resourceVersion and isn't written.
    • Delete honours UID and resourceVersion preconditions. It's immediate, with no finalizers.
    • Dry run works for create, update and delete.
  • Selectors. Label selectors run in SQL, except > and <, which are applied in Go. Field selectors map to columns through FieldColumns; unknown fields return BadRequest.
  • Paging. Results are ordered by name and paged with a continue token. Pages aren't a consistent snapshot, which is documented on List.
  • Status subresource. pg.NewStatusStore serves get and update with the status strategy, and passes the reset fields through so server-side apply works.
  • Scopes.
    • pg.PlatformScoped is for posts, releases and changelog entries.
    • pg.UserOwned is for user-owned kinds. In the user context it scopes to the caller. In the platform context it covers every user: staff can list and get, a name that exists for two users returns BadRequest, and creating returns Forbidden. Other contexts return Forbidden.
    • Field-level visibility is Identity, contexts and authorization #10.
  • Transactions. db.InTx(ctx, fn) puts a transaction on the context. Store calls made with that context join it as savepoints, so a Vote and its Subscription commit or roll back together.

Not supported yet: watch and deletecollection. The controller will need watch on submissions later.

Testing

go test ./... passes. The integration tests start pgvector/pgvector:pg17 through testcontainers, apply the real migrations, and run 10 tests. They cover:

  • CRUD, conflicts and no-op updates;
  • create-via-update and generated names;
  • delete preconditions;
  • the status store keeping spec unchanged;
  • 11 label and field selector cases;
  • paging across three pages, and a bad continue token;
  • isolation between two users and the platform view;
  • commit and rollback across two stores, plus a failed nested write that rolls back only its savepoint;
  • dry run.

Without Docker the tests skip. GitHub's ubuntu-latest runners have Docker, so CI runs them.

A generic Postgres-backed rest.Storage, configured per kind with pg.Kind,
and the schema for every Postgres-stored resource.

- One table per kind with common columns (scope, name, uid,
  resource_version from a global sequence, labels, object JSON) plus
  kind-specific columns copied from the object for field selectors and
  sorting. Also post_embeddings (dimension left open until the model is
  chosen) and digest_events
- Get, list, create, update (so patch and apply) and delete, with
  optimistic concurrency, no-op updates that keep the resourceVersion,
  delete preconditions, dry run and a status subresource store
- Label selectors in SQL (Gt/Lt applied in Go), field selectors mapped to
  columns, keyset paging with continue tokens
- Scopes: platform-scoped kinds, and user-owned kinds keyed by user UID
  that staff can list across users from the platform context
- Transactions carried on the context; nested store calls join as
  savepoints, so side effects commit or roll back with the request
- Integration tests against pgvector Postgres via testcontainers; they
  skip without Docker

Closes #3.
@mattdjenkinson

Copy link
Copy Markdown
Contributor Author

Closing: the design moved from an aggregated apiserver with Postgres to CRDs in Milo with an operator (datum-cloud/enhancements#925). The CRD version starts in #32.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant