v5 data platform, Phase 1: schema, migrations, PGlite client, Notion import by philosophercode · Pull Request #33 · philosophercode/makerlab-tools

philosophercode · 2026-09-15T01:36:57Z

Phase 1 of the v5 data platform spec (§9): the schema, migrations, database client, demo seed, and the one-time Notion import with its verification. No runtime change — the app still reads Notion until Phase 2.

What's here

Schema and migrations — v5/src/lib/db/

Drizzle schema, one file per table group: categories, locations, tools, units, resources, attachments, maintenance_logs, feedback, projects + project_tools, audit_events. Vocabularies are text columns with named CHECK constraints (spec §4.1), including the New and Closed options the 2026-08-14 audit found defined but unused.
Three committed migrations: pg_trgm, the schema, and the updated_at triggers.
getDb(): Neon when DATABASE_URL is set, otherwise PGlite (in-process Postgres) with the demo seed. Lazy, memoised on globalThis, no Proxy.
The demo seed is the mock catalogue's two tools as real rows.

Import — v5/src/lib/import/ and three scripts

npm run import:notion -- --dry-run reads Notion into an in-memory PGlite and prints the report; no Neon needed to rehearse.
Pre-flight compares Notion's defined select options with the vocabularies and stops on anything unmapped, naming database, property and option.
Idempotent by notion_page_id; slugs are assigned once and kept; files are keyed by source_key so a re-run never uploads twice.
Files are downloaded and their bytes uploaded to Blob (Notion URLs expire). Dead Airtable URLs are skipped with a warning; a failed download is a warning, never a rollback.
Composed ticket descriptions are split back into columns; anything not exactly the template is kept whole.
npm run verify:import checks counts, relations and Blob bytes. The five-tool field comparison waits for Phase 2's read path.
npm run db:migrate applies migrations to Neon and is a no-op without DATABASE_URL.

Tests

npm run test:all with every env var unset: lint, typecheck, 836 unit tests (43 new for the import, 22 for the schema and client), 33 E2E. The schema tests run the real migrations on PGlite and exercise every CHECK, unique index, cascade and the trigger.

Spec

An as-built amendment records the details the spec left to implementation (the new npm scripts, DB_MIGRATIONS_DIR, attachments.source_key, the deferred user foreign keys, why the db/import modules use .ts-extension imports). npm run spec:coverage: 73 items, 0 undocumented.

Not in this PR

Installing Neon and linking Blob (Phase 0, needs the account owner).
Running the import against production (Phase 2, by hand, after a --dry-run).
Wiring db:migrate into the Vercel build (Phase 2).

🤖 Generated with Claude Code

https://claude.ai/code/session_01NP2xC6APXbSucoE4n5mtoj

…mport Drizzle schema for the v5 data platform with CHECK-constrained vocabularies, three committed migrations (pg_trgm, tables, updated_at triggers), a lazy getDb() that uses Neon with DATABASE_URL and PGlite otherwise, and a demo seed. The one-time import reads Notion, stops on any unmapped select option, upserts by notion_page_id, keeps slugs stable, copies file bytes to Blob keyed by source_key, and reports counts and warnings; --dry-run rehearses it in an in-memory PGlite. verify:import checks counts, relations and Blob bytes. No runtime change; the app still reads Notion until Phase 2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NP2xC6APXbSucoE4n5mtoj

vercel · 2026-09-15T01:36:59Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
makerlab-tools	Ready	Preview	Sep 15, 2026 3:03am UTC
makerlab-tools-v5	Ready	Preview	Sep 15, 2026 3:03am UTC

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5bca54337e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-15T01:42:06Z

+  check("resources with a tool", linkedResources, Number(resourcesWithTool.n));
+
+  console.log("\nFiles (every attachment row has bytes in Blob)");
+  const rows = await db.select({ pathname: attachments.blobPathname }).from(attachments);


Compare Blob rows with source attachments

When a Notion file download is skipped or fails, runImport creates no attachment row for it, so this query never sees the missing source file. If all successfully inserted rows still exist in Blob, verify:import prints Verified. and exits 0 even though the migration lost files. Compare the snapshot's expected attachments or source keys with the database as well, while accounting for any explicitly accepted stale URLs.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-15T01:42:06Z

+      const bytes = new Uint8Array(await response.arrayBuffer());
+      if (bytes.byteLength === 0) throw new FileCopyError(sourceUrl, "download was empty");
+      if (bytes.byteLength > MAX_FILE_BYTES) {
+        throw new FileCopyError(sourceUrl, `file is ${bytes.byteLength} bytes, over the ${MAX_FILE_BYTES} limit`);


Enforce the size cap while streaming

When a source property contains a large file or video, arrayBuffer() downloads and allocates the entire response before MAX_FILE_BYTES is checked. A sufficiently large file can therefore exhaust the import process instead of being recorded as an ordinary failed copy. Reject a trustworthy oversized Content-Length up front and enforce the limit while consuming the response stream.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-15T01:42:06Z

+      for (const [position, file] of (list ?? []).entries()) {
+        if (existingKeys.has(file.id)) {
+          fileCounts.skipped += 1;
+          continue;


Reconcile changed source attachments on reruns

When a file is replaced, removed, or reordered in Notion between import attempts, the synthetic source key remains tied only to its page/property/index. This skip consequently preserves the old Blob row at an occupied index, and attachments removed from the source are never deleted, so an otherwise idempotent rerun does not reflect the current snapshot. Reconcile each owner's stored attachments against the snapshot rather than treating every previously seen positional key as permanently complete.

Useful? React with 👍 / 👎.

Copilot

🟡 Changes recommended

Unresolved critical import-correctness findings and moderate verification, validation, and safety findings remain.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Phase 1 adds the v5 Postgres foundation and one-time Notion import while leaving runtime reads on Notion until Phase 2.

Changes:

Adds Drizzle schema, migrations, Neon/PGlite clients, demo seed, and database utilities.
Adds Notion snapshotting, preflight validation, mapping, Blob copying, and verification.
Adds fixtures and tests for database and import behavior.

File summaries

File	Summary
`v5/test/msw/handlers.ts`	Adds Notion database-schema API mocks.
`v5/test/fixtures/notion.ts`	Adds schema, entity, flag, and project fixtures.
`v5/test/db.ts`	Adds database constraint-test helpers.
`v5/src/lib/notion.ts`	Adds schema reads, snapshots, email parsing, and flag support.
`v5/src/lib/import/vocabulary-map.ts`	Maps Notion options to database vocabularies.
`v5/src/lib/import/vocabulary-map.test.ts`	Tests vocabulary mappings.
`v5/src/lib/import/ticket-description.ts`	Parses composed ticket descriptions.
`v5/src/lib/import/ticket-description.test.ts`	Tests ticket description parsing.
`v5/src/lib/import/source.ts`	Reads Notion schemas and records. moderate · 3 votes: Pass snapshot throttling through project queries and fallbacks.
`v5/src/lib/import/source.test.ts`	Tests Notion source access.
`v5/src/lib/import/run.ts`	Executes entity, relation, and attachment imports. critical · 2 votes: Refresh category/location keys after updates (lines 121 and 165). critical · 2 votes: Refresh serial keys when known units change. critical · 1 vote: Use stable file identity to handle replacements and reordered attachments. moderate · 1 vote: Warn when project/tool relations are dropped.
`v5/src/lib/import/run.test.ts`	Tests import execution, updates, duplicates, and relations.
`v5/src/lib/import/preflight.ts`	Validates Notion options before import.
`v5/src/lib/import/preflight.test.ts`	Tests preflight validation.
`v5/src/lib/import/mappers.ts`	Maps Notion records into database rows. moderate · 1 vote: Validate real calendar dates rather than regex shape alone. moderate · 1 vote: Warn on invalid `date_reported` values even when using fallback text.
`v5/src/lib/import/mappers.test.ts`	Tests record mapping.
`v5/src/lib/import/files.ts`	Downloads and prepares attachment files. moderate · 3 votes: Enforce the size limit with bounded reads before buffering responses. moderate · 2 votes: Redact signed URL query strings from error messages.
`v5/src/lib/import/files.test.ts`	Tests attachment handling.
`v5/src/lib/import/blob-uploader.ts`	Uploads imported files to Blob.
`v5/src/lib/db/types.ts`	Defines database and import types.
`v5/src/lib/db/slug.ts`	Provides slug generation.
`v5/src/lib/db/slug.test.ts`	Tests slug generation.
`v5/src/lib/db/schema/vocabulary.ts`	Defines database vocabulary values.
`v5/src/lib/db/schema/units.ts`	Defines unit schema.
`v5/src/lib/db/schema/tools.ts`	Defines tool schema.
`v5/src/lib/db/schema/taxonomy.ts`	Defines category and location schema.
`v5/src/lib/db/schema/resources.ts`	Defines resource schema.
`v5/src/lib/db/schema/projects.ts`	Defines project and project-tool schema.
`v5/src/lib/db/schema/maintenance.ts`	Defines maintenance log schema.
`v5/src/lib/db/schema/index.ts`	Exports database schemas.
`v5/src/lib/db/schema/helpers.ts`	Provides shared schema helpers.
`v5/src/lib/db/schema/feedback.ts`	Defines feedback schema.
`v5/src/lib/db/schema/audit.ts`	Defines audit event schema. moderate · 2 votes: Add a database CHECK constraint for allowed audit actions.
`v5/src/lib/db/schema/attachments.ts`	Defines attachment schema.
`v5/src/lib/db/schema.test.ts`	Tests constraints, indexes, cascades, and triggers.
`v5/src/lib/db/raw.ts`	Provides raw database helpers.
`v5/src/lib/db/pglite.ts`	Configures the PGlite client.
`v5/src/lib/db/neon.ts`	Configures the Neon client.
`v5/src/lib/db/migrations/meta/0001_snapshot.json`	Stores Drizzle schema snapshot metadata.
`v5/src/lib/db/migrations/meta/0000_snapshot.json`	Stores initial Drizzle schema snapshot metadata.
`v5/src/lib/db/migrations/meta/_journal.json`	Tracks database migrations.
`v5/src/lib/db/migrations/0002_updated_at_triggers.sql`	Adds `updated_at` triggers.
`v5/src/lib/db/migrations/0001_initial_schema.sql`	Creates the initial database schema.
`v5/src/lib/db/migrations/0000_extensions.sql`	Enables required database extensions.
`v5/src/lib/db/migrations-folder.ts`	Resolves the migrations directory.
`v5/src/lib/db/demo-seed.ts`	Seeds demo tool data.
`v5/src/lib/db/demo-seed.test.ts`	Tests demo seed behavior.
`v5/src/lib/db/client.ts`	Provides the lazy memoized database client.
`v5/src/lib/db/client.test.ts`	Tests database client behavior.
`v5/scripts/verify-import.ts`	Verifies imported counts, relations, and files. moderate · 2 votes: Compare each relation to its Notion page mapping rather than aggregate counts. moderate · 1 vote: Account for merged duplicate category and location keys during count verification.
`v5/scripts/import-notion.ts`	Provides the Notion import CLI.
`v5/scripts/db-migrate.ts`	Applies production migrations.
`v5/package.json`	Adds database and import scripts and dependencies.
`v5/drizzle.config.ts`	Configures Drizzle schema and migrations.
`v5/.env.example`	Documents database configuration. nit · 1 vote: Clarify that the app remains on Notion until Phase 2 and PGlite applies only to helpers, tests, and dry runs.
`docs/specs/2026-09-14-v5-data-platform-design.md`	Records as-built Phase 1 implementation details.

Review details

Suppressed comments (6)

v5/.env.example:94

This Phase 1 does not wire getDb() into the app: current catalog reads still use Notion or the mock fallback. The example therefore incorrectly tells operators that an unset DATABASE_URL makes the app use PGlite; limit this wording to the database helpers/tests and dry-run, and state that the app remains on Notion until Phase 2.

# With it unset the app, the tests and `npm run import:notion -- --dry-run`
# use an in-process Postgres (PGlite) with sample data, and `npm run
# db:migrate` does nothing. Set it to import for real and to run migrations.

v5/scripts/verify-import.ts:58

The importer intentionally merges duplicate categories and locations by normalized key, but verification compares raw Notion page counts. A duplicate source page—the behavior covered by run.test.ts—will therefore make this check fail even when the import is behaving as designed; compare against the number of unique import keys or account for the merge warnings.

  check("categories", snapshot.categories.length, await rowCount(db, categories));
  check("locations", snapshot.locations.length, await rowCount(db, locations));

v5/src/lib/import/mappers.ts:61

The regex checks only the shape, so values such as 2024-99-99 or 2024-02-31 are accepted and later sent to PostgreSQL date columns. That can abort the import transaction instead of producing the documented null result. Validate that the captured date round-trips as a real calendar date before returning it.
v5/src/lib/import/mappers.ts:205
A non-empty date_reported that is not parseable is silently discarded here, unlike date_resolved below, which emits a warning. Legacy rich-text values such as Sept 1 can therefore disappear without being reported; detect and warn on an invalid raw date even when the description fallback is used.
v5/src/lib/import/run.ts:166
The location update path leaves both byKey and mapTags unchanged. After a room/zone or map-tag edit, subsequent records can be merged using the old location key or have the newly freed map tag dropped as a duplicate; refresh both in-memory indexes when updating a known row.
v5/src/lib/import/run.ts:378
Unresolved project/tool relations are silently filtered here, unlike the other relation mappers that add warnings. A deleted, archived, or otherwise missing tool therefore disappears from project_tools with no indication in the import report; retain the filtering required by the foreign key, but emit a warning for each dropped Notion relation.

Files reviewed: 57/58 changed files
Comments generated: 8
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        if (known) {
+          await tx.update(categories).set({ name: row.name, group: row.group }).where(eq(categories.id, known));
+          categoryIds.set(record.id, known);
+          updated += 1;
+          continue;
+        }


+        if (known) {
+          await tx.update(units).set(patchOf(row, "createdAt")).where(eq(units.id, known));
+          unitIds.set(record.id, known);
+          updated += 1;


+        if (existingKeys.has(file.id)) {
+          fileCounts.skipped += 1;


+  console.log("\nRelations (Notion relations that point at an existing tool)");
+  const toolIds = new Set(snapshot.tools.map((t) => t.id));
+  const linkedUnits = snapshot.units.filter((u) => u.fields.tool?.some((id) => toolIds.has(id))).length;
+  const [unitsWithTool] = await db.select({ n: count() }).from(units).where(isNotNull(units.toolId));
+  check("units with a tool", linkedUnits, Number(unitsWithTool.n));


+    id: uuid("id").primaryKey().defaultRandom(),
+    at: timestamp("at", { withTimezone: true }).notNull().defaultNow(),
+    actorUserId: text("actor_user_id"),
+    action: text("action").notNull(),


+  readonly status: number | undefined;
+
+  constructor(url: string, message: string, status?: number) {
+    super(`${message} (${url})`);


+      const bytes = new Uint8Array(await response.arrayBuffer());
+      if (bytes.byteLength === 0) throw new FileCopyError(sourceUrl, "download was empty");
+      if (bytes.byteLength > MAX_FILE_BYTES) {
+        throw new FileCopyError(sourceUrl, `file is ${bytes.byteLength} bytes, over the ${MAX_FILE_BYTES} limit`);


+  const projects = hasProjectsEnv() ? await fetchAllProjects() : [];
+  log(`read ${projects.length} projects${hasProjectsEnv() ? "" : " (NOTION_DB_PROJECTS unset)"}`);


Copilot AI lite review requested due to automatic review settings September 15, 2026 01:36

Copilot started reviewing on behalf of philosophercode September 15, 2026 01:37 View session

vercel Bot deployed to Preview – makerlab-tools September 15, 2026 01:37 View deployment

vercel Bot deployed to Preview – makerlab-tools-v5 September 15, 2026 01:38 View deployment

chatgpt-codex-connector Bot reviewed Sep 15, 2026

View reviewed changes

Copilot AI reviewed Sep 15, 2026

View reviewed changes

philosophercode force-pushed the v5/data-platform-phase-1 branch from f25a8ff to 5bca543 Compare September 15, 2026 03:02

philosophercode mentioned this pull request Sep 15, 2026

v5 data platform, Phase 2: every read path on Postgres #34

Open

Navigation Menu

v5 data platform, Phase 1: schema, migrations, PGlite client, Notion import - #33

v5 data platform, Phase 1: schema, migrations, PGlite client, Notion import#33
philosophercode wants to merge 1 commit into
mainfrom
v5/data-platform-phase-1

philosophercode commented Sep 15, 2026

Uh oh!

vercel Bot commented Sep 15, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 15, 2026

Uh oh!

chatgpt-codex-connector Bot Sep 15, 2026

Uh oh!

chatgpt-codex-connector Bot Sep 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		const projects = hasProjectsEnv() ? await fetchAllProjects() : [];
		log(`read ${projects.length} projects${hasProjectsEnv() ? "" : " (NOTION_DB_PROJECTS unset)"}`);

Conversation

philosophercode commented Sep 15, 2026

What's here

Tests

Spec

Not in this PR

Uh oh!

vercel Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (6)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Sep 15, 2026 •

edited

Loading