diff --git a/docs/extending.md b/docs/extending.md index b953b76..58e9e5e 100644 --- a/docs/extending.md +++ b/docs/extending.md @@ -204,7 +204,8 @@ Each `${value}` renders as the rest of the builder renders it: | a column, expression, or another template | its SQL | | a `param.*` | a placeholder: bound on Postgres, a literal on ClickHouse | | a builder query | `(subquery)`, compiled with the outer query, its tenant scope counted | -| a string, number, boolean, `Date`, `DateTime.Utc`, `null` | the dialect's escaped literal | +| a string | bound on Postgres (`$n`), the escaped literal on ClickHouse | +| a number, boolean, `Date`, `DateTime.Utc`, `null` | the dialect's escaped literal | | `CH.sql.ident(name)` | the name quoted by the dialect; plain names only, dotted for `schema.table` | | `CH.sql.raw(text)` | the text as-is — never from input | | `CH.sql.join(values, separator?)` | each value rendered, joined by `", "` or `separator`; not parenthesized, so it fits `IN (${…})`; an empty list fails the compile | diff --git a/docs/migrations.md b/docs/migrations.md index 0a32052..1066837 100644 --- a/docs/migrations.md +++ b/docs/migrations.md @@ -293,7 +293,9 @@ are not modeled yet; write them in a `--custom` migration, and declare a view yo (`ALTER COLUMN`), and a primary key, an index or a foreign key by dropping and re-creating it, so `generate` reports nothing as unsupported. A type change is labeled `rewrite`: Postgres rewrites the table under an exclusive lock. Drops still need confirmation, and renames still -read as a drop and an add. Generated files carry `"dialect": "postgres"`. +read as a drop and an add. Generated files carry `"dialect": "postgres"`. With `emit: "sql"` in +the config, `generate` writes the rendered statements to `migration.sql` instead, each under a +comment naming its label, so a tool that applies drizzle-kit folders applies the migration too. **Applying.** Each migration runs in one transaction with its ledger row, under a transaction-scoped advisory lock, so concurrent deploys wait for each other rather than diff --git a/docs/postgres.md b/docs/postgres.md index 97422d0..ecfeef8 100644 --- a/docs/postgres.md +++ b/docs/postgres.md @@ -36,8 +36,9 @@ const query = PG.from(Requests) .orderBy(["count", "desc"]) export const compiled = PG.compileUnsafe(query, { orgId: "org_1", since: new Date("2026-01-01T00:00:00Z") }) -// compiled.sql: ... WHERE "requests"."OrgId" = $1 AND "requests"."At" >= $2 ... -// compiled.parameters: ["org_1", "2026-01-01T00:00:00.000Z"] +// compiled.sql: ... FILTER (WHERE "requests"."DurationMs" >= $1) ... +// ... WHERE "requests"."OrgId" = $2 AND "requests"."At" >= $3 ... +// compiled.parameters: [500, "org_1", "2026-01-01T00:00:00.000Z"] const db = new PGlite() // The CREATE TABLE comes from the definition itself; migrations.md shows the managed way. @@ -67,6 +68,7 @@ insert may leave it out: `PG.InsertRowOf` makes it optional. Se | Identifiers | Bare: `events.OrgId` | Quoted: `"events"."OrgId"` | | String literals | Backslash escapes: `'it\'s'` | Doubled quotes: `'it''s'` | | Params | Written in as literals; `parameters` empty | Bound as `$1`, `$2`, … in `parameters` | +| Values compared with a column, `LIKE` patterns | Written in as literals | Bound too, one param per distinct value | | `param.bool` | `1` / `0` | `true` / `false` | | `param.dateTime` | `'2026-01-01 00:00:00'` (UTC, zoneless) | `'2026-01-01T00:00:00.000Z'` | | `GROUP BY` keys | Select aliases | Select-list positions (`GROUP BY 1`) | @@ -77,6 +79,11 @@ Postgres reads a bare name in `GROUP BY` as an input column before a select alia `select({ Service: lower($.Service) }).groupBy("Service")` would group by the raw column. Writing the position instead keeps ClickHouse's meaning. +A value compared with a column (`$.email.eq(email)`, `in_`, `between`, a `LIKE` pattern) is +bound like a param, so the statement text carries no values: it stays out of logs and traces, and +one query shape is one statement. Two exceptions stay literals: an `onConflict*` `targetWhere`, +which Postgres matches against a partial index's predicate as written, and DDL. + Every string that reaches the SQL as a literal is escaped for Postgres. A value that spells the param marker `__PARAM_` is written as an `E'…'` string with the marker hex-escaped, and a literal that still contained it would fail the compile with `InvalidLiteral`. @@ -89,6 +96,7 @@ literal that still contained it would fail the compile with `InvalidLiteral`. | `bool` | `boolean` | `boolean` | boolean | | `int2`, `int4`, `int8`, `float4`, `float8`, `numeric` | same | `number` | number, numeric string, `bigint` | | `timestamptz` | `timestamptz` | `DateTime.Utc` | `Date`, or text such as `2026-01-01 00:00:00+00` | +| `timestamptzMillis` | `timestamptz` | epoch milliseconds (`number`) | the same | | `jsonb(schema?)` | `jsonb` | the schema's type (`unknown` by default) | a parsed value | | `array(type)` | `type[]` | `ReadonlyArray` | array | | `nullable(type)` | the same type | `T \| null` | the same, or `null` | @@ -103,6 +111,10 @@ both as a `bigint`. A `timestamptz` compared against a `Date`, a `DateTime.Utc` or a string is written as an ISO-8601 instant, which no session time zone can reinterpret; a zoneless string is read as UTC. +Comparisons with a literal-union column take only its members: with `status` typed +`"open" | "closed"`, `$.status.eq("opne")` is a type error. A param of the primitive +(`param.string`) still compares, for a value known only at run time. + ## Functions | Function | SQL | Notes | @@ -118,6 +130,14 @@ which no session time zone can reinterpret; a zoneless string is read as UTC. | `lower`, `upper`, `length` | same | | | `coalesce(x, fallback)` | `coalesce(x, fallback)` | No longer nullable | | `jsonText(x, key)` | `(x ->> key)` | `null` when absent | +| `greatest(a, ...)`, `least(a, ...)` | same | NULL arguments are skipped | +| `caseWhen([[c, v], ...], otherwise)` | `CASE WHEN c THEN v ... ELSE otherwise END` | | +| `asBoolean(c)` | `(c)` | A condition as a value, to select or `set` | +| `typedValue(type, value)` | a bound param | Encoded as `type` writes it: `typedValue(T.columns.at, ms)` | + +`undecoded($.column)` reads a column without its codec: a jsonb document as `unknown`, a branded +id as its string. Use it where the reader decodes stored values itself (to tolerate an older +document shape, or to name the bad row in its own error); every other read stays typed. The shared operators (`eq`, `in_`, `like`, `ilike`, `and`, `or`, `not`, arithmetic, `lit`) work unchanged. `/postgres` exports only functions Postgres has, plus `nullIf`, which renders the diff --git a/docs/queries.md b/docs/queries.md index 9cb4bca..802db79 100644 --- a/docs/queries.md +++ b/docs/queries.md @@ -17,7 +17,15 @@ _(Backed by `docs/queries.md > Queries are immutable`.)_ ## `select` -Two forms. +Three forms. + +**Every column** — `select()` with no arguments reads every column of the FROM table, under +its key, as drizzle's bare `select()` does: + +```ts +CH.from(Events).select() +// SELECT Name AS Name, DurationMs AS DurationMs, ... FROM events +``` **By column name** — output keys match the column names: @@ -36,7 +44,9 @@ CH.from(Events).select(($) => ({ })) ``` -The object keys become the SQL aliases _and_ the keys of the output row type. `select` is +The object keys become the SQL aliases _and_ the keys of the output row type. Spreading `$` +gives every column of the FROM table under its key, so `select(($) => ({ ...$, lowered: +CH.lower($.Name) }))` reads them all plus one more; `returning` takes the same. `select` is required: compiling without one raises `QueryBuilderDefect`, which stays a defect — no request value can remove a `select()`. @@ -116,6 +126,13 @@ Takes `[column, direction]` tuples, one per sort key: // ORDER BY count DESC, name ASC ``` +To sort by something not selected, pass a callback returning `[expression, direction]` pairs; +it reads the source's columns as `where` does: + +```ts +.orderBy(($) => [[$.Timestamp, "desc"], [CH.lower($.Name), "asc"]]) +``` + Passing two bare strings, `.orderBy("count", "desc")`, is a type error. Untyped callers receive `QueryBuilderDefect`; use a tuple for each sort key. diff --git a/docs/reference.md b/docs/reference.md index a3c6ebb..3fef497 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -110,6 +110,7 @@ time; see [Params and compilation](./params-and-compilation.md#what-each-kind-ac | `when(value, fn)` | `Condition \| undefined`; skips `undefined`/`null`/`false` | | `whenTrue(flag, fn)` | Boolean-gated variant | | `inList(expr, values)` | `expr IN ('a', 'b')` | +| `undecoded(column)` | the column as the driver sends it, typed as its wire form | | `inExprList(expr, exprs)` | Same for expression lists | | `notInList(expr, values)` | `expr NOT IN ('a', 'b')` | | `not(condition)` | `NOT (…)` | @@ -450,7 +451,7 @@ const Users = PG.table("users", { ### Column types `text`, `uuid`, `bool`, `int2`, `int4`, `int8`, `float4`, `float8`, `numeric`, `timestamptz`, -`jsonb(schema?)`, `array(type)`, `nullable(type)`, `custom(sql, schema, literalSchema?)`, +`timestamptzMillis` (a timestamptz read as epoch milliseconds), `jsonb(schema?)`, `array(type)`, `nullable(type)`, `custom(sql, schema, literalSchema?)`, `brand(type, schema)`. See [Postgres column types](./postgres.md#column-types). Types: `PgType` (a Postgres column type; a `CHType`), `PgArray`, `PgNullable`. Codecs: @@ -463,7 +464,9 @@ which normalizes Postgres timestamp text to ISO-8601. `count()`, `countDistinct(x)`, `countIf(c)`, `sum(x)`, `sumIf(x, c)`, `avg(x)`, `min(x)`, `max(x)`, `percentileCont(f, x)`, `arrayAgg(x)`, `dateTrunc(unit, ts)` (`DateTruncUnit` is the unit union), `dateBin(seconds, ts)`, `now()`, `lower(x)`, `upper(x)`, `length(x)`, -`coalesce(x, fallback)`, `nullIf(x, value)`, `jsonText(x, key)`. See +`coalesce(x, fallback)`, `nullIf(x, value)`, `jsonText(x, key)`, `greatest(a, ...)`, `least(a, ...)`, +`caseWhen([[c, v], ...], otherwise)`, `asBoolean(c)` (a condition as a value) and +`typedValue(type, value)` (a value bound as a column type writes it). See [Postgres functions](./postgres.md#functions) for the SQL each writes. ### Dialect diff --git a/docs/tables-and-types.md b/docs/tables-and-types.md index 6f1260e..9249e11 100644 --- a/docs/tables-and-types.md +++ b/docs/tables-and-types.md @@ -56,6 +56,10 @@ const Users = PG.table("users", { }) ``` +A column's key is its name in SQL unless `PG.column` gives it another: `orgId: PG.column(PG.text, +{ name: "org_id" })` is stored as `org_id` and read, written, filtered and indexed as `orgId` +everywhere else, as drizzle's `text("org_id")` is. Rows decode under the key. + A Postgres column is `NOT NULL` unless its type is `PG.nullable(...)`. See [Postgres](./postgres.md) for its types and [migrations](./migrations.md#postgres) for its indexes and foreign keys. diff --git a/package.json b/package.json index 412a67f..c46dce4 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@maple-dev/effect-orm", - "version": "0.4.0", + "version": "0.5.0", "description": "Type-safe ClickHouse and Postgres queries, result decoding, and reproducible benchmarks for Effect and TypeScript", "keywords": [ "clickhouse", @@ -116,4 +116,4 @@ "patchedDependencies": { "@effect/vitest@4.0.0": "patches/@effect%2Fvitest@4.0.0.patch" } -} +} \ No newline at end of file diff --git a/scripts/check-doc-examples.mjs b/scripts/check-doc-examples.mjs index 06e588e..f9ba68e 100644 --- a/scripts/check-doc-examples.mjs +++ b/scripts/check-doc-examples.mjs @@ -103,8 +103,8 @@ const fragments = await import("@maple-dev/effect-orm/sql") assert.equal(escapedSql.predicate, "Name = " + fragments.compile(fragments.str("O'Reilly"))) assert.doesNotMatch(escapedSql.predicate, /\\[object Object\\]/) const postgres = await import("./postgres-quickstart") -assert.deepEqual(postgres.compiled.parameters, ["org_1", "2026-01-01T00:00:00.000Z"]) -assert.ok(sql(postgres.compiled).includes('"requests"."OrgId" = $1 AND "requests"."At" >= $2')) +assert.deepEqual(postgres.compiled.parameters, [500, "org_1", "2026-01-01T00:00:00.000Z"]) +assert.ok(sql(postgres.compiled).includes('"requests"."OrgId" = $2 AND "requests"."At" >= $3')) assert.deepEqual(postgres.rows, [ { route: "/checkout", count: 2, slow: 1, p50: 510 }, { route: "/search", count: 1, slow: 0, p50: 40 }, diff --git a/src/ch/brand.test-d.ts b/src/ch/brand.test-d.ts index 4e63c98..4bfcdd0 100644 --- a/src/ch/brand.test-d.ts +++ b/src/ch/brand.test-d.ts @@ -74,8 +74,12 @@ CH.from(Dashboards).select("id").where(($) => [$.budget.gt(100)]) // @ts-expect-error a plain-string column is not an OrgId column CH.from(Dashboards).innerJoin(Plain, "p", (main, joined) => main.org_id.eq(joined.org_id)).select("id") -// A literal union stays a comparison on its primitive; the server checks the value. +// A literal union compares against its own members, or a param of its primitive checked by the server. CH.from(Dashboards).select("id").where(($) => [$.status.eq("open"), $.status.eq(CH.param.string("s"))]) +// @ts-expect-error not a member of the union +CH.from(Dashboards).select("id").where(($) => [$.status.eq("opne")]) +// @ts-expect-error nor in a list +CH.from(Dashboards).select("id").where(($) => [$.status.in_("open", "opne")]) // String operators still work on a branded string. CH.from(Dashboards).select("id").where(($) => [$.org_id.like("org_%")]) diff --git a/src/ch/compile.ts b/src/ch/compile.ts index abbb673..b917a0e 100644 --- a/src/ch/compile.ts +++ b/src/ch/compile.ts @@ -8,7 +8,7 @@ // 4. Assembling into SqlQuery and calling the existing compileQuery() import { compiled, compiledUnsafe, fail } from "./failure" -import { custom, dateTime, dateTime64, type CHType, type ColumnDefs } from "./types" +import { columnSqlName, custom, dateTime, dateTime64, type CHType, type ColumnDefs } from "./types" import type { CHQuery, CHQueryState, NeedsSelect } from "./query" import type { CHUnionQuery } from "./union" import { isInsert, type CHInsert } from "./insert" @@ -18,6 +18,7 @@ import { createColumnAccessor, createQualifiedColumnAccessor, createJoinedColumn import { aliased, columnTypeOf, isExprLike, type Condition, type Expr, type ParamsSatisfied } from "./expr" import { raw, identPath, quoteIdent, quoteIdentPath, compile as compileSqlFragment, type SqlFragment } from "../sql/sql-fragment" import { splitTerminalClauses } from "../sql/terminal-clauses" +import { activeLiteralBinder, withLiteralBinder } from "../sql/literal-binder" import { compileQuery, type SqlQuery } from "../sql/sql-query" import { track, untracked, type RenderTrack } from "../sql/render-tracker" import { PARAM_MARKER_PREFIX, PARAM_PLACEHOLDER_PATTERN, param, paramSchema, type ParamKind } from "./param" @@ -98,6 +99,15 @@ const orderByClause = (specs: ReadonlyArray<[string, "asc" | "desc"]>): Array, "asc" | "desc"]>): Array => + specs.map(([expr, direction]) => { + if (!isExprLike(expr) || (direction !== "asc" && direction !== "desc")) { + return fail(new QueryBuilderDefect({ message: "CHQuery: orderBy(($) => ...) returns [expression, \"asc\" | \"desc\"] pairs" }), "") + } + return `${compileSqlFragment(expr.toFragment())} ${direction.toUpperCase()}` + }) + /** * A `limit` / `offset` value. It often comes from a request (a page size), so a * bad one is a failure, not a defect: `LIMIT -1` or `LIMIT NaN` is not SQL, and @@ -649,13 +659,14 @@ export function compileCHRaw( params?: Record, options?: any, ): CompiledQuery { - return withDialect(options?.dialect ?? currentDialect(), () => + const dialect = options?.dialect ?? currentDialect() + return withDialect(dialect, () => withBoundLiterals(dialect, () => isInsert(query) ? compileInsert(query, params ?? {}) : isUpdate(query) || isDelete(query) ? compileUpdateOrDelete(query, params ?? {}) : compileInner(query as CHQuery, params ?? {}, options), - ) + )) } /** @@ -921,7 +932,9 @@ function compileInner< // rows are already aggregated, so the scan that produced them crossed // tenants no matter what this filters out. having: having.map(({ sql }) => raw(sql)), - orderBy: orderByClause(state.orderBySpecs).map(raw), + orderBy: state.orderByFn === undefined + ? orderByClause(state.orderBySpecs).map(raw) + : orderByExprs(state.orderByFn($)).map(raw), limit: rowCount("limit", state.limitValue), offset: rowCount("offset", state.offsetValue), format: options?.skipFormat ? undefined : formatClause(state.formatValue), @@ -1308,7 +1321,8 @@ export function compileUnionRaw>( params?: object, options?: { rowSchema?: CompiledQueryRowSchema; deferParams?: boolean; dialect?: Dialect }, ): CompiledQuery { - return withDialect(options?.dialect ?? currentDialect(), () => compileUnionInner(union, params ?? {}, options)) + const dialect = options?.dialect ?? currentDialect() + return withDialect(dialect, () => withBoundLiterals(dialect, () => compileUnionInner(union, params ?? {}, options))) } /** The recursion behind {@link compileUnionUnsafe}; see {@link compileInner}. */ @@ -1449,11 +1463,12 @@ function renderParams( const style = dialect.params const resolved = sql.replace(PARAM_PLACEHOLDER_PATTERN, (placeholder, kind: string, name: string) => { - if (!(name in params)) { + const literal = boundLiterals?.get(name) + if (!(name in params) && literal === undefined) { missing.push(name) return placeholder } - const value = encodeParam(dialect, kind as ParamKind, name, params[name]) + const value = literal !== undefined ? literal.wire : encodeParam(dialect, kind as ParamKind, name, params[name]) if (style._tag === "inline") return checkedLiteral(dialect, value, paramContext(kind, name)) const key = `${kind}\0${name}` @@ -1540,6 +1555,40 @@ const insertWireValue = custom("insert value", Schema.Unknown) /** Prefix of the params an insert's literal values become. */ const VALUE_PARAM = "$$v" +/** Prefix of the params a compared value becomes on a dialect that binds. */ +const LITERAL_PARAM = "$$l" + +/** The values bound by the enclosing compile, by param name; see `withBoundLiterals`. */ +let boundLiterals: Map | undefined + +/** + * Run a compile with its literal values bound, on a dialect that binds params: + * each value becomes a param of its own, the same value the same param, so the + * statement text carries no values and renders the same way every time it is + * rendered. A nested compile shares the outer one's values. + */ +function withBoundLiterals(dialect: Dialect, body: () => A): A { + if (dialect.params._tag !== "bind" || activeLiteralBinder() !== undefined) return body() + const values = new Map() + const names = new Map() + const previous = boundLiterals + boundLiterals = values + try { + return withLiteralBinder((wire) => { + const key = `${typeof wire}\0${JSON.stringify(wire)}` + let name = names.get(key) + if (name === undefined) { + name = `${LITERAL_PARAM}${names.size}` + names.set(key, name) + values.set(name, { wire }) + } + return compileSqlFragment(param.of(insertWireValue, name).toFragment()) + }, body) + } finally { + boundLiterals = previous + } +} + const EMPTY_ROW = Schema.Struct({}) as unknown as CompiledQueryRowSchema /** @@ -1575,8 +1624,10 @@ const onConflictClause = ( return fail(new QueryBuilderDefect({ message: `${where}: the conflict target must name columns of the table` }), "") } const predicate = conflict.targetWhere?.(createColumnAccessor(table.columns)) - target = ` (${targetColumns.map(quoteIdent).join(", ")})${ - predicate === undefined ? "" : ` WHERE ${compileSqlFragment(predicate.toFragment())}` + // Inline, never bound: Postgres infers the partial unique index from this predicate's text. + const predicateSql = predicate === undefined ? undefined : withLiteralBinder(undefined, () => compileSqlFragment(predicate.toFragment())) + target = ` (${targetColumns.map((column) => quoteIdent(columnSqlName(table.columns, column))).join(", ")})${ + predicateSql === undefined ? "" : ` WHERE ${predicateSql}` }` } } else if (conflict.targetWhere !== undefined) { @@ -1713,7 +1764,7 @@ const setAssignments = ( } const sql = cell(column, value, `${context} set`) wrote(column, value, sql) - return [`${quoteIdent(column)} = ${sql}`] + return [`${quoteIdent(columnSqlName(table.columns, column))} = ${sql}`] }) if (assignments.length === 0) { return fail(new QueryBuilderError({ code: "InvalidArguments", message: `${where}: ${context} sets no columns` }),[]) @@ -1856,7 +1907,7 @@ function compileInsert(insert: CHInsert, params: Record quoteIdent(columnSqlName(table.columns, column))).join(", ")})${writeSettingsClause(table, insert._state.settings, where)}\n${source}${conflictSql}${returning?.sql ?? ""}`, values, dialect, ) diff --git a/src/ch/dialect.test.ts b/src/ch/dialect.test.ts index 64f932b..2ac55b5 100644 --- a/src/ch/dialect.test.ts +++ b/src/ch/dialect.test.ts @@ -74,7 +74,8 @@ describe("dialect params", () => { .select(($) => ({ count: $.Count })) .where(($) => [$.OrgId.eq("org"), $.Timestamp.gte(CH.param.dateTime("start"))]) const compiled = compileCHUnsafe(query, { start: new Date("2026-01-01T00:00:00.250Z") }, { dialect: numbered }) - expect(compiled.parameters).toEqual(["2026-01-01 00:00:00.250"]) + // The compared literal is bound too, as its wire value. + expect(compiled.parameters).toEqual(["org", "2026-01-01 00:00:00.250"]) }) it("numbered placeholders reuse one slot for a repeated param; positional ones bind it again", () => { diff --git a/src/ch/expr.ts b/src/ch/expr.ts index 02da2db..ab439be 100644 --- a/src/ch/expr.ts +++ b/src/ch/expr.ts @@ -10,7 +10,8 @@ import { type Brand, DateTime, Result, Schema } from "effect" import type { SqlFragment } from "../sql/sql-fragment" import { raw, str, ident, compile, as_ as sqlAs, known } from "../sql/sql-fragment" import { activeSqlSyntax } from "../sql/sql-syntax" -import { chDateTimeLiteral, CHFloatResult, CHNumber, string as chString, type CHType, type InferTS } from "./types" +import { activeLiteralBinder } from "../sql/literal-binder" +import { chDateTimeLiteral, CHFloatResult, CHNumber, string as chString, type CHType, type InferEncoded, type InferTS } from "./types" import { encodeColumnLiteral } from "./literal" import { QueryBuilderError } from "./errors" import { fail } from "./failure" @@ -46,6 +47,19 @@ export type Widen = TSType extends Brand.Brand ? number : TSType +/** + * What a plain value compared with a column may be. A literal union stays + * itself, so `$.kind.eq("nope")` is a type error when `kind` is + * `"open" | "closed"`; anything else widens as `Widen` does. A param or + * expression of the primitive still compares (see `Operand`), for a value only + * known at run time. + */ +export type ComparableValue = TSType extends string + ? [Exclude] extends [never] + ? TSType + : Widen + : Widen + // Params in the type // // An expression remembers the `param.*` placeholders it contains, so a query @@ -174,10 +188,11 @@ export interface Expr { lt(other: Operand): Condition

lte(other: Operand): Condition

- // String operations. A `Nullable(String)` matches like a `String`. - like(this: Expr, pattern: string): Condition

- notLike(this: Expr, pattern: string): Condition

- ilike(this: Expr, pattern: string): Condition

+ // String operations. A `Nullable(String)` matches like a `String`. The pattern is a + // string, bound on a dialect that binds params, or a param or expression of one. + like(this: Expr, pattern: string | Expr): Condition

+ notLike(this: Expr, pattern: string | Expr): Condition

+ ilike(this: Expr, pattern: string | Expr): Condition

// NULL and ranges /** `expr IS NULL`. */ @@ -191,8 +206,8 @@ export interface Expr { // IN / NOT IN. An empty list is false (`IN`) or true (`NOT IN`), written // `1 = 0` / `1 = 1`, rather than the `IN ()` no database accepts. - in_(...values: Array>>>): Condition

- notIn(...values: Array>>>): Condition

+ in_(...values: Array>>>): Condition

+ notIn(...values: Array>>>): Condition

// JSON represents non-finite division results as null. Other arithmetic // propagates SQL NULL from either operand. @@ -217,7 +232,7 @@ export interface Expr { * `null`), or an expression of it. */ export type Operand = - | Comparable>> + | Comparable>> | Expr | Expr, Q> @@ -301,6 +316,17 @@ export function toFragment(value: unknown): SqlFragment { const untypedLiteral = (value: boolean): string => activeSqlSyntax()?.literal(value, "an untyped boolean") ?? (value ? "1" : "0") +/** A string value, bound when the compile binds values, else a quoted literal. */ +const boundString = (value: string): SqlFragment => + known(() => { + const bind = activeLiteralBinder() + return bind !== undefined ? bind(value) : compile(str(value)) + }) + +/** A LIKE pattern: an expression as it renders, a string as `boundString`. */ +const likePattern = (pattern: string | Expr): string => + compile(typeof pattern === "string" ? boundString(pattern) : pattern.toFragment()) + const dateTimeLiteral = (value: DateTime.Utc): string => activeSqlSyntax()?.dateTimeLiteral(value) ?? compile(str(chDateTimeLiteral(value))) @@ -423,9 +449,9 @@ export function makeExpr( notBetween: (low, high) => makeCond(known(() => `${compile(fragment)} NOT BETWEEN ${compile(operand(low))} AND ${compile(operand(high))}`)), - like: (pattern: string) => makeCond(known(() => `${compile(fragment)} LIKE ${compile(str(pattern))}`)), - notLike: (pattern: string) => makeCond(known(() => `${compile(fragment)} NOT LIKE ${compile(str(pattern))}`)), - ilike: (pattern: string) => makeCond(known(() => `${compile(fragment)} ILIKE ${compile(str(pattern))}`)), + like: (pattern) => makeCond(known(() => `${compile(fragment)} LIKE ${likePattern(pattern)}`)), + notLike: (pattern) => makeCond(known(() => `${compile(fragment)} NOT LIKE ${likePattern(pattern)}`)), + ilike: (pattern) => makeCond(known(() => `${compile(fragment)} ILIKE ${likePattern(pattern)}`)), in_: (...values) => inCond(fragment, "IN", values.map((v) => () => operand(v))), notIn: (...values) => inCond(fragment, "NOT IN", values.map((v) => () => operand(v))), @@ -470,6 +496,17 @@ export function makeUntypedExpr( const columnTypes = new WeakMap, CHType>() export const columnTypeOf = (expr: Expr): CHType | undefined => columnTypes.get(expr) +/** + * A column read as the driver sends it, without its type's decoding: a jsonb + * document as `unknown`, a branded id as its string. For a reader that decodes + * stored values itself, say to tolerate an older document shape. Writes and + * every other read keep the column's type. + */ +export function undecoded>(column: ColumnRef): Expr> { + // The value passes through as the driver sent it, which is what the column's wire type describes. + return makeExpr>(column.toFragment(), Schema.Unknown as Schema.Codec, unknown>) +} + // ColumnRef implementation export function makeColumnRef>( @@ -495,7 +532,10 @@ export function makeColumnRef>( fragment, columnType?.schema as Schema.Codec, any> | undefined, @@ -503,7 +543,11 @@ export function makeColumnRef known(() => encodeColumnLiteral(columnType, value, columnName ?? name)), ) - if (columnType !== undefined) columnTypes.set(base, columnType) + // Without `sqlName`: a query reading this column as a subquery's output sees it under its alias. + if (columnType !== undefined) { + const { sqlName: _, ...outputType } = columnType + columnTypes.set(base, sqlName === undefined ? columnType : outputType) + } const isTenantColumn = tenantColumn !== undefined && (columnName ?? name) === tenantColumn const baseEq = base.eq const baseIn = base.in_ @@ -598,7 +642,7 @@ export function outerRef(name: string): Expr { } export function inList(expr: Expr, values: readonly string[]): Condition { - return inCond(expr.toFragment(), "IN", values.map((v) => () => str(v))) + return inCond(expr.toFragment(), "IN", values.map((v) => () => boundString(v))) } export function inExprList(expr: Expr, values: readonly Expr[]): Condition { @@ -606,7 +650,7 @@ export function inExprList(expr: Expr, values: readonly Expr[]): Condit } export function notInList(expr: Expr, values: readonly string[]): Condition { - return inCond(expr.toFragment(), "NOT IN", values.map((v) => () => str(v))) + return inCond(expr.toFragment(), "NOT IN", values.map((v) => () => boundString(v))) } /** diff --git a/src/ch/insert.test.ts b/src/ch/insert.test.ts index 383b6eb..7017ed1 100644 --- a/src/ch/insert.test.ts +++ b/src/ch/insert.test.ts @@ -182,9 +182,9 @@ describe("insertInto", () => { expect(compiled.sql).toBe( 'INSERT INTO "counters" ("org", "key", "count", "locked")\nVALUES ($1, $2, $3, $4)\n' + 'ON CONFLICT ("org", "key") DO UPDATE SET "count" = "counters"."count" + "excluded"."count", "locked" = $5 ' + - 'WHERE "counters"."locked" = FALSE\nRETURNING "count" AS "count"', + 'WHERE "counters"."locked" = $6\nRETURNING "count" AS "count"', ) - expect(compiled.parameters).toEqual(["o", "k", 1, false, true]) + expect(compiled.parameters).toEqual(["o", "k", 1, false, true, false]) }) it("a SET that writes another tenant makes the insert cross-tenant", () => { diff --git a/src/ch/literal.ts b/src/ch/literal.ts index 61eb006..6da9561 100644 --- a/src/ch/literal.ts +++ b/src/ch/literal.ts @@ -11,6 +11,7 @@ // Object]` and `Tags = a,b` — SQL that is either an error or, worse, a bare // identifier that happens to parse. +import { activeLiteralBinder } from "../sql/literal-binder" import { Result, Schema } from "effect" import { QueryBuilderError } from "./errors" import { quoteClickHouseString } from "../sql/sql-fragment" @@ -90,7 +91,9 @@ export function sqlLiteral(value: unknown, context: string): string { * fails here — while building the SQL — instead of becoming part of it. */ export function encodeLiteral(schema: Schema.Codec, value: unknown, context: string): string { - return (activeSqlSyntax()?.literal ?? sqlLiteral)(encodeValue(schema, value, context), context) + const wire = encodeValue(schema, value, context) + const bind = activeLiteralBinder() + return bind !== undefined ? bind(wire) : (activeSqlSyntax()?.literal ?? sqlLiteral)(wire, context) } /** diff --git a/src/ch/predicates.test.ts b/src/ch/predicates.test.ts index 69beb94..515f721 100644 --- a/src/ch/predicates.test.ts +++ b/src/ch/predicates.test.ts @@ -39,8 +39,8 @@ describe("null and range predicates", () => { CH.from(Jobs).select("id").where(($) => [$.id.between(CH.param.int("lo"), 9), $.done_at.isNull()]), { lo: 1 }, ) - expect(where(compiled.sql)).toBe('WHERE "jobs"."id" BETWEEN $1 AND 9\n AND "jobs"."done_at" IS NULL') - expect(compiled.parameters).toEqual([1]) + expect(where(compiled.sql)).toBe('WHERE "jobs"."id" BETWEEN $1 AND $2\n AND "jobs"."done_at" IS NULL') + expect(compiled.parameters).toEqual([1, 9]) }) }) diff --git a/src/ch/query.ts b/src/ch/query.ts index 342edab..2767211 100644 --- a/src/ch/query.ts +++ b/src/ch/query.ts @@ -32,6 +32,25 @@ import type { TenantScope } from "./compile" // Type utilities +/** + * Proxy traps that let `{ ...$ }` spread an accessor into every column of the + * table, under its key: `returning(($) => ({ ...$, txid }))`. A joined table's + * alias is not a column and is left out. + */ +const spreadColumns = (columns: ColumnDefs | undefined, get: (column: string) => unknown) => ({ + ownKeys: () => Object.keys(columns ?? {}), + getOwnPropertyDescriptor: (_target: object, prop: string | symbol) => + typeof prop === "string" && columns !== undefined && Object.hasOwn(columns, prop) + ? { enumerable: true, configurable: true, writable: false, value: get(prop) } + : undefined, +}) + +/** A select callback reading each named column under its own key. */ +const selectEvery = + (columns: ReadonlyArray) => + ($: any): Record => + Object.fromEntries(columns.map((column) => [column, $[column]])) + export type ColumnAccessor = { readonly [K in keyof Cols & string]: ColumnRef } @@ -113,6 +132,8 @@ export interface CHQueryState { * scope — see `having()` on the interface. */ readonly havingFn?: ($: any) => ReadonlyArray readonly orderBySpecs: Array<[string, "asc" | "desc"]> + /** Set by the callback form of `orderBy`, which sorts by expressions. */ + readonly orderByFn?: ($: any) => ReadonlyArray, "asc" | "desc"]> readonly limitValue?: number readonly offsetValue?: number readonly formatValue?: string @@ -210,6 +231,9 @@ export interface CHQuery< /** phantom */ readonly _phantom?: { cols: Cols; output: Output; joins: Joins; route: Route; params: (entries: Params) => void } + /** Select every column of the FROM table, as drizzle's bare `select()` does. Output keys are the column keys. */ + select(): CHQuery }, Joins, Route, Params> + /** Select specific columns by name. Output keys match column names. */ select( ...columns: K[] @@ -248,6 +272,10 @@ export interface CHQuery< ): CHQuery> orderBy(...specs: Array>): CHQuery + /** Sort by expressions of the source's columns, selected or not: `orderBy(($) => [[$.CreatedAt, "desc"]])`. */ + orderBy( + fn: ($: JoinedColumnAccessor) => ReadonlyArray, "asc" | "desc"]>, + ): CHQuery /** At most `n` rows: a non-negative integer. */ limit(n: RowCount): CHQuery @@ -432,17 +460,18 @@ export function createColumnAccessor( tenantColumn?: string, ): ColumnAccessor { const cache = new Map>>() + const get = (prop: string) => { + let ref = cache.get(prop) + if (!ref) { + ref = makeColumnRef(prop, undefined, tenantColumn, columns[prop]) + cache.set(prop, ref) + } + return ref + } - return new Proxy({} as ColumnAccessor, { - get(_target, prop) { - if (typeof prop !== "string") return undefined - let ref = cache.get(prop) - if (!ref) { - ref = makeColumnRef(prop, undefined, tenantColumn, columns[prop]) - cache.set(prop, ref) - } - return ref - }, + return new Proxy>({} as ColumnAccessor, { + get: (_target, prop) => (typeof prop === "string" ? get(prop) : undefined), + ...spreadColumns(columns, get), }) } @@ -454,17 +483,18 @@ export function createQualifiedColumnAccessor( columns?: ColumnDefs, ): ColumnAccessor { const cache = new Map>>() + const get = (prop: string) => { + let ref = cache.get(prop) + if (!ref) { + ref = makeColumnRef(`${alias}.${prop}`, prop, tenantColumn, columns?.[prop]) + cache.set(prop, ref) + } + return ref + } - return new Proxy({} as ColumnAccessor, { - get(_target, prop) { - if (typeof prop !== "string") return undefined - let ref = cache.get(prop) - if (!ref) { - ref = makeColumnRef(`${alias}.${prop}`, prop, tenantColumn, columns?.[prop]) - cache.set(prop, ref) - } - return ref - }, + return new Proxy>({} as ColumnAccessor, { + get: (_target, prop) => (typeof prop === "string" ? get(prop) : undefined), + ...spreadColumns(columns, get), }) } @@ -486,7 +516,8 @@ export function createJoinedColumnAccessor() const aliasSet = new Set(joinAliases) - return new Proxy({} as JoinedColumnAccessor, { + const accessor: JoinedColumnAccessor = new Proxy>({} as JoinedColumnAccessor, { + ...spreadColumns(columns, (column) => (accessor as Record)[column]), get(_target, prop) { if (typeof prop !== "string") return undefined let cached = cache.get(prop) @@ -506,6 +537,7 @@ export function createJoinedColumnAccessor ({ Col1: $.Col1, Col2: $.Col2 })) - if (typeof args[0] === "string") { - const columns = args as string[] - return makeQuery({ - ...state, - selectFn: ($: any) => { - const result: Record = {} - for (const col of columns) result[col] = $[col] - return result - }, - }) - } + if (typeof args[0] === "string") return makeQuery({ ...state, selectFn: selectEvery(args as string[]) }) // Callback overload: select($ => ({ ... })) return makeQuery({ ...state, selectFn: args[0] }) }, @@ -565,8 +588,9 @@ function makeQuery< return makeQuery({ ...state, havingFn: appendConditions(state.havingFn, fn) }) }, - orderBy(...specs) { - return makeQuery({ ...state, orderBySpecs: specs as Array<[string, "asc" | "desc"]> }) + orderBy(...specs: Array): any { + if (typeof specs[0] === "function") return makeQuery({ ...state, orderBySpecs: [], orderByFn: specs[0] }) + return makeQuery({ ...state, orderBySpecs: specs as Array<[string, "asc" | "desc"]>, orderByFn: undefined }) }, limit(n) { diff --git a/src/ch/sql-template.test.ts b/src/ch/sql-template.test.ts index a51c4f0..baec3d0 100644 --- a/src/ch/sql-template.test.ts +++ b/src/ch/sql-template.test.ts @@ -21,8 +21,9 @@ describe("CH.sql", () => { expect(compiled.sql).toContain('(pg_current_xact_id()::xid::text) AS "txid"') expect(compiled.sql).toContain('("keys"."uses" + 1) AS "next"') expect(compiled.sql).toContain(`("keys"."meta" @> $2::jsonb)`) - expect(compiled.sql).toContain(`("keys"."id" <> 'it''s')`) - expect(compiled.parameters).toEqual(["o", '{"a":1}']) + // A plain string is bound on Postgres, like a param. + expect(compiled.sql).toContain(`("keys"."id" <> $3)`) + expect(compiled.parameters).toEqual(["o", '{"a":1}', "it's"]) expect(compiled.rowSchemaSource).toBe("derived") expect(compiled.tenantScope).toBe("single-tenant") @@ -65,10 +66,10 @@ describe("CH.sql", () => { ]), { c: "z" }, ) - expect(compiled.sql).toContain(`("keys"."org" IN ('a', 'b', $1))`) - expect(compiled.sql).toMatch(/\("keys"\."id" IN \(SELECT[\s\S]*"other"\."org" = \$1\)\)/) + expect(compiled.sql).toContain(`("keys"."org" IN ($1, $2, $3))`) + expect(compiled.sql).toMatch(/\("keys"\."id" IN \(SELECT[\s\S]*"other"\."org" = \$3\)\)/) expect(compiled.sql).toContain(`((length("keys"."org")) > 2)`) - expect(compiled.parameters).toEqual(["z"]) + expect(compiled.parameters).toEqual(["a", "b", "z"]) expect(compiled.tenantScope).toBe("cross-tenant") }) diff --git a/src/ch/sql-template.ts b/src/ch/sql-template.ts index 31c2181..61193e9 100644 --- a/src/ch/sql-template.ts +++ b/src/ch/sql-template.ts @@ -21,6 +21,7 @@ // `sql.ident` values are recognised by identity (a private WeakSet), never by a // field, so an object parsed from request JSON cannot pass for one. +import { activeLiteralBinder } from "../sql/literal-binder" import { fail } from "./failure" import { DateTime } from "effect" import { currentDialect } from "./dialect" @@ -132,6 +133,9 @@ const renderValueRaw = (value: unknown): string => { if (isExprLike(value) && "_paramName" in value && currentDialect().params._tag === "inline") { return `(${compile(value.toFragment())})` } + // A string is data more often than not (an id, an email, a JSON document): bound where the compile binds. + const bind = typeof value === "string" ? activeLiteralBinder() : undefined + if (bind !== undefined) return bind(value) if ( isExprLike(value) || typeof value === "string" || diff --git a/src/ch/types.ts b/src/ch/types.ts index 543dc8d..24b2088 100644 --- a/src/ch/types.ts +++ b/src/ch/types.ts @@ -42,9 +42,18 @@ export interface CHType { * (`$.Attrs.get(k)`) needs `V` to know how the result decodes. */ readonly element?: CHType + /** + * The column's name in the database, when a table declares it under another + * key (`PG.column(PG.text, { name: "org_id" })` read as `$.orgId`). Every + * place that writes the column's identifier writes this instead of the key. + */ + readonly sqlName?: string readonly _phantom?: A } +/** The name a table's column has in the database: its `sqlName`, or its key. */ +export const columnSqlName = (columns: ColumnDefs, key: string): string => columns[key]?.sqlName ?? key + const chType = ( _tag: Tag, sql: string, diff --git a/src/ch/update.test.ts b/src/ch/update.test.ts index 43d3fe5..b4e72c9 100644 --- a/src/ch/update.test.ts +++ b/src/ch/update.test.ts @@ -24,9 +24,9 @@ describe("update", () => { { org: "o" }, ) expect(compiled.sql).toBe( - 'UPDATE "counters" SET "count" = "count" + 1, "meta" = $1\nWHERE "org" = $2\n AND "key" = \'k\'\nRETURNING "count" AS "count"', + 'UPDATE "counters" SET "count" = "count" + 1, "meta" = $1\nWHERE "org" = $2\n AND "key" = $3\nRETURNING "count" AS "count"', ) - expect(compiled.parameters).toEqual(['{"a":1}', "o"]) + expect(compiled.parameters).toEqual(['{"a":1}', "o", "k"]) expect(compiled.kind).toBe("update") expect(compiled.returning).toEqual(["count"]) expect(compiled.tenantScope).toBe("single-tenant") @@ -150,7 +150,7 @@ describe("deleteFrom", () => { expect(error.message).toContain("would write every row") } expect(PG.compileUnsafe(CH.deleteFrom(Counters).where(($) => [CH.rawCond(""), $.key.eq("k")])).sql).toBe( - 'DELETE FROM "counters"\nWHERE "key" = \'k\'', + 'DELETE FROM "counters"\nWHERE "key" = $1', ) }), ) diff --git a/src/core.ts b/src/core.ts index 46eee19..1570b94 100644 --- a/src/core.ts +++ b/src/core.ts @@ -32,6 +32,7 @@ export { when, whenTrue, inList, + undecoded, inExprList, notInList, not, diff --git a/src/kit/generate.ts b/src/kit/generate.ts index 483b351..d9df747 100644 --- a/src/kit/generate.ts +++ b/src/kit/generate.ts @@ -6,7 +6,7 @@ import { join, resolve } from "node:path" import { pathToFileURL } from "node:url" import { Effect, Schema } from "effect" import type { Layer } from "effect" -import { fromRecord, isStagingName, type LoadedMigration, type MigrationInput } from "../migrate/source" +import { fromRecord, isStagingName, STATEMENT_BREAKPOINT, type LoadedMigration, type MigrationInput } from "../migrate/source" import type { MigrationDriver } from "../migrate/driver" import { diffSchemas, type DiffResult, type Hint } from "../schema/diff" import { fromDrizzleSnapshot } from "../schema/drizzle" @@ -39,6 +39,12 @@ export interface KitConfig { readonly render?: RenderOptions /** Needed by `migrate`, `status`, and `verify`. Build it from your `SqlClient`. */ readonly driver?: Layer.Layer + /** + * What a generated Postgres migration holds. `ops` (default) writes `migration.json`; + * `sql` writes the rendered statements to `migration.sql`, split by + * `--> statement-breakpoint`, so a tool that applies drizzle-kit folders applies it too. + */ + readonly emit?: "ops" | "sql" } /** Typed identity, for `effect-orm.config.ts`. */ @@ -152,6 +158,12 @@ export interface GenerateResult { readonly plan: ReadonlyArray<{ readonly label: string; readonly sql: ReadonlyArray }> } +/** Each op's statements under a comment naming its label, one statement per breakpoint. */ +const renderPgSqlFile = (ops: ReadonlyArray): string => + `${ops + .flatMap((op) => renderPgOp(op).map((statement) => `-- ${labelOfPg(op)}\n${statement};`)) + .join(`\n${STATEMENT_BREAKPOINT}\n`)}\n` + /** `effect-orm generate`. */ export const generate = (config: KitConfig, cwd: string, options: GenerateOptions = {}) => Effect.gen(function* () { @@ -265,7 +277,9 @@ export const generate = (config: KitConfig, cwd: string, options: GenerateOption ? "-- Baseline: the schema as it stood when effect-orm took over this folder. Runs nothing.\n" : "-- Custom SQL migration. Separate statements with a line holding only:\n-- --> statement-breakpoint\n", ) - : writeFile(join(staging, "migration.json"), `${JSON.stringify(file, null, "\t")}\n`), + : "dialect" in file && config.emit === "sql" + ? writeFile(join(staging, "migration.sql"), renderPgSqlFile(file.ops)) + : writeFile(join(staging, "migration.json"), `${JSON.stringify(file, null, "\t")}\n`), `Cannot write ${dir}`, ) yield* io(() => writeFile(join(staging, "snapshot.json"), serializeSnapshot(snapshot)), `Cannot write ${dir}`) diff --git a/src/kit/kit.test.ts b/src/kit/kit.test.ts index 2f6f787..6b3b190 100644 --- a/src/kit/kit.test.ts +++ b/src/kit/kit.test.ts @@ -138,6 +138,20 @@ export const Dashboards = PG.table("dashboards", { expect(await cli("generate", "--config", "ch.config.ts")).toBe(1) }) + it("writes migration.sql with emit: sql, which the next generate reads as its parent", async () => { + writeFileSync(join(dir, "pg1.ts"), pgModule()) + writeFileSync(join(dir, "sql.config.ts"), `export default { dialect: "postgres", emit: "sql", schema: "./pg1.ts", out: "./migrations" }\n`) + expect(await cli("generate", "--name", "init", "--config", "sql.config.ts")).toBe(0) + const [first] = folders() + expect(readdirSync(join(dir, "migrations", first!)).sort()).toEqual(["migration.sql", "snapshot.json"]) + const sql = readFileSync(join(dir, "migrations", first!, "migration.sql"), "utf8") + expect(sql).toMatch(/^-- metadata\nCREATE TABLE IF NOT EXISTS "dashboards"/) + expect(sql).toContain(`;\n--> statement-breakpoint\n-- metadata\nCREATE INDEX IF NOT EXISTS "dashboards_open_idx"`) + expect(await cli("generate", "--config", "sql.config.ts")).toBe(0) + expect(folders()).toHaveLength(1) + expect(await cli("check", "--config", "sql.config.ts")).toBe(0) + }) + it("adopts a drizzle-kit folder: baseline from its last snapshot, then diff the definitions against it", async () => { const legacy = join(dir, "migrations", "20260101000000_drizzle_init") mkdirSync(legacy, { recursive: true }) diff --git a/src/pg/bound-literals.test.ts b/src/pg/bound-literals.test.ts new file mode 100644 index 0000000..b45b6f5 --- /dev/null +++ b/src/pg/bound-literals.test.ts @@ -0,0 +1,101 @@ +// On Postgres a value compared with a column is bound, not written into the +// statement: the text carries no values (logs, traces, query shapes), and the +// same value is one param. ClickHouse and DDL keep writing literals. + +import { PGlite } from "@electric-sql/pglite" +import { afterAll, describe, expect, it } from "@effect/vitest" +import { Effect } from "effect" +import * as CH from "../ch/index" +import * as PG from "../postgres" +import * as S from "../schema" + +const Users = PG.table("users", { + columns: { + orgId: PG.column(PG.text, { name: "org_id" }), + email: PG.text, + active: PG.column(PG.bool, { default: true }), + }, + primaryKey: ["orgId", "email"], + indexes: [PG.uniqueIndex("users_active_email_unq", ["email"], { where: ($) => $.active.eq(true) })], + tenantColumn: "orgId", +}) + +const db = new PGlite() +afterAll(() => db.close()) + +describe("bound literals", () => { + it("binds compared values, once per distinct value, and keeps the tenant scope", () => { + const compiled = PG.compileUnsafe( + PG.from(Users) + .select("email") + .where(($) => [$.orgId.eq("org_1"), $.email.in_("a@x", "org_1"), $.email.ilike(PG.param.string("q"))]), + { q: "%a%" }, + ) + expect(compiled.sql).not.toContain("org_1") + expect(compiled.sql).not.toContain("a@x") + expect(compiled.sql).toContain(`"org_id" = $1`) + expect(compiled.sql).toContain(`"email" IN ($2, $1)`) + expect(compiled.sql).toContain(`"email" ILIKE $3`) + expect(compiled.parameters).toEqual(["org_1", "a@x", "%a%"]) + expect(compiled.tenantScope).toBe("single-tenant") + }) + + it("binds a plain LIKE pattern", () => { + const compiled = PG.compileUnsafe(PG.from(Users).select("email").where(($) => [$.email.like("secret%")])) + expect(compiled.sql).toContain(`LIKE $1`) + expect(compiled.parameters).toEqual(["secret%"]) + }) + + it("writes ON CONFLICT's index predicate inline, so Postgres can match the partial index", async () => { + for (const statement of S.renderPgSchema(Effect.runSync(S.pgEntitiesOf([Users])))) await db.exec(statement) + const upsert = PG.insertInto(Users) + .values({ orgId: "o", email: "a@x" }) + .onConflictDoNothing({ target: ["email"], targetWhere: ($) => $.active.eq(true) }) + const compiled = PG.compileUnsafe(upsert) + expect(compiled.sql).toContain(`ON CONFLICT ("email") WHERE "active" = TRUE DO NOTHING`) + await db.query(compiled.sql, [...compiled.parameters]) + await db.query(compiled.sql, [...compiled.parameters]) + expect((await db.query(`SELECT count(*)::int AS n FROM users`)).rows).toEqual([{ n: 1 }]) + }) + + it("leaves ClickHouse and DDL writing literals", () => { + const Events = CH.table("events", { OrgId: CH.string, Name: CH.string }, { tenantColumn: "OrgId" }) + const compiled = CH.compileUnsafe(CH.from(Events).select("Name").where(($) => [$.OrgId.eq("o"), $.Name.like("a%")])) + expect(compiled.sql).toContain(`OrgId = 'o'`) + expect(compiled.sql).toContain(`LIKE 'a%'`) + const ddl = S.renderPgSchema(Effect.runSync(S.pgEntitiesOf([Users]))).join("\n") + expect(ddl).toContain(`"active" boolean NOT NULL DEFAULT true`) + expect(ddl).toContain(`WHERE "active" = TRUE`) + }) +}) + +describe("orderBy and inList", () => { + it("sorts by an expression that is not selected, and binds an inList", () => { + const compiled = PG.compileUnsafe( + PG.from(Users) + .select("email") + .where(($) => [PG.inList($.orgId, ["o1", "o2"])]) + .orderBy(($) => [ + [$.active, "desc"], + [PG.lower($.email), "asc"], + ]), + ) + expect(compiled.sql).toContain(`"org_id" IN ($1, $2)`) + expect(compiled.sql).toContain(`ORDER BY "users"."active" DESC, lower("users"."email") ASC`) + expect(compiled.parameters).toEqual(["o1", "o2"]) + }) +}) + +describe("orderBy params", () => { + it("accepts columns of any table and requires the params its expressions use", () => { + const byKeys = (keys: ReadonlyArray<"email" | "orgId">) => + PG.from(Users).select("email").orderBy(($) => keys.map((key) => [$[key], "asc"] as const)) + expect(PG.compileUnsafe(byKeys(["orgId", "email"])).sql).toContain(`ORDER BY "users"."org_id" ASC, "users"."email" ASC`) + const byParam = PG.from(Users) + .select("email") + .orderBy(($) => [[PG.coalesce($.email, PG.param.string("fallback")), "asc"]]) + // @ts-expect-error the fallback param is required + expect(() => PG.compileUnsafe(byParam, {})).toThrow("fallback") + expect(PG.compileUnsafe(byParam, { fallback: "z" }).parameters).toEqual(["z"]) + }) +}) diff --git a/src/pg/column-names.test.ts b/src/pg/column-names.test.ts new file mode 100644 index 0000000..84c9a47 --- /dev/null +++ b/src/pg/column-names.test.ts @@ -0,0 +1,191 @@ +// Columns declared under a key other than their database name: +// `orgId: PG.column(PG.text, { name: "org_id" })`. Every statement here runs on +// PGlite, so a test passes only if each place that writes an identifier wrote +// the database name and every row decodes under the key. + +import { PgliteClient } from "@effect/sql-pglite" +import { describe, expect, it, layer } from "@effect/vitest" +import { DateTime, Effect, Exit, Layer, Schema } from "effect" +import * as SqlClient from "effect/sql/SqlClient" +import * as Db from "../database" +import * as PG from "../postgres" +import * as S from "../schema" + +const Teams = PG.table("teams", { + columns: { + orgId: PG.column(PG.text, { name: "org_id" }), + teamId: PG.column(PG.text, { name: "team_id" }), + displayName: PG.column(PG.text, { name: "display_name" }), + }, + primaryKey: ["orgId", "teamId"], + tenantColumn: "orgId", +}) + +const Members = PG.table("members", { + columns: { + id: PG.column(PG.int4, { name: "member_id", identity: "always" }), + orgId: PG.column(PG.text, { name: "org_id" }), + teamId: PG.column(PG.text, { name: "team_id" }), + email: PG.text, + visitCount: PG.column(PG.int4, { name: "visit_count", default: 0 }), + joinedAt: PG.column(PG.timestamptz, { name: "joined_at", defaultExpr: "now()" }), + leftAt: PG.column(PG.nullable(PG.timestamptz), { name: "left_at" }), + }, + primaryKey: ["id"], + indexes: [ + PG.uniqueIndex("members_email_unq", ["orgId", "email"]), + PG.index("members_active_idx", ($) => [$.orgId, PG.lower($.email)], { where: ($) => $.leftAt.isNull() }), + ], + foreignKeys: [PG.foreignKey({ columns: ["orgId", "teamId"], references: Teams, foreignColumns: ["orgId", "teamId"] })], + tenantColumn: "orgId", +}) + +const Live = Layer.effect( + Db.Database, + Effect.gen(function* () { + return Db.fromSqlClient(yield* SqlClient.SqlClient, { dialect: PG.postgresDialect }) + }), +).pipe(Layer.provideMerge(PgliteClient.layer({ postgresqlconf: "timezone = 'UTC'" }))) + +describe("column names", () => { + it.effect("writes the database names into the DDL and keeps keys out of it", () => + Effect.gen(function* () { + const ddl = S.renderPgSchema(yield* S.pgEntitiesOf([Teams, Members])).join("\n") + expect(ddl).toContain(`"org_id" text NOT NULL`) + expect(ddl).toContain(`"visit_count" integer NOT NULL DEFAULT 0`) + expect(ddl).toContain(`PRIMARY KEY ("org_id", "team_id")`) + expect(ddl).toContain(`ON "members" USING btree ("org_id", "email")`) + expect(ddl).toContain(`USING btree ("org_id", lower("email")) WHERE "left_at" IS NULL`) + expect(ddl).toContain(`FOREIGN KEY ("org_id", "team_id") REFERENCES "teams" ("org_id", "team_id")`) + expect(ddl).toContain(`members_org_id_team_id_teams_org_id_team_id_fk`) + expect(ddl).not.toMatch(/"(orgId|teamId|visitCount|joinedAt|leftAt|displayName)"/) + }), + ) + + it("leaves a renamed column without a default required on insert", () => { + // @ts-expect-error displayName has a name but no default, so it is required + PG.insertInto(Teams).values({ orgId: "o", teamId: "t" }) + PG.insertInto(Members).values({ orgId: "o", teamId: "t", email: "a@x" }) + }) + + layer(Live)((it) => { + it.effect("reads, writes, upserts and deletes under the keys", () => + Effect.gen(function* () { + for (const statement of S.renderPgSchema(yield* S.pgEntitiesOf([Teams, Members]))) { + yield* Db.execute({ sql: statement }) + } + yield* Db.run(PG.insertInto(Teams).values({ orgId: "o1", teamId: "t1", displayName: "Core" })) + + const inserted = yield* Db.run( + PG.insertInto(Members) + .values([ + { orgId: "o1", teamId: "t1", email: "a@x" }, + { orgId: "o1", teamId: "t1", email: "b@x", visitCount: 3 }, + ]) + .returning(), + ) + expect(inserted.map((row) => [row.email, row.visitCount, row.leftAt])).toEqual([ + ["a@x", 0, null], + ["b@x", 3, null], + ]) + expect(DateTime.isDateTime(inserted[0]!.joinedAt)).toBe(true) + + const upserted = yield* Db.run( + PG.insertInto(Members) + .values({ orgId: "o1", teamId: "t1", email: "a@x", visitCount: 5 }) + .onConflictDoUpdate({ + target: ["orgId", "email"], + set: ($, excluded) => ({ visitCount: $.visitCount.add(excluded.visitCount) }), + }) + .returning("email", "visitCount"), + ) + expect(upserted).toEqual([{ email: "a@x", visitCount: 5 }]) + + const updated = yield* Db.run( + PG.update(Members) + .set({ leftAt: new Date("2026-01-01T00:00:00Z") }) + .where(($) => [$.orgId.eq(PG.param.string("orgId")), $.email.eq("b@x")]) + .returning(($) => ({ email: $.email, left: $.leftAt })), + { orgId: "o1" }, + ) + expect(updated.map((row) => row.left === null ? null : DateTime.formatIso(row.left))).toEqual([ + "2026-01-01T00:00:00.000Z", + ]) + + const joined = yield* Db.run( + PG.from(Members) + .innerJoin(Teams, "t", (m, t) => PG.and(m.orgId.eq(t.orgId), m.teamId.eq(t.teamId))) + .select(($) => ({ email: $.email, team: $.t.displayName, visits: $.visitCount })) + .where(($) => [$.orgId.eq("o1"), $.leftAt.isNull()]) + .orderBy(["email", "asc"]), + ) + expect(joined).toEqual([{ email: "a@x", team: "Core", visits: 5 }]) + + // A subquery's output is read under its alias, not the column's database name. + const inner = PG.from(Members) + .select(($) => ({ orgId: $.orgId, visitCount: $.visitCount })) + .where(($) => [$.orgId.eq("o1")]) + const total = yield* Db.run( + PG.fromQuery(inner, "m").select(($) => ({ total: PG.sum($.visitCount) })), + ) + expect(total).toEqual([{ total: 8 }]) + + const all = yield* Db.run(PG.from(Members).select().where(($) => [$.orgId.eq("o1")]).orderBy(["email", "asc"])) + expect(Object.keys(all[0]!).sort()).toEqual( + ["email", "id", "joinedAt", "leftAt", "orgId", "teamId", "visitCount"].sort(), + ) + + const deleted = yield* Db.run( + PG.deleteFrom(Members).where(($) => [$.orgId.eq("o1"), $.leftAt.isNotNull()]).returning("email"), + ) + expect(deleted).toEqual([{ email: "b@x" }]) + }), + ) + + it.effect("scopes a query by a renamed tenant column", () => + Effect.sync(() => { + const compiled = PG.compileUnsafe( + PG.from(Members).select("email").where(($) => [$.orgId.eq(PG.param.string("orgId"))]), + { orgId: "o1" }, + ) + expect(compiled.tenantScope).toBe("single-tenant") + expect(compiled.sql).toContain(`"org_id" = $1`) + }), + ) + }) +}) + +describe("spreading the accessor", () => { + it("reads every column under its key, beside computed ones", () => { + const compiled = PG.compileUnsafe( + PG.update(Members) + .set({ visitCount: 1 }) + .where(($) => [$.orgId.eq("o1")]) + .returning(($) => ({ ...$, txid: PG.sql(PG.text)`pg_current_xact_id()::xid::text` })), + ) + expect(compiled.returning).toEqual(["id", "orgId", "teamId", "email", "visitCount", "joinedAt", "leftAt", "txid"]) + expect(compiled.sql).toContain(`"member_id" AS "id"`) + const selected = PG.compileUnsafe(PG.from(Members).select(($) => ({ ...$, lowered: PG.lower($.email) }))) + expect(selected.sql).toContain(`"members"."left_at" AS "leftAt"`) + }) +}) + +describe("undecoded", () => { + it("reads a column as the driver sends it, typed as its wire form", async () => { + const Docs = PG.table("docs_undecoded", { + columns: { + id: PG.text, + body: PG.column(PG.nullable(PG.jsonb(Schema.Struct({ v: Schema.Number }))), { name: "body_json" }), + }, + primaryKey: ["id"], + }) + const query = PG.from(Docs).select(($) => ({ id: $.id, body: PG.undecoded($.body) })) + const compiled = PG.compileUnsafe(query) + expect(compiled.sql).toContain(`"docs_undecoded"."body_json" AS "body"`) + const rows = await Effect.runPromise(compiled.decodeRows([{ id: "a", body: { old: "shape" } }])) + expect(rows).toEqual([{ id: "a", body: { old: "shape" } }]) + // The typed read refuses the same row. + const strict = PG.compileUnsafe(PG.from(Docs).select("id", "body")) + expect(Exit.isFailure(await Effect.runPromiseExit(strict.decodeRows([{ id: "a", body: { old: "shape" } }])))).toBe(true) + }) +}) diff --git a/src/pg/functions.ts b/src/pg/functions.ts index 554fc8c..827d894 100644 --- a/src/pg/functions.ts +++ b/src/pg/functions.ts @@ -11,7 +11,9 @@ import { Schema, type DateTime } from "effect" import { QueryBuilderDefect } from "../ch/errors" import { makeExpr, type Condition, type Expr } from "../ch/expr" import { schemaOf, withoutNull } from "../ch/define-fn" -import { compile, str } from "../sql/sql-fragment" +import { compile, known, str } from "../sql/sql-fragment" +import { encodeLiteral } from "../ch/literal" +import type { CHType } from "../ch/types" import { builtins } from "../ch/functions/builtin" import * as T from "./types" @@ -131,3 +133,37 @@ export const coalesce = (expr: Expr, fa /** `expr ->> key`: a jsonb field as text, NULL when it is absent. */ export const jsonText = (expr: Expr, key: string): Expr => makeExpr(scalar.lazy(() => `(${sql(expr)} ->> ${compile(str(key))})`), Schema.NullOr(Schema.String) as Schema.Codec) + +/** `greatest(a, b, ...)`, decoding as `first` does. Postgres skips NULL arguments. */ +export const greatest = (first: Expr, ...rest: ReadonlyArray>): Expr => + makeExpr(scalar.lazy(() => `greatest(${[first, ...rest].map(sql).join(", ")})`), schemaOf(first)) + +/** `least(a, b, ...)`, decoding as `first` does. Postgres skips NULL arguments. */ +export const least = (first: Expr, ...rest: ReadonlyArray>): Expr => + makeExpr(scalar.lazy(() => `least(${[first, ...rest].map(sql).join(", ")})`), schemaOf(first)) + +/** + * `CASE WHEN c1 THEN v1 ... ELSE otherwise END`, decoding as `otherwise` does. + * Every branch has the result's type. + */ +export const caseWhen = ( + branches: ReadonlyArray, Expr]>, + otherwise: Expr, +): Expr => + makeExpr( + scalar.lazy( + () => `CASE ${branches.map(([when, then]) => `WHEN ${sql(when)} THEN ${sql(then)}`).join(" ")} ELSE ${sql(otherwise)} END`, + ), + schemaOf(otherwise), + ) + +/** A condition as a boolean value, to select it or `set` a column from it. */ +export const asBoolean = (condition: Condition): Expr => + makeExpr(scalar.lazy(() => `(${sql(condition)})`), Schema.Boolean as Schema.Codec) + +/** + * A value written as a column type writes it: encoded by its codec and bound, + * so `typedValue(T.columns.at, ms)` is a timestamptz wherever an expression goes. + */ +export const typedValue = (type: CHType, value: A): Expr => + makeExpr(known(() => encodeLiteral(type.literalSchema, value, "typedValue")), type.schema as Schema.Codec) diff --git a/src/pg/postgres.test.ts b/src/pg/postgres.test.ts index c82156a..336d4ae 100644 --- a/src/pg/postgres.test.ts +++ b/src/pg/postgres.test.ts @@ -69,8 +69,11 @@ describe("postgres dialect", () => { .orderBy(["service", "asc"]) const compiled = PG.compileUnsafe(query, { orgId: "org_1" }) - expect(compiled.sql).toContain(`"OrgId" = $1`) - expect(compiled.parameters).toEqual(["org_1"]) + // Values are bound like params, in the order the statement reads them. + expect(compiled.sql).toContain(`FILTER (WHERE "events"."Ok" = $1)`) + expect(compiled.sql).toContain(`"OrgId" = $2`) + expect(compiled.sql).toContain(`"Service" IN ($3, $4)`) + expect(compiled.parameters).toEqual([false, "org_1", "api", "web"]) expect(compiled.tenantScope).toBe("single-tenant") expect(await run(compiled)).toEqual([ { service: "api", events: 2, failures: 1, total: 30, average: 15 }, @@ -162,8 +165,9 @@ describe("postgres dialect", () => { .select(($) => ({ count: $.Count })) .where(($) => [$.OrgId.eq(CH.param.string("orgId")), $.Service.eq(tricky), $.Service.like("it's \\\\%")]) const compiled = PG.compileUnsafe(byLiteral, { orgId: "org_1" }) - expect(compiled.sql).toContain(`E'it''s \\\\ a \\x5F_PARAM_string_orgId__ value'`) - expect(compiled.parameters).toEqual(["org_1"]) + // Bound, never written into the statement: no quote, backslash or marker to escape. + expect(compiled.sql).not.toContain("it''s") + expect(compiled.parameters).toEqual(["org_1", tricky, "it's \\\\%"]) expect(await run(compiled)).toEqual([{ count: 1 }]) }) @@ -178,7 +182,7 @@ describe("postgres dialect", () => { .groupBy("region") .orderBy(["region", "asc"]) const compiled = PG.compileUnsafe(query, { ok: true }) - expect(compiled.parameters).toEqual([true]) + expect(compiled.parameters).toEqual(["org_1", true, '{"region":"us"}']) expect(await run(compiled)).toEqual([ { region: "eu", regions: ["api"] }, { region: null, regions: [tricky] }, diff --git a/src/pg/timestamptz-millis.test.ts b/src/pg/timestamptz-millis.test.ts new file mode 100644 index 0000000..d22cf25 --- /dev/null +++ b/src/pg/timestamptz-millis.test.ts @@ -0,0 +1,60 @@ +import { PgliteClient } from "@effect/sql-pglite" +import { describe, expect, layer } from "@effect/vitest" +import { DateTime, Effect, Layer, Schema } from "effect" +import * as SqlClient from "effect/sql/SqlClient" +import * as Db from "../database" +import * as PG from "../postgres" + +const Jobs = PG.table("jobs", { + columns: { + id: PG.text, + runAt: PG.column(PG.timestamptzMillis, { name: "run_at" }), + doneAt: PG.column(PG.nullable(PG.timestamptzMillis), { name: "done_at" }), + }, + primaryKey: ["id"], +}) + +const Live = Layer.effect( + Db.Database, + Effect.gen(function* () { + return Db.fromSqlClient(yield* SqlClient.SqlClient, { dialect: PG.postgresDialect }) + }), +).pipe(Layer.provideMerge(PgliteClient.layer({ postgresqlconf: "timezone = 'UTC'" }))) + +const t0 = Date.parse("2026-01-01T00:00:00.250Z") + +describe("timestamptzMillis", () => { + layer(Live)((it) => { + it.effect("writes and reads epoch milliseconds, and compares against every timestamp form", () => + Effect.gen(function* () { + yield* Db.execute(Db.sql`CREATE TABLE jobs (id text PRIMARY KEY, run_at timestamptz NOT NULL, done_at timestamptz)`) + yield* Db.run( + PG.insertInto(Jobs).values([ + { id: "a", runAt: t0 }, + { id: "b", runAt: t0 + 1000, doneAt: t0 + 2000 }, + ]), + ) + const rows = yield* Db.run(PG.from(Jobs).select().orderBy(["id", "asc"])) + expect(rows).toEqual([ + { id: "a", runAt: t0, doneAt: null }, + { id: "b", runAt: t0 + 1000, doneAt: t0 + 2000 }, + ]) + + // Typed as milliseconds; the codec also takes the other timestamp forms a driver or caller has. + const since = (value: number) => Db.run(PG.from(Jobs).select("id").where(($) => [$.runAt.gt(value)])) + for (const value of [t0, new Date(t0), DateTime.makeUnsafe(t0), "2026-01-01T00:00:00.250Z"]) { + expect(yield* since(value as number)).toEqual([{ id: "b" }]) + } + expect( + yield* Db.run(PG.from(Jobs).select("id").where(($) => [$.runAt.lte(PG.param.of(Jobs.columns.runAt, "at"))]), { + at: t0, + }), + ).toEqual([{ id: "a" }]) + + // The text form some drivers send decodes to the same instant. + const text = yield* Db.query(Db.sql`SELECT run_at::text AS run_at FROM jobs WHERE id = 'a'`) + expect(Schema.decodeUnknownSync(PG.timestamptzMillis.schema)(text[0]!.run_at)).toBe(t0) + }), + ) + }) +}) diff --git a/src/pg/types.ts b/src/pg/types.ts index 1d30818..f42b8c7 100644 --- a/src/pg/types.ts +++ b/src/pg/types.ts @@ -96,6 +96,34 @@ const isoInstant = (epochMillis: number): string => new Date(epochMillis).toISOS export const PgTimestampLiteral: Schema.Codec = timestampLiteral(isoInstant) +/** A timestamptz as a driver sends it, read as epoch milliseconds. */ +const PgTimestamptzMillis: Schema.Codec = Schema.Union([ + Schema.Date.pipe( + Schema.decodeTo(Schema.Finite, { + decode: SchemaGetter.transform((value: Date) => value.getTime()), + encode: SchemaGetter.transform((value: number) => new Date(value)), + }), + ), + Schema.String.pipe( + Schema.check(isTimestamp), + Schema.decodeTo(Schema.Finite, { + decode: SchemaGetter.transform((value: string) => Date.parse(pgTimestampToIso(value))), + encode: SchemaGetter.transform((value: number) => isoInstant(value)), + }), + ), +]) + +/** What a `timestamptzMillis` takes: epoch milliseconds, or anything a timestamptz does. */ +const PgTimestampMillisLiteral: Schema.Codec = Schema.Union([ + Schema.String.pipe( + Schema.decodeTo(Schema.Finite, { + decode: SchemaGetter.transform((value: string) => Date.parse(value)), + encode: SchemaGetter.transform((value: number) => isoInstant(value)), + }), + ), + PgTimestampLiteral, +]) as Schema.Codec + // Scalars export const text: PgType<"text", string> = custom("text", Schema.String) @@ -114,6 +142,17 @@ export const timestamptz: PgType<"timestamptz", DateTime.Utc, Date | string> = c PgTimestampLiteral, ) +/** + * A timestamptz read as epoch milliseconds, for code that keeps time as a + * number. Inserts and comparisons take milliseconds as well as a `Date`, a + * `DateTime.Utc` or a timestamp string. + */ +export const timestamptzMillis: PgType<"timestamptz", number, Date | string> = custom( + "timestamptz", + PgTimestamptzMillis, + PgTimestampMillisLiteral, +) + /** * A jsonb column, decoded with `schema` (anything, by default). Drivers parse * jsonb themselves, so rows arrive as values; a compared literal is written as diff --git a/src/schema/define.ts b/src/schema/define.ts index 00562f7..58839a9 100644 --- a/src/schema/define.ts +++ b/src/schema/define.ts @@ -212,16 +212,20 @@ export const externalTable = ( definition: { readonly columns: Record; readonly tenantColumn?: string }, isSpec: (input: unknown) => input is { readonly type: CHType; readonly options: object }, computedOptions: ReadonlyArray, + /** The query-side type of a spec; the Postgres one carries the column's `name`. */ + typeOf: (spec: { readonly type: CHType; readonly options: object }) => CHType = (spec) => spec.type, ): Table => { const inputs = Object.entries(definition.columns) const specs = inputs.filter((entry): entry is [string, { readonly type: CHType; readonly options: object }] => isSpec(entry[1]), ) const isComputed = (options: object) => computedOptions.some((key) => (options as Record)[key] !== undefined) - const given = (options: object) => Object.values(options).some((value) => value !== undefined) + // `name` renames a column; it gives the column no default. + const given = (options: object) => + Object.entries(options).some(([key, value]) => key !== "name" && value !== undefined) return table( name, - Object.fromEntries(inputs.map(([column, input]) => [column, isSpec(input) ? input.type : input])) as ColumnDefs, + Object.fromEntries(inputs.map(([column, input]) => [column, isSpec(input) ? typeOf(input) : input])) as ColumnDefs, { ...(definition.tenantColumn !== undefined ? { tenantColumn: definition.tenantColumn } : undefined), defaults: specs.filter(([, spec]) => given(spec.options) && !isComputed(spec.options)).map(([column]) => column), diff --git a/src/schema/pg-define.ts b/src/schema/pg-define.ts index 4f4d8a4..a2bdb93 100644 --- a/src/schema/pg-define.ts +++ b/src/schema/pg-define.ts @@ -13,7 +13,7 @@ import type { Condition, Expr } from "../ch/expr" import { encodeColumnLiteral } from "../ch/literal" import { createColumnAccessor, type ColumnAccessor } from "../ch/query" import type { Table } from "../ch/table" -import type { CHType, ColumnDefs, InferTS } from "../ch/types" +import { columnSqlName, type CHType, type ColumnDefs, type InferTS } from "../ch/types" import { postgresDialect } from "../pg/dialect" import { externalTable, type DdlExpr, type DdlKey } from "./define" import { withRenderProblems, type DefinitionProblem, type ProblemSink } from "./problems" @@ -41,11 +41,19 @@ const renderExpr = (expr: DdlExpr | DdlPredicate< /** Key parts: a column name is written quoted, an expression as it renders. */ const renderKeyParts = (key: DdlKey, columns: Cols): ReadonlyArray => - typeof key === "function" ? key(createColumnAccessor(columns)).map(renderValue) : key.map(quoteIdent) + typeof key === "function" + ? key(createColumnAccessor(columns)).map(renderValue) + : key.map((column) => quoteIdent(columnSqlName(columns, column))) // Columns export interface ColumnOptions> { + /** + * The column's name in the database, when it differs from its key in + * `columns`: `orgId: PG.column(PG.text, { name: "org_id" })` is read and + * written as `$.orgId` and stored as `org_id`. + */ + readonly name?: string /** A literal default, encoded through the column's own type. */ readonly default?: InferTS /** `DEFAULT `, for a computed default such as `"now()"`. */ @@ -85,13 +93,23 @@ export type ColumnsOf> = { /** Columns declared with `default`, `defaultExpr` or `identity`: an insert may leave them out. */ export type DefaultedColumnsOf> = { - [K in keyof I]: I[K] extends ColumnSpec ? ([Given] extends [never] ? never : K) : never + [K in keyof I]: I[K] extends ColumnSpec + ? [Exclude] extends [never] + ? never + : K + : never }[keyof I] & string const isColumnSpec = (input: ColumnInput): input is ColumnSpec, any> => "_tag" in input && input._tag === "PgColumnSpec" +/** The query-side type of a column input, carrying its `name` as `sqlName`. */ +export const columnTypeOfInput = (input: ColumnInput): CHType => { + if (!isColumnSpec(input)) return input + return input.options.name === undefined ? input.type : { ...input.type, sqlName: input.options.name } +} + // Indexes and foreign keys export interface IndexOptions { @@ -154,15 +172,21 @@ export const foreignKey = ____fk`, the name drizzle-kit gives. */ readonly name?: string -}): ForeignKeySpec => ({ - _tag: "PgForeignKeySpec", - columns: spec.columns, - references: typeof spec.references === "string" ? spec.references : spec.references.name, - foreignColumns: spec.foreignColumns, - onDelete: (spec.onDelete?.toUpperCase() ?? "NO ACTION") as PgReferentialAction, - onUpdate: (spec.onUpdate?.toUpperCase() ?? "NO ACTION") as PgReferentialAction, - name: spec.name, -}) +}): ForeignKeySpec => { + const references = spec.references + return { + _tag: "PgForeignKeySpec", + columns: spec.columns, + references: typeof references === "string" ? references : references.name, + foreignColumns: + typeof references === "string" + ? spec.foreignColumns + : spec.foreignColumns.map((column) => columnSqlName(references.columns, column)), + onDelete: (spec.onDelete?.toUpperCase() ?? "NO ACTION") as PgReferentialAction, + onUpdate: (spec.onUpdate?.toUpperCase() ?? "NO ACTION") as PgReferentialAction, + name: spec.name, + } +} // Tables @@ -288,7 +312,13 @@ export function table | ExternalTableDefinition, ): PgSchemaTable, DefaultedColumnsOf> | Table { if ("external" in definition) { - return externalTable(name, definition, (input): input is ColumnSpec> => isColumnSpec(input as ColumnInput), []) + return externalTable( + name, + definition, + (input): input is ColumnSpec> => isColumnSpec(input as ColumnInput), + [], + (spec) => columnTypeOfInput(spec as ColumnSpec>), + ) } return withRenderProblems(name, () => buildTable(name, definition)) } @@ -301,20 +331,19 @@ function buildTable [column, isColumnSpec(input) ? input.type : input]), - ) as ColumnsOf + const types = Object.fromEntries(inputs.map(([column, input]) => [column, columnTypeOfInput(input)])) as ColumnsOf + const sqlName = (column: string) => columnSqlName(types, column) const has = (column: string) => Object.hasOwn(types, column) const columnEntities = inputs.map(([column, input], position): PgColumnEntity => { - checkIdentifier(problems, `${name}.${column}`, column) + checkIdentifier(problems, `${name}.${column}`, sqlName(column)) const spec: ColumnSpec> = isColumnSpec(input) ? input : { _tag: "PgColumnSpec", type: input as CHType, options: {} } return { kind: "column", table: name, - name: column, + name: sqlName(column), position, type: canonicalPgType(spec.type.sql), notNull: spec.type._tag !== "Nullable", @@ -339,7 +368,7 @@ function buildTable c.name === column)?.notNull === false) { + if (columnEntities.find((c) => c.name === sqlName(column))?.notNull === false) { problems.push({ object: `${name}.${column}`, message: "a primary key column cannot be nullable; Postgres would make it NOT NULL anyway", @@ -369,7 +398,10 @@ function buildTable { - const fkName = spec.name ?? defaultForeignKeyName(name, spec.columns, spec.references, spec.foreignColumns) + const fkColumns = spec.columns.map(sqlName) + // A table referencing itself names its own keys; another table's were mapped by `foreignKey`. + const fkForeignColumns = spec.references === name ? spec.foreignColumns.map(sqlName) : [...spec.foreignColumns] + const fkName = spec.name ?? defaultForeignKeyName(name, fkColumns, spec.references, fkForeignColumns) checkIdentifier(problems, `${name} foreign key`, fkName) for (const column of spec.columns) { if (!has(column)) problems.push({ object: `${name} foreign key ${fkName}`, message: `${column} is not a column` }) @@ -384,9 +416,9 @@ function buildTable c.default !== null || c.identity !== null).map((c) => c.name) + const defaults = inputs + .filter((_, position) => columnEntities[position]!.default !== null || columnEntities[position]!.identity !== null) + .map(([column]) => column) return { _tag: "Table", name, diff --git a/src/sql/literal-binder.ts b/src/sql/literal-binder.ts new file mode 100644 index 0000000..9739632 --- /dev/null +++ b/src/sql/literal-binder.ts @@ -0,0 +1,25 @@ +// Where a compile for a dialect that binds params sends literal values. +// +// A value compared with a typed column is encoded deep inside expression code, +// as a literal. Postgres binds params, and should bind those values too: kept +// out of the statement text, they stay out of logs and traces, and one query +// shape is one statement whatever the values. `compile` installs a binder here +// for the duration of the compile, as it installs the dialect's syntax. + +/** Takes an encoded wire value; returns the SQL that stands for it. */ +export type LiteralBinder = (wire: unknown) => string + +let current: LiteralBinder | undefined + +export function withLiteralBinder(binder: LiteralBinder | undefined, body: () => A): A { + const previous = current + current = binder + try { + return body() + } finally { + current = previous + } +} + +/** The binder installed by the enclosing compile, if there is one. */ +export const activeLiteralBinder = (): LiteralBinder | undefined => current diff --git a/tests/__snapshots__/core-sql.test.ts.snap b/tests/__snapshots__/core-sql.test.ts.snap index c5dbe8c..bd6e9a2 100644 --- a/tests/__snapshots__/core-sql.test.ts.snap +++ b/tests/__snapshots__/core-sql.test.ts.snap @@ -583,6 +583,9 @@ exports[`core SQL (postgres) > and-or 1`] = ` { "parameters": [ "org_1", + "paid", + 15, + "globex", ], "sql": "WITH "orders" AS ( @@ -594,7 +597,7 @@ SELECT "orders"."Id" AS "id" FROM "orders" WHERE "orders"."OrgId" = $1 - AND (("orders"."Status" = 'paid' AND "orders"."Amount" > 15) OR "orders"."Customer" = 'globex') + AND (("orders"."Status" = $2 AND "orders"."Amount" > $3) OR "orders"."Customer" = $4) ORDER BY "id" ASC", } `; @@ -603,6 +606,7 @@ exports[`core SQL (postgres) > arithmetic 1`] = ` { "parameters": [ "org_1", + 4, ], "sql": "WITH "orders" AS ( @@ -618,7 +622,7 @@ SELECT "orders"."Amount" / 2 AS "divided" FROM "orders" WHERE "orders"."OrgId" = $1 - AND "orders"."Id" = 4", + AND "orders"."Id" = $2", } `; @@ -626,6 +630,10 @@ exports[`core SQL (postgres) > comparisons 1`] = ` { "parameters": [ "org_1", + "void", + 5, + 10, + 20, ], "sql": "WITH "orders" AS ( @@ -637,11 +645,11 @@ SELECT "orders"."Id" AS "id" FROM "orders" WHERE "orders"."OrgId" = $1 - AND "orders"."Status" != 'void' - AND "orders"."Amount" > 5 - AND "orders"."Amount" >= 10 - AND "orders"."Amount" < 20 - AND "orders"."Amount" <= 10", + AND "orders"."Status" != $2 + AND "orders"."Amount" > $3 + AND "orders"."Amount" >= $4 + AND "orders"."Amount" < $5 + AND "orders"."Amount" <= $4", } `; @@ -649,6 +657,8 @@ exports[`core SQL (postgres) > cross-join 1`] = ` { "parameters": [ "org_1", + "org_1", + 2, ], "sql": "WITH "orders" AS ( @@ -662,8 +672,8 @@ SELECT FROM "orders" CROSS JOIN "customers" AS "c" WHERE "orders"."OrgId" = $1 - AND "c"."OrgId" = 'org_1' - AND "orders"."Id" <= 2 + AND "c"."OrgId" = $2 + AND "orders"."Id" <= $3 ORDER BY "id" ASC, "customer" ASC", } `; @@ -672,6 +682,7 @@ exports[`core SQL (postgres) > cross-join-subquery 1`] = ` { "parameters": [ "org_1", + 2, ], "sql": "WITH "orders" AS ( @@ -694,7 +705,7 @@ SELECT FROM "customers" WHERE "customers"."OrgId" = $1) AS "k" WHERE "orders"."OrgId" = $1 - AND "orders"."Id" <= 2 + AND "orders"."Id" <= $2 ORDER BY "id" ASC", } `; @@ -703,6 +714,7 @@ exports[`core SQL (postgres) > cte 1`] = ` { "parameters": [ "org_1", + "paid", ], "sql": "WITH "orders" AS ( @@ -722,7 +734,7 @@ SELECT "orders"."Amount" AS "Amount" FROM "orders" WHERE "orders"."OrgId" = $1 - AND "orders"."Status" = 'paid' + AND "orders"."Status" = $2 ) SELECT sum("paid"."Amount") AS "total" @@ -773,6 +785,7 @@ exports[`core SQL (postgres) > from-subquery 1`] = ` { "parameters": [ "org_1", + 6, ], "sql": "SELECT "t"."customer" AS "customer", @@ -790,7 +803,7 @@ SELECT FROM "orders" WHERE "orders"."OrgId" = $1 GROUP BY 1, 2) AS "t" - WHERE "t"."total" > 6 + WHERE "t"."total" > $2 ORDER BY "total" DESC", } `; @@ -798,6 +811,7 @@ SELECT exports[`core SQL (postgres) > group-having 1`] = ` { "parameters": [ + "paid", "org_1", ], "sql": "WITH "orders" AS ( @@ -809,12 +823,12 @@ exports[`core SQL (postgres) > group-having 1`] = ` SELECT "orders"."Customer" AS "customer", count(*) AS "orders", - count(*) FILTER (WHERE "orders"."Status" = 'paid') AS "paid", + count(*) FILTER (WHERE "orders"."Status" = $1) AS "paid", sum("orders"."Amount") AS "total", min("orders"."Amount") AS "smallest", max("orders"."Amount") AS "largest" FROM "orders" - WHERE "orders"."OrgId" = $1 + WHERE "orders"."OrgId" = $2 GROUP BY 1 HAVING count(*) >= 2 ORDER BY "customer" ASC", @@ -866,6 +880,7 @@ exports[`core SQL (postgres) > join-subqueries 1`] = ` { "parameters": [ "org_1", + "silver", ], "sql": "WITH "orders" AS ( @@ -889,7 +904,7 @@ SELECT "customers"."Tier" AS "tier" FROM "customers" WHERE "customers"."OrgId" = $1 - AND "customers"."Tier" != 'silver') AS "t" ON "customers"."Name" = "t"."name" + AND "customers"."Tier" != $2) AS "t" ON "customers"."Name" = "t"."name" LEFT JOIN (WITH "orders" AS ( ), @@ -911,6 +926,7 @@ exports[`core SQL (postgres) > left-join 1`] = ` { "parameters": [ "org_1", + 3, ], "sql": "WITH "orders" AS ( @@ -924,7 +940,7 @@ SELECT FROM "orders" LEFT JOIN "customers" AS "c" ON ("orders"."Customer" = "c"."Name" AND "orders"."OrgId" = "c"."OrgId") WHERE "orders"."OrgId" = $1 - AND "orders"."Id" >= 3 + AND "orders"."Id" >= $2 ORDER BY "id" ASC", } `; @@ -954,6 +970,7 @@ exports[`core SQL (postgres) > locking 1`] = ` { "parameters": [ "org_1", + 1, ], "sql": "WITH "orders" AS ( @@ -965,7 +982,7 @@ SELECT "orders"."Id" AS "id" FROM "orders" WHERE "orders"."OrgId" = $1 - AND "orders"."Id" = 1 + AND "orders"."Id" = $2 FOR UPDATE SKIP LOCKED", } `; @@ -974,7 +991,9 @@ exports[`core SQL (postgres) > null-and-range 1`] = ` { "parameters": [ "org_1", + 5, 20, + 3, ], "sql": "WITH "orders" AS ( @@ -987,8 +1006,8 @@ SELECT FROM "orders" WHERE "orders"."OrgId" = $1 AND "orders"."Note" IS NULL - AND "orders"."Amount" BETWEEN 5 AND $2 - AND "orders"."Id" NOT BETWEEN 3 AND 3 + AND "orders"."Amount" BETWEEN $2 AND $3 + AND "orders"."Id" NOT BETWEEN $4 AND $4 ORDER BY "id" ASC", } `; @@ -997,6 +1016,8 @@ exports[`core SQL (postgres) > nulls-and-literals 1`] = ` { "parameters": [ "org_1", + 2, + 4, ], "sql": "WITH "orders" AS ( @@ -1011,7 +1032,7 @@ SELECT 'x''y' AS "label" FROM "orders" WHERE "orders"."OrgId" = $1 - AND "orders"."Id" IN (2, 4) + AND "orders"."Id" IN ($2, $3) ORDER BY "id" ASC", } `; @@ -1046,6 +1067,12 @@ exports[`core SQL (postgres) > patterns-and-lists 1`] = ` { "parameters": [ "org_1", + "%e%", + "i%", + "PA%", + "open", + 3, + 2, ], "sql": "WITH "orders" AS ( @@ -1057,10 +1084,10 @@ SELECT "orders"."Id" AS "id" FROM "orders" WHERE "orders"."OrgId" = $1 - AND ("orders"."Customer" LIKE '%e%' AND "orders"."Customer" NOT LIKE 'i%') - AND ("orders"."Status" ILIKE 'PA%' OR "orders"."Status" IN ('open')) - AND "orders"."Id" NOT IN (3) - AND NOT ("orders"."Id" = 2) + AND ("orders"."Customer" LIKE $2 AND "orders"."Customer" NOT LIKE $3) + AND ("orders"."Status" ILIKE $4 OR "orders"."Status" IN ($5)) + AND "orders"."Id" NOT IN ($6) + AND NOT ("orders"."Id" = $7) ORDER BY "id" ASC", } `; @@ -1084,6 +1111,7 @@ exports[`core SQL (postgres) > select-columns 1`] = ` { "parameters": [ "org_1", + "paid", ], "sql": "WITH "orders" AS ( @@ -1096,7 +1124,7 @@ SELECT "orders"."Customer" AS "Customer" FROM "orders" WHERE "orders"."OrgId" = $1 - AND "orders"."Status" = 'paid' + AND "orders"."Status" = $2 ORDER BY "Id" ASC", } `; @@ -1106,6 +1134,7 @@ exports[`core SQL (postgres) > selected-param 1`] = ` "parameters": [ true, "org_1", + 1, ], "sql": "WITH "orders" AS ( @@ -1118,7 +1147,7 @@ SELECT $1::boolean AS "flagged" FROM "orders" WHERE "orders"."OrgId" = $2 - AND "orders"."Id" = 1", + AND "orders"."Id" = $3", } `; diff --git a/tests/dialect-cases.postgres.ts b/tests/dialect-cases.postgres.ts index f77aecb..7a7e850 100644 --- a/tests/dialect-cases.postgres.ts +++ b/tests/dialect-cases.postgres.ts @@ -38,6 +38,8 @@ const typed = PG.table("typed", { Exact: PG.custom("int8", Schema.Union([Schema.BigInt, Schema.BigIntFromString])), At: PG.timestamptz, AtText: PG.timestamptz, + AtMillis: PG.timestamptzMillis, + AtMillisText: PG.timestamptzMillis, // Compared only: string-typed, for the dateTimeString and dateTimeSeconds params. AtString: PG.custom("timestamptz", Schema.String), Doc: PG.jsonb(Schema.Struct({ region: Schema.String })), @@ -61,6 +63,8 @@ const typedRow = `SELECT 9007199254740993::int8 AS "Exact", '2026-01-01T00:00:00.25Z'::timestamptz AS "At", '2026-01-01 00:00:00.25+00'::text AS "AtText", + '2026-01-01T00:00:00.25Z'::timestamptz AS "AtMillis", + '2026-01-01 00:00:00.25+00'::text AS "AtMillisText", '2026-01-01T00:00:00.25Z'::timestamptz AS "AtString", '{"region": "eu"}'::jsonb AS "Doc", '12'::text AS "Branded", @@ -173,6 +177,7 @@ export const postgresCases: readonly PostgresCase[] = [ "custom", "brand", "timestamptz", + "timestamptzMillis", "jsonb", "array", "nullable", @@ -194,6 +199,8 @@ export const postgresCases: readonly PostgresCase[] = [ "Exact", "At", "AtText", + "AtMillis", + "AtMillisText", "Doc", "Tags", "Missing", @@ -216,6 +223,8 @@ export const postgresCases: readonly PostgresCase[] = [ Exact: 9007199254740993n, At: utc("2026-01-01T00:00:00.250Z"), AtText: utc("2026-01-01T00:00:00.250Z"), + AtMillis: Date.parse("2026-01-01T00:00:00.250Z"), + AtMillisText: Date.parse("2026-01-01T00:00:00.250Z"), Doc: { region: "eu" }, Tags: ["x", "y"], Missing: null, @@ -249,6 +258,37 @@ export const postgresCases: readonly PostgresCase[] = [ ), expected: [{ region: "eu", absent: null }], }, + { + id: "conditionals-and-typed-values", + covers: pg("greatest", "least", "caseWhen", "asBoolean", "typedValue"), + build: () => + PG.compileUnsafe( + typedRows().select(($) => ({ + high: PG.greatest($.Int4, $.Int8, PG.typedValue(PG.int8, 6)), + low: PG.least($.Int4, $.Int8), + later: PG.greatest($.AtMillis, PG.typedValue(PG.timestamptzMillis, Date.parse("2026-01-02T00:00:00Z"))), + size: PG.caseWhen( + [ + [$.Int8.gt(100), PG.typedValue(PG.text, "large")], + [$.Int8.gt(5), PG.typedValue(PG.text, "medium")], + ], + PG.typedValue(PG.text, "small"), + ), + big: PG.asBoolean($.Int8.gt(5)), + raw: PG.undecoded($.Doc), + })), + ), + expected: [ + { + high: 8, + low: 4, + later: Date.parse("2026-01-02T00:00:00Z"), + size: "medium", + big: true, + raw: { region: "eu" }, + }, + ], + }, { id: "compile-entry-points", covers: pg("compile", "postgresDialect"),