Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .agents/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ flowchart LR

VALUES and SET use the same scalar expression resolver as SELECT. UPDATE supplies the target relation; VALUES supplies only parameter/local bindings. Arithmetic lookup follows only grammar wrappers spanning the complete expression, rather than scanning operand subtrees. ANTLR operator contexts preserve precedence and scalar parentheses are unwrapped only after excluding tuples, lambdas and subqueries. DML values resolve supported comparisons with their nullability. Assignment validation is shared with SELECT-backed DML and permits exact types, optional destinations, contextual NULL and lossless integer widening; it does not insert casts or evaluate SQL. Direct parameter assignment inference remains separate from resolving operands in compound expressions.

IN subqueries reuse the SELECT semantic core with an independent relation scope. Outer column, predicate, inference and aggregate traversals stop at nested SELECT boundaries; shared external parameter types remain consistent across scopes. Tuple keys are resolved only for the supported IN operands and are not exposed as a new generated parameter/result contract. Per-SELECT resolved bindings accompany the original ANTLR contexts so the jOOQ renderer can preserve shadowed aliases without reanalyzing SQL.

`internal/model` is the boundary between analysis and generation: YQL type identity, parameters, result sets and source locations. Nullability is an `Optional` type, compound type metadata is retained, and Struct fields have name-based identity independent of declaration order. `Bytes` normalizes to binary `String`; `Text` normalizes to Unicode `Utf8`. A table catalog and a query projection are distinct: `SELECT name` does not generate the whole table. Projection columns retain their API name and, where different, the exact YDB result name in `WireName`. For example, an unaliased `b.title` in a join is returned as `b.title`. Name-based decoders use `Column.ResultName()`, which selects `WireName` when present and otherwise `Name`; positional decoders retain the analyzed projection order. Generated API fields continue to use `Name`. `AnalyzedQuery.SQL` contains executable YQL with supported wildcard projections expanded into explicit quoted columns; unnamed computed expressions combined with a wildcard receive explicit aliases preserving their original YDB names, so expansion cannot change references such as ORDER BY column1. Declarations and text outside those wildcard replacements and alias insertions are retained. The analyzer records explicitly declared parameter names in `DeclaredParameters`. Generators retain user-written declarations in executable SQL. SDK adapters use this metadata to avoid duplicate declarations while binding inferred parameters with their resolved types. Parameter names omit the leading `$`; their types and result column types must be resolved. TypeScript result properties use `Column.ResultName()` verbatim, including qualified names as quoted properties. No target-specific SQL alias rewriting or result-key conversion is needed for this target. `analyzer.Analyze` returns an error whenever its result contains diagnostics.

The language packages in `internal/codegen` produce files from that resolved model. They handle naming, runtime-specific parameter binding, result decoding and resource lifetimes. They do not analyze SQL or load external code. jOOQ uses typed JDBC execution for structured batches and explicitly declared queries, with resolved table references rendered through jOOQ to preserve table mappings. For the jOOQ DSL target, `AnalyzedQuery.Syntax` retains ANTLR contexts and analyzer-resolved column/table bindings for the executable SQL, including the positions after wildcard expansion. The renderer walks these contexts directly; it does not reparse text or construct a second AST. Unsupported DSL constructs fail in the target without restricting other generators. Lexical adaptation of parameter placeholders for a driver is separate from semantic query analysis and must preserve strings, comments and identifiers. `internal/codegen/jdbc` implements shared SQL rendering for Java and Kotlin; their SDK binding and result decoding remain in each generator.
Expand Down
4 changes: 3 additions & 1 deletion .agents/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ go tool cover -html=coverage.out -o coverage.html

`make coverage` runs the root Go module's tests without caching and writes `coverage.out` in atomic mode. `-coverpkg=./...` includes calls across package boundaries, so CLI and golden tests contribute to analyzer and generator coverage. The final `total` from `go tool cover` is the combined statement coverage; the per-test-package percentages are not independent package coverage figures. The denominator contains only sqlc-ydb runtime code, including the CLI entry point even when untested. Unit and end-to-end tests run and contribute coverage of that code, but their own source is not measured. Examples, the `internal/endtoend` harness, `*_test.go` files, golden outputs, `.github` tooling and the external ANTLR parser dependency are outside this scope. Go already excludes test source and the separate examples module from the profile; `codecov.yml` also explicitly excludes the repository's non-runtime paths. This local profile excludes live YDB tests unless their environment variables are set, and excludes optional SDK checks unless explicitly enabled.

CI saves the offline profile together with `coverage-sdk.out` and `coverage-typescript.out` from compiler/SDK-backed generator tests as the `generator-coverage` artifact and uploads them to Codecov with the `unit` flag on pushes to `main` and pull requests. The `ydb-acceptance` job also instruments the semantic, integer pagination, table-path-prefix, callback, shared-expression, Go generator and Python generator live tests and uploads their seven profiles with the `integration` flag, saving them as the `generator-integration-coverage` artifact. Codecov merges these profiles; generated application runtime coverage is not measured. SDK checks contribute coverage of the generator code they execute. The PR comment updates as reports arrive, so the first report can show only offline coverage. [codecov.yml](../codecov.yml) compares project coverage with the base commit (allowing a one percentage point drop) and requires 80% patch coverage. It enables one updated PR comment with the coverage difference and impacted files, including on the first PR without a base report. A successful `main` upload establishes the comparison baseline and populates the README badge.
CI saves the offline profile together with `coverage-sdk.out` and `coverage-typescript.out` from compiler/SDK-backed generator tests as the `generator-coverage` artifact and uploads them to Codecov with the `unit` flag on pushes to `main` and pull requests. The `ydb-acceptance` job also instruments the semantic, integer pagination, table-path-prefix, callback, shared-expression, IN-subquery, Go generator and Python generator live tests and uploads their eight profiles with the `integration` flag, saving them as the `generator-integration-coverage` artifact. Codecov merges these profiles; generated application runtime coverage is not measured. SDK checks contribute coverage of the generator code they execute. The PR comment updates as reports arrive, so the first report can show only offline coverage. [codecov.yml](../codecov.yml) compares project coverage with the base commit (allowing a one percentage point drop) and requires 80% patch coverage. It enables one updated PR comment with the coverage difference and impacted files, including on the first PR without a base report. A successful `main` upload establishes the comparison baseline and populates the README badge. Codecov reports line coverage, including partially covered lines; its percentages are not directly comparable to Go's statement coverage. Verify the updated Codecov report for the final PR head before claiming that a coverage regression is fixed.

Repository administrators must enable `ydb-platform/sqlc-ydb` in Codecov and grant the [Codecov GitHub App](https://github.com/apps/codecov) access so it can post PR comments. Set the repository Actions secret `CODECOV_TOKEN` to the Codecov upload token, as in ydb-go-sdk. Alternatively, the organization can allow tokenless public uploads with **Global Upload Token → Not required** in Codecov; the action accepts an empty secret in that mode. Public fork PR uploads do not need access to the secret. See [Codecov token authentication](https://docs.codecov.com/docs/codecov-tokens) and [PR comments](https://docs.codecov.com/docs/pull-request-comments). An upload failure fails the CI job rather than silently leaving stale coverage.

Expand Down Expand Up @@ -175,3 +175,5 @@ The [TypeScript](../docs/typescript.md), [Rust](../docs/rust.md) and [PHP](../do
`TestLiveYDBEach` generates the callback fixture and runs both Go profiles sequentially using unique disposable tables. It checks full and empty results, a 16 MiB streaming result, cancellation, early termination, no replay of retryable callback errors, connection reuse, sessions and transactions. Run `YDB_CONNECTION_STRING=grpc://localhost:2136/local go test -p 1 -count=1 -timeout=240s ./internal/endtoend -run '^TestLiveYDBEach$' -v`. Stable and nightly CI run this suite in a separate sequential step with its own 240-second test budget; stable CI includes `coverage-each-live.out` in integration coverage. `TestEachGeneratedGoCompiles` checks the same harness offline; `TestEachRuntime` in the Go generator package covers deterministic error/cleanup paths and retained-memory bounds.

`TestLiveYDBSharedExpressions` checks Boolean values, conditional aggregates, string concatenation, verified casts, literal-aware COALESCE and implicit result names against server metadata and generated Go native/database/sql execution. Wildcard collision checks compare original and normalized SQL metadata independently and verify positional decoding with distinct values and types. Run `YDB_CONNECTION_STRING=grpc://localhost:2136/local go test -p 1 -count=1 -timeout=240s ./internal/endtoend -run '^TestLiveYDBSharedExpressions$' -v` sequentially with the other live suites; set `SQLC_YDB_TEST_MAVEN=mvn` to include generated jOOQ execution for output-alias ordering and wildcard collisions. Stable and nightly CI run the suite in a separate step; stable CI enables jOOQ and uploads `coverage-shared-expressions-live.out` with the integration coverage. The authors examples additionally execute reporting, prefix search and export metadata through the generated clients.

`TestLiveYDBInSubqueries` checks noncorrelated scalar and tuple-key IN/NOT IN subqueries against YDB and both generated Go profiles. It covers single-column subquery contracts, full composite keys, empty inputs, nullable operands, alias scopes and mutations. Run `YDB_CONNECTION_STRING=grpc://localhost:2136/local go test -p 1 -count=1 -timeout=240s ./internal/endtoend -run '^TestLiveYDBInSubqueries$' -v` sequentially with the other live suites; set `SQLC_YDB_TEST_MAVEN=mvn` to include jOOQ execution and table mappings. Stable and nightly CI run the suite in its own step; stable CI enables jOOQ and uploads `coverage-in-subqueries-live.out`. `TestInSubqueriesGeneratedGoCompiles` checks the same generated clients without a database. See [YQL evidence](yql-evidence.md) for the pinned-server contracts and any observed server limitations.
4 changes: 3 additions & 1 deletion .agents/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,13 @@ Current behavior is in [compatibility](../docs/compatibility.md), stage ownershi

The following issues track planned work; their examples and acceptance criteria define the scope. Name-mapped INSERT/UPSERT SELECT, trailing Struct commas, verified integer aliases and compatible LIMIT/OFFSET types, secondary-index metadata with VIEW selection, and static absolute TablePathPrefix resolution are implemented; their original scope is in [#25](https://github.com/ydb-platform/sqlc-ydb/issues/25), [#26](https://github.com/ydb-platform/sqlc-ydb/issues/26), [#27](https://github.com/ydb-platform/sqlc-ydb/issues/27), [#28](https://github.com/ydb-platform/sqlc-ydb/issues/28), and [#29](https://github.com/ydb-platform/sqlc-ydb/issues/29). The [compatibility contract](../docs/compatibility.md#current-analyzer-coverage) records their boundaries. Subsequent changes should remain separate, reviewable PRs.

The noncorrelated scalar/tuple IN subquery portion of [#32](https://github.com/ydb-platform/sqlc-ydb/issues/32) is implemented; its remaining scope is listed below.

| Order | Planned capability | Tracking |
| --- | --- | --- |
| 2 | ALTER COLUMN DROP NOT NULL | [#30](https://github.com/ydb-platform/sqlc-ydb/issues/30) |
| 3 | Shared Boolean expressions, conditional aggregates and scalar conversions | [#31](https://github.com/ydb-platform/sqlc-ydb/issues/31) |
| 4 | Scoped tabular expressions, IN subqueries and collection aggregation | [#32](https://github.com/ydb-platform/sqlc-ydb/issues/32) |
| 4 | Derived FROM/JOIN sources, named SELECT bindings and collection aggregation | [#32](https://github.com/ydb-platform/sqlc-ydb/issues/32) |
| 5 | Multi-statement query scripts with at most one typed result | [#33](https://github.com/ydb-platform/sqlc-ydb/issues/33) |
| 6 | Typed lambdas and JSON/Yson collection transformations | [#34](https://github.com/ydb-platform/sqlc-ydb/issues/34) |
| Independent | Typed streaming results with explicit cancellation and ownership | [#35](https://github.com/ydb-platform/sqlc-ydb/issues/35) |
Expand Down
4 changes: 4 additions & 0 deletions .agents/sdk-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,3 +160,7 @@ Direct-session validation also observed `SESSION_BUSY` (`Pending previous query
Native early-stop cleanup preserves the callback error but can also join `context.Canceled` returned by the pinned SDK's `Close`, depending on whether asynchronous stream cancellation has completed. `TestEarlyStopCleanupCancellation` in the generated runtime harness covers both nil and canceled close results, checks preservation of the callback sentinel and verifies that the caller's original context remains active. Cancellation-before-close is retained to avoid draining an unfinished stream; no extra runtime error-classification policy is introduced. Decimal validation on `:each` uses its existing deferred query-boundary stack wrapper once; `TestEachDecimalValidation` checks exactly one generated query frame.

The public `examples/streaming` recipe was generated, compiled and executed on 2026-09-23 against the same Docker image and pinned Go SDK. `TestCallbackExports` in `tests/examples/go/streaming` passed for native client/session/transaction and database/sql client/transaction, checking ordered NDJSON with nullable and Unicode fields, empty results, no-parameter queries and one-call termination on writer failure. `TestLiveYDBEach` passed again after the review changes; both live suites ran sequentially.

## jOOQ IN subquery projection contract (2026-09-23)

A compiled probe against published jOOQ 3.21.0, YDB dialect 2.0.0 and JDBC 2.4.1 confirmed that `select(row(id, value))` renders a `row (...)` expression. This is not YQL's single tuple projection `SELECT (id, value)`, and selecting two separate fields also changes the IN subquery contract. The generator therefore supports scalar IN subqueries through typed DSL and rejects tuple-key subqueries in that path with an explicit alternative. Queries with authored DECLARE statements continue to use their existing typed JDBC contract, preserving the tuple SQL and analyzer-resolved table mappings. No new SQL fallback or runtime dependency was introduced.
Loading
Loading