From 99abd73567aae3aab18da8c21b17788f1301ed2b Mon Sep 17 00:00:00 2001 From: eric-wang-1990 Date: Wed, 22 Jul 2026 19:01:20 -0700 Subject: [PATCH 01/12] docs: add connection parameter reference (Thrift vs SEA/Kernel) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents every public ConnectionOptions / OpenSessionRequest / per-statement parameter and whether it is honored on the Thrift backend, the SEA/Kernel backend (useKernel), or both — making protocol support gaps explicit. Mirrors the databricks-sql-kernel connection-parameters reference format. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 --- docs/connection-parameters.md | 176 ++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 docs/connection-parameters.md diff --git a/docs/connection-parameters.md b/docs/connection-parameters.md new file mode 100644 index 00000000..f3386e55 --- /dev/null +++ b/docs/connection-parameters.md @@ -0,0 +1,176 @@ +# Connection parameter reference + +This document lists **every public connection / session parameter** the +Node.js connector accepts, and — because the driver ships two backends — +whether each parameter is honored on the **Thrift** backend (the default), +the **SEA / Kernel** backend (opt-in via `useKernel: true`), or both. + +The goal is to make protocol gaps explicit: a parameter honored on one +backend but ignored (or rejected) on the other is called out in the **Gap** +column. + +> **Backend selection.** The connector defaults to Thrift. The SEA backend is +> selected by passing `useKernel: true`, an **internal, unstable (M0)** option +> that is intentionally absent from the published `.d.ts` and may be removed +> without notice (`lib/contracts/InternalConnectionOptions.ts`). Treat every +> "Kernel" column below as describing an experimental path. + +## Legend + +| Symbol | Meaning | +|--------|---------| +| ✅ | Honored — the option is read and forwarded to the backend. | +| ❌ | Ignored or rejected — see the Gap column. | +| ⚠️ | Partially supported or behaves differently from the other backend. | +| — | Not applicable / no public equivalent on this backend. | + +## Sources of truth + +- Public option shape ← `lib/contracts/IDBSQLClient.ts` (`ConnectionOptions`, + `AuthOptions`, `OpenSessionRequest`). +- Internal / kernel-only flags ← `lib/contracts/InternalConnectionOptions.ts`. +- Thrift wiring ← `lib/DBSQLClient.ts` (`getConnectionOptions`, + `createAuthProvider`), `lib/thrift-backend/ThriftBackend.ts`, + `lib/connection/connections/HttpConnection.ts`. +- Kernel wiring ← `lib/kernel/KernelAuth.ts` (`buildKernelConnectionOptions`, + `buildKernelTlsOptions`, `buildKernelHttpOptions`, `buildKernelProxyOptions`, + `buildKernelRetryOptions`), `lib/kernel/KernelBackend.ts`, + `lib/kernel/KernelSessionBackend.ts`. +- Kernel-core parameter semantics ← databricks-sql-kernel + [`docs/connection-parameters.md`](https://github.com/databricks/databricks-sql-kernel/pull/184). + +--- + +## Connection identity + +| Option | Type | Thrift | Kernel | Gap | +|--------|------|:------:|:------:|-----| +| `host` | `string` | ✅ | ✅ | Required on both. | +| `path` | `string` | ✅ | ✅ | HTTP path; on the kernel path the org id is auto-parsed from a `?o=` query param and sent as `x-databricks-org-id`. | +| `port` | `number` | ✅ | ⚠️ | Thrift defaults to `443`. The kernel derives host/port from `host` + `path`; a standalone `port` is not separately threaded. | +| `userAgentEntry` | `string` | ✅ | ✅ | Folded into the composed `User-Agent` on both. | + +## Authentication + +| Option | Thrift | Kernel | Gap | +|--------|:------:|:------:|-----| +| `authType: 'access-token'` + `token` (PAT) | ✅ | ✅ | | +| `authType: 'databricks-oauth'` — M2M (`oauthClientId` + `oauthClientSecret`) | ✅ | ✅ | Kernel runs OIDC discovery + client-credentials internally. | +| `authType: 'databricks-oauth'` — U2M (browser) | ✅ | ⚠️ | See U2M gaps below. | +| `oauthScopes` | ✅ | ✅ | Kernel default U2M scopes = `['sql','offline_access']` (Thrift parity); M2M = `['all-apis']`. | +| `oauthClientId` (U2M) | ✅ | ❌ | **Kernel-side gap.** Kernel U2M hardcodes `client_id` and **rejects** a custom `oauthClientId`; Thrift honors it. | +| `oauthClientId` + no secret | ✅ (U2M) | ❌ | **Divergence.** Thrift routes to U2M with that id; kernel keys the flow off `oauthClientSecret` presence and throws an M2M "secret required" error. | +| `azureTenantId` / `useDatabricksOAuthInAzure` | ✅ | ❌ | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | +| `persistence` (custom OAuth token store) | ✅ | ❌ | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | +| `authType: 'custom'` (`provider`) | ✅ | ❌ | **Thrift-only.** Kernel supports only `access-token` and `databricks-oauth`. | +| `authType: 'token-provider'` (`tokenProvider`) | ✅ | ❌ | **Thrift-only.** | +| `authType: 'external-token'` (`getToken`) | ✅ | ❌ | **Thrift-only.** | +| `authType: 'static-token'` (`staticToken`) | ✅ | ❌ | **Thrift-only.** | +| `enableTokenFederation` / `federationClientId` | ✅ | ❌ | **Thrift-only** (available on the token-provider / external-token / static-token arms, none of which the kernel supports). | + +## HTTP client, proxy, retries + +| Option | Thrift | Kernel | Gap | +|--------|:------:|:------:|-----| +| `socketTimeout` | ✅ | ⚠️ | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | +| `proxy` (`{protocol, host, port, auth}`) | ✅ | ⚠️ | Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | +| `noProxy` (internal) | ✅ | ✅ | Forwarded to the kernel as `bypassHosts`. | +| `customHeaders` | ✅ | ✅ | Kernel drops reserved `Authorization` / `x-databricks-org-id`, rejects CR/LF/NUL, and appends the connector `User-Agent` last. | +| `retryMaxAttempts` | ✅ | ✅ | Total-attempt semantics on both; kernel converts to retries-after-first. | +| `retriesTimeout` | ✅ | ✅ | Kernel converts ms → whole seconds. | +| `retryDelayMin` | ✅ | ✅ | Kernel converts ms → seconds. | +| `retryDelayMax` | ✅ | ✅ | Kernel converts ms → seconds. | +| `maxConnections` (pool size) | ❌ | ✅ | **Kernel-only** (`InternalConnectionOptions`). Thrift has no connection pool. | + +## TLS / SSL + +> **Important Thrift caveat.** The public `connect()` surface on the Thrift +> backend does **not** expose any TLS customization. `getConnectionOptions` +> (`lib/DBSQLClient.ts`) maps only `host` / `port` / `path` / `socketTimeout` / +> `proxy` / `User-Agent`; the internal `IConnectionOptions.ca/cert/key` fields +> are never populated from a public option, and `HttpConnection` hardcodes +> `rejectUnauthorized: false`. All TLS-verification and custom-CA / mTLS +> controls below are therefore **kernel-only** in practice. + +| Option | Thrift | Kernel | Gap | +|--------|:------:|:------:|-----| +| `checkServerCertificate` | ❌ | ✅ | **Kernel-only.** Master verify toggle (secure-by-default; set `false` for accept-anything). | +| `checkServerCertificateHostname` | ❌ | ✅ | **Kernel-only.** Independent hostname-vs-SNI check; no-op when `checkServerCertificate: false`. | +| `customCaCert` (PEM string / `Buffer`) | ❌ | ✅ | **Kernel-only.** Added on top of system roots. | +| `clientCertPem` (mTLS) | ❌ | ✅ | **Kernel-only.** Must be paired with `clientKeyPem`; supplying one alone is rejected. | +| `clientKeyPem` (mTLS) | ❌ | ✅ | **Kernel-only.** PKCS#8 recommended. | + +## Results & type rendering + +| Option | Thrift | Kernel | Gap | +|--------|:------:|:------:|-----| +| `preserveBigNumericPrecision` | ✅ | ✅ | DECIMAL → exact string, BIGINT → `bigint` on both. | +| `enableMetricViewMetadata` | ✅ | ❌ | **Thrift-only in the connector.** Thrift auto-injects `spark.sql.thriftserver.metadata.metricview.enabled=true` (`ThriftBackend.ts`); `KernelBackend` does **not** auto-inject it. The kernel core *can* accept the raw conf key via `session_conf`, so a caller could pass it manually in `OpenSessionRequest.configuration`. | + +## Session defaults (`openSession(request)`) + +| Option | Thrift | Kernel | Gap | +|--------|:------:|:------:|-----| +| `initialCatalog` | ✅ | ✅ | Kernel → `DefaultOpts.catalog` on `CreateSession`. | +| `initialSchema` | ✅ | ✅ | Kernel → `DefaultOpts.schema`. | +| `configuration` (session confs) | ✅ | ⚠️ | Kernel matches keys **case-insensitively against an allowlist** and uppercases them; **non-allowlisted keys are dropped with a warning**. Thrift forwards the map more freely. | +| `queryTags` | ✅ | ✅ | Both serialize into the reserved `QUERY_TAGS` conf; `queryTags` takes precedence over `configuration.QUERY_TAGS`. | + +## Telemetry + +All `telemetry*` options (`telemetryEnabled`, `telemetryBatchSize`, +`telemetryFlushIntervalMs`, `telemetryMaxRetries`, +`telemetryAuthenticatedExport`, `telemetryCircuitBreakerThreshold`, +`telemetryCircuitBreakerTimeout`, `telemetryCloseTimeoutMs`, +`telemetryMaxStatementMetrics`, `telemetryMaxPendingMetrics`) live in the +driver-layer `ClientConfig`, not in either backend, so they are read +regardless of `useKernel`. + +| Aspect | Thrift | Kernel | Gap | +|--------|:------:|:------:|-----| +| Telemetry config knobs | ✅ | ✅ | Backend-agnostic (driver layer). | +| Statement / CloudFetch telemetry *events* | ✅ | ⚠️ | The kernel owns result fetching internally, so it emits fewer per-statement / cloud-fetch events than the Thrift path. | + +## Per-statement options (`session.executeStatement(sql, options)`) + +| Option | Thrift | Kernel | Gap | +|--------|:------:|:------:|-----| +| `maxRows` | ✅ | ✅ | Kernel applies it at fetch time in the facade rather than on the request. | +| `useCloudFetch` | ✅ | ❌ | **Thrift-only.** Kernel ignores it (logs a no-op warning); CloudFetch is governed by the kernel's result configuration, not per-statement. | +| `useLZ4Compression` | ✅ | ❌ | **Thrift-only.** Kernel ignores it (no-op warning); the kernel auto-detects and decompresses `LZ4_FRAME` from the server result manifest. | +| `stagingAllowedLocalPath` (volume ops) | ✅ | ❌ | **Thrift-only.** Not supported on the kernel path. | +| `runAsync` | ✅ | ⚠️ | Deprecated; not threaded on the kernel path. | + +--- + +## Summary of gaps + +### Supported on Thrift, missing / ignored on Kernel + +1. `enableMetricViewMetadata` — no auto-injection on the kernel path. +2. Auth types `custom`, `token-provider`, `external-token`, `static-token`, + plus `enableTokenFederation` / `federationClientId`. +3. `azureTenantId` / `useDatabricksOAuthInAzure` (Azure-direct OAuth). +4. `persistence` (custom OAuth token store). +5. Custom `oauthClientId` on the U2M flow (and `oauthClientId` + no secret). +6. SOCKS proxies. +7. Per-statement `useCloudFetch`, `useLZ4Compression`, + `stagingAllowedLocalPath`. + +### Supported on Kernel, no Thrift public equivalent + +1. `maxConnections` (connection-pool sizing). +2. TLS controls: `checkServerCertificate`, `checkServerCertificateHostname`, + `customCaCert`, `clientCertPem`, `clientKeyPem`. The Thrift backend exposes + **no** public TLS options and hardcodes `rejectUnauthorized: false`. + +### Behavioral divergences to watch + +- **U2M flow selection** keys off `oauthClientSecret` presence on the kernel + path but is honored differently on Thrift. +- **`socketTimeout: 0`** means "indefinite" on Thrift but is dropped on the + kernel path (kernel default kept). +- **`configuration`** is allowlist-filtered on the kernel path but forwarded + more freely on Thrift. + +> All kernel-path behavior reflects the **M0 stub** and is subject to change. From 789c465e730c944452bf735c37011f378cc02b42 Mon Sep 17 00:00:00 2001 From: eric-wang-1990 Date: Wed, 22 Jul 2026 19:23:50 -0700 Subject: [PATCH 02/12] docs: move connection param reference to root, link from README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Relocate docs/connection-parameters.md → CONNECTION_PARAMETERS.md at the repo root and add a README "Configuration" section linking to it, mirroring the existing Telemetry section's summary+link pattern. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 --- docs/connection-parameters.md => CONNECTION_PARAMETERS.md | 0 README.md | 6 ++++++ 2 files changed, 6 insertions(+) rename docs/connection-parameters.md => CONNECTION_PARAMETERS.md (100%) diff --git a/docs/connection-parameters.md b/CONNECTION_PARAMETERS.md similarity index 100% rename from docs/connection-parameters.md rename to CONNECTION_PARAMETERS.md diff --git a/README.md b/README.md index 5d31e138..0eab0fab 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,12 @@ client }); ``` +## Configuration + +See [CONNECTION_PARAMETERS.md](CONNECTION_PARAMETERS.md) for every connection, +session, and per-statement parameter the driver accepts, and whether each one +applies to the Thrift backend (default), the SEA backend, or both. + ## Telemetry The driver emits connection, statement, and CloudFetch metrics plus From 0836b8af331f3ea63f630e9a4d026c8029b76ca1 Mon Sep 17 00:00:00 2001 From: eric-wang-1990 Date: Mon, 27 Jul 2026 22:38:06 -0700 Subject: [PATCH 03/12] docs: apply prettier formatting to connection parameters table Aligns markdown table columns to satisfy the repo prettier check (prettier . --check). Whitespace-only; no content changes. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 --- CONNECTION_PARAMETERS.md | 132 +++++++++++++++++++-------------------- 1 file changed, 66 insertions(+), 66 deletions(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index f3386e55..4c521b08 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -17,12 +17,12 @@ column. ## Legend -| Symbol | Meaning | -|--------|---------| -| ✅ | Honored — the option is read and forwarded to the backend. | -| ❌ | Ignored or rejected — see the Gap column. | -| ⚠️ | Partially supported or behaves differently from the other backend. | -| — | Not applicable / no public equivalent on this backend. | +| Symbol | Meaning | +| ------ | ------------------------------------------------------------------ | +| ✅ | Honored — the option is read and forwarded to the backend. | +| ❌ | Ignored or rejected — see the Gap column. | +| ⚠️ | Partially supported or behaves differently from the other backend. | +| — | Not applicable / no public equivalent on this backend. | ## Sources of truth @@ -43,44 +43,44 @@ column. ## Connection identity -| Option | Type | Thrift | Kernel | Gap | -|--------|------|:------:|:------:|-----| -| `host` | `string` | ✅ | ✅ | Required on both. | -| `path` | `string` | ✅ | ✅ | HTTP path; on the kernel path the org id is auto-parsed from a `?o=` query param and sent as `x-databricks-org-id`. | -| `port` | `number` | ✅ | ⚠️ | Thrift defaults to `443`. The kernel derives host/port from `host` + `path`; a standalone `port` is not separately threaded. | -| `userAgentEntry` | `string` | ✅ | ✅ | Folded into the composed `User-Agent` on both. | +| Option | Type | Thrift | Kernel | Gap | +| ---------------- | -------- | :----: | :----: | ---------------------------------------------------------------------------------------------------------------------------- | +| `host` | `string` | ✅ | ✅ | Required on both. | +| `path` | `string` | ✅ | ✅ | HTTP path; on the kernel path the org id is auto-parsed from a `?o=` query param and sent as `x-databricks-org-id`. | +| `port` | `number` | ✅ | ⚠️ | Thrift defaults to `443`. The kernel derives host/port from `host` + `path`; a standalone `port` is not separately threaded. | +| `userAgentEntry` | `string` | ✅ | ✅ | Folded into the composed `User-Agent` on both. | ## Authentication -| Option | Thrift | Kernel | Gap | -|--------|:------:|:------:|-----| -| `authType: 'access-token'` + `token` (PAT) | ✅ | ✅ | | -| `authType: 'databricks-oauth'` — M2M (`oauthClientId` + `oauthClientSecret`) | ✅ | ✅ | Kernel runs OIDC discovery + client-credentials internally. | -| `authType: 'databricks-oauth'` — U2M (browser) | ✅ | ⚠️ | See U2M gaps below. | -| `oauthScopes` | ✅ | ✅ | Kernel default U2M scopes = `['sql','offline_access']` (Thrift parity); M2M = `['all-apis']`. | -| `oauthClientId` (U2M) | ✅ | ❌ | **Kernel-side gap.** Kernel U2M hardcodes `client_id` and **rejects** a custom `oauthClientId`; Thrift honors it. | -| `oauthClientId` + no secret | ✅ (U2M) | ❌ | **Divergence.** Thrift routes to U2M with that id; kernel keys the flow off `oauthClientSecret` presence and throws an M2M "secret required" error. | -| `azureTenantId` / `useDatabricksOAuthInAzure` | ✅ | ❌ | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | -| `persistence` (custom OAuth token store) | ✅ | ❌ | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | -| `authType: 'custom'` (`provider`) | ✅ | ❌ | **Thrift-only.** Kernel supports only `access-token` and `databricks-oauth`. | -| `authType: 'token-provider'` (`tokenProvider`) | ✅ | ❌ | **Thrift-only.** | -| `authType: 'external-token'` (`getToken`) | ✅ | ❌ | **Thrift-only.** | -| `authType: 'static-token'` (`staticToken`) | ✅ | ❌ | **Thrift-only.** | -| `enableTokenFederation` / `federationClientId` | ✅ | ❌ | **Thrift-only** (available on the token-provider / external-token / static-token arms, none of which the kernel supports). | +| Option | Thrift | Kernel | Gap | +| ---------------------------------------------------------------------------- | :------: | :----: | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `authType: 'access-token'` + `token` (PAT) | ✅ | ✅ | | +| `authType: 'databricks-oauth'` — M2M (`oauthClientId` + `oauthClientSecret`) | ✅ | ✅ | Kernel runs OIDC discovery + client-credentials internally. | +| `authType: 'databricks-oauth'` — U2M (browser) | ✅ | ⚠️ | See U2M gaps below. | +| `oauthScopes` | ✅ | ✅ | Kernel default U2M scopes = `['sql','offline_access']` (Thrift parity); M2M = `['all-apis']`. | +| `oauthClientId` (U2M) | ✅ | ❌ | **Kernel-side gap.** Kernel U2M hardcodes `client_id` and **rejects** a custom `oauthClientId`; Thrift honors it. | +| `oauthClientId` + no secret | ✅ (U2M) | ❌ | **Divergence.** Thrift routes to U2M with that id; kernel keys the flow off `oauthClientSecret` presence and throws an M2M "secret required" error. | +| `azureTenantId` / `useDatabricksOAuthInAzure` | ✅ | ❌ | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | +| `persistence` (custom OAuth token store) | ✅ | ❌ | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | +| `authType: 'custom'` (`provider`) | ✅ | ❌ | **Thrift-only.** Kernel supports only `access-token` and `databricks-oauth`. | +| `authType: 'token-provider'` (`tokenProvider`) | ✅ | ❌ | **Thrift-only.** | +| `authType: 'external-token'` (`getToken`) | ✅ | ❌ | **Thrift-only.** | +| `authType: 'static-token'` (`staticToken`) | ✅ | ❌ | **Thrift-only.** | +| `enableTokenFederation` / `federationClientId` | ✅ | ❌ | **Thrift-only** (available on the token-provider / external-token / static-token arms, none of which the kernel supports). | ## HTTP client, proxy, retries -| Option | Thrift | Kernel | Gap | -|--------|:------:|:------:|-----| -| `socketTimeout` | ✅ | ⚠️ | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | -| `proxy` (`{protocol, host, port, auth}`) | ✅ | ⚠️ | Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | -| `noProxy` (internal) | ✅ | ✅ | Forwarded to the kernel as `bypassHosts`. | -| `customHeaders` | ✅ | ✅ | Kernel drops reserved `Authorization` / `x-databricks-org-id`, rejects CR/LF/NUL, and appends the connector `User-Agent` last. | -| `retryMaxAttempts` | ✅ | ✅ | Total-attempt semantics on both; kernel converts to retries-after-first. | -| `retriesTimeout` | ✅ | ✅ | Kernel converts ms → whole seconds. | -| `retryDelayMin` | ✅ | ✅ | Kernel converts ms → seconds. | -| `retryDelayMax` | ✅ | ✅ | Kernel converts ms → seconds. | -| `maxConnections` (pool size) | ❌ | ✅ | **Kernel-only** (`InternalConnectionOptions`). Thrift has no connection pool. | +| Option | Thrift | Kernel | Gap | +| ---------------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `socketTimeout` | ✅ | ⚠️ | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | +| `proxy` (`{protocol, host, port, auth}`) | ✅ | ⚠️ | Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | +| `noProxy` (internal) | ✅ | ✅ | Forwarded to the kernel as `bypassHosts`. | +| `customHeaders` | ✅ | ✅ | Kernel drops reserved `Authorization` / `x-databricks-org-id`, rejects CR/LF/NUL, and appends the connector `User-Agent` last. | +| `retryMaxAttempts` | ✅ | ✅ | Total-attempt semantics on both; kernel converts to retries-after-first. | +| `retriesTimeout` | ✅ | ✅ | Kernel converts ms → whole seconds. | +| `retryDelayMin` | ✅ | ✅ | Kernel converts ms → seconds. | +| `retryDelayMax` | ✅ | ✅ | Kernel converts ms → seconds. | +| `maxConnections` (pool size) | ❌ | ✅ | **Kernel-only** (`InternalConnectionOptions`). Thrift has no connection pool. | ## TLS / SSL @@ -92,29 +92,29 @@ column. > `rejectUnauthorized: false`. All TLS-verification and custom-CA / mTLS > controls below are therefore **kernel-only** in practice. -| Option | Thrift | Kernel | Gap | -|--------|:------:|:------:|-----| -| `checkServerCertificate` | ❌ | ✅ | **Kernel-only.** Master verify toggle (secure-by-default; set `false` for accept-anything). | -| `checkServerCertificateHostname` | ❌ | ✅ | **Kernel-only.** Independent hostname-vs-SNI check; no-op when `checkServerCertificate: false`. | -| `customCaCert` (PEM string / `Buffer`) | ❌ | ✅ | **Kernel-only.** Added on top of system roots. | -| `clientCertPem` (mTLS) | ❌ | ✅ | **Kernel-only.** Must be paired with `clientKeyPem`; supplying one alone is rejected. | -| `clientKeyPem` (mTLS) | ❌ | ✅ | **Kernel-only.** PKCS#8 recommended. | +| Option | Thrift | Kernel | Gap | +| -------------------------------------- | :----: | :----: | ----------------------------------------------------------------------------------------------- | +| `checkServerCertificate` | ❌ | ✅ | **Kernel-only.** Master verify toggle (secure-by-default; set `false` for accept-anything). | +| `checkServerCertificateHostname` | ❌ | ✅ | **Kernel-only.** Independent hostname-vs-SNI check; no-op when `checkServerCertificate: false`. | +| `customCaCert` (PEM string / `Buffer`) | ❌ | ✅ | **Kernel-only.** Added on top of system roots. | +| `clientCertPem` (mTLS) | ❌ | ✅ | **Kernel-only.** Must be paired with `clientKeyPem`; supplying one alone is rejected. | +| `clientKeyPem` (mTLS) | ❌ | ✅ | **Kernel-only.** PKCS#8 recommended. | ## Results & type rendering -| Option | Thrift | Kernel | Gap | -|--------|:------:|:------:|-----| -| `preserveBigNumericPrecision` | ✅ | ✅ | DECIMAL → exact string, BIGINT → `bigint` on both. | -| `enableMetricViewMetadata` | ✅ | ❌ | **Thrift-only in the connector.** Thrift auto-injects `spark.sql.thriftserver.metadata.metricview.enabled=true` (`ThriftBackend.ts`); `KernelBackend` does **not** auto-inject it. The kernel core *can* accept the raw conf key via `session_conf`, so a caller could pass it manually in `OpenSessionRequest.configuration`. | +| Option | Thrift | Kernel | Gap | +| ----------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `preserveBigNumericPrecision` | ✅ | ✅ | DECIMAL → exact string, BIGINT → `bigint` on both. | +| `enableMetricViewMetadata` | ✅ | ❌ | **Thrift-only in the connector.** Thrift auto-injects `spark.sql.thriftserver.metadata.metricview.enabled=true` (`ThriftBackend.ts`); `KernelBackend` does **not** auto-inject it. The kernel core _can_ accept the raw conf key via `session_conf`, so a caller could pass it manually in `OpenSessionRequest.configuration`. | ## Session defaults (`openSession(request)`) -| Option | Thrift | Kernel | Gap | -|--------|:------:|:------:|-----| -| `initialCatalog` | ✅ | ✅ | Kernel → `DefaultOpts.catalog` on `CreateSession`. | -| `initialSchema` | ✅ | ✅ | Kernel → `DefaultOpts.schema`. | -| `configuration` (session confs) | ✅ | ⚠️ | Kernel matches keys **case-insensitively against an allowlist** and uppercases them; **non-allowlisted keys are dropped with a warning**. Thrift forwards the map more freely. | -| `queryTags` | ✅ | ✅ | Both serialize into the reserved `QUERY_TAGS` conf; `queryTags` takes precedence over `configuration.QUERY_TAGS`. | +| Option | Thrift | Kernel | Gap | +| ------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `initialCatalog` | ✅ | ✅ | Kernel → `DefaultOpts.catalog` on `CreateSession`. | +| `initialSchema` | ✅ | ✅ | Kernel → `DefaultOpts.schema`. | +| `configuration` (session confs) | ✅ | ⚠️ | Kernel matches keys **case-insensitively against an allowlist** and uppercases them; **non-allowlisted keys are dropped with a warning**. Thrift forwards the map more freely. | +| `queryTags` | ✅ | ✅ | Both serialize into the reserved `QUERY_TAGS` conf; `queryTags` takes precedence over `configuration.QUERY_TAGS`. | ## Telemetry @@ -126,20 +126,20 @@ All `telemetry*` options (`telemetryEnabled`, `telemetryBatchSize`, driver-layer `ClientConfig`, not in either backend, so they are read regardless of `useKernel`. -| Aspect | Thrift | Kernel | Gap | -|--------|:------:|:------:|-----| -| Telemetry config knobs | ✅ | ✅ | Backend-agnostic (driver layer). | -| Statement / CloudFetch telemetry *events* | ✅ | ⚠️ | The kernel owns result fetching internally, so it emits fewer per-statement / cloud-fetch events than the Thrift path. | +| Aspect | Thrift | Kernel | Gap | +| ----------------------------------------- | :----: | :----: | ---------------------------------------------------------------------------------------------------------------------- | +| Telemetry config knobs | ✅ | ✅ | Backend-agnostic (driver layer). | +| Statement / CloudFetch telemetry _events_ | ✅ | ⚠️ | The kernel owns result fetching internally, so it emits fewer per-statement / cloud-fetch events than the Thrift path. | ## Per-statement options (`session.executeStatement(sql, options)`) -| Option | Thrift | Kernel | Gap | -|--------|:------:|:------:|-----| -| `maxRows` | ✅ | ✅ | Kernel applies it at fetch time in the facade rather than on the request. | -| `useCloudFetch` | ✅ | ❌ | **Thrift-only.** Kernel ignores it (logs a no-op warning); CloudFetch is governed by the kernel's result configuration, not per-statement. | -| `useLZ4Compression` | ✅ | ❌ | **Thrift-only.** Kernel ignores it (no-op warning); the kernel auto-detects and decompresses `LZ4_FRAME` from the server result manifest. | -| `stagingAllowedLocalPath` (volume ops) | ✅ | ❌ | **Thrift-only.** Not supported on the kernel path. | -| `runAsync` | ✅ | ⚠️ | Deprecated; not threaded on the kernel path. | +| Option | Thrift | Kernel | Gap | +| -------------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------ | +| `maxRows` | ✅ | ✅ | Kernel applies it at fetch time in the facade rather than on the request. | +| `useCloudFetch` | ✅ | ❌ | **Thrift-only.** Kernel ignores it (logs a no-op warning); CloudFetch is governed by the kernel's result configuration, not per-statement. | +| `useLZ4Compression` | ✅ | ❌ | **Thrift-only.** Kernel ignores it (no-op warning); the kernel auto-detects and decompresses `LZ4_FRAME` from the server result manifest. | +| `stagingAllowedLocalPath` (volume ops) | ✅ | ❌ | **Thrift-only.** Not supported on the kernel path. | +| `runAsync` | ✅ | ⚠️ | Deprecated; not threaded on the kernel path. | --- From 4856e49247c06e524f985d78d95ee5935606f120 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Tue, 28 Jul 2026 05:51:53 +0000 Subject: [PATCH 04/12] ai: apply changes for #457 (3 review threads) Addresses: - #3663057241 at CONNECTION_PARAMETERS.md:60 - #3663100673 at CONNECTION_PARAMETERS.md:108 - #3663100689 at CONNECTION_PARAMETERS.md:62 Signed-off-by: peco-engineer-bot[bot] --- CONNECTION_PARAMETERS.md | 21 ++++++++++++--------- lib/kernel/KernelAuth.ts | 23 ++++++++++------------- lib/kernel/KernelBackend.ts | 3 ++- 3 files changed, 24 insertions(+), 23 deletions(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 4c521b08..dcc925c1 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -57,9 +57,9 @@ column. | `authType: 'access-token'` + `token` (PAT) | ✅ | ✅ | | | `authType: 'databricks-oauth'` — M2M (`oauthClientId` + `oauthClientSecret`) | ✅ | ✅ | Kernel runs OIDC discovery + client-credentials internally. | | `authType: 'databricks-oauth'` — U2M (browser) | ✅ | ⚠️ | See U2M gaps below. | -| `oauthScopes` | ✅ | ✅ | Kernel default U2M scopes = `['sql','offline_access']` (Thrift parity); M2M = `['all-apis']`. | -| `oauthClientId` (U2M) | ✅ | ❌ | **Kernel-side gap.** Kernel U2M hardcodes `client_id` and **rejects** a custom `oauthClientId`; Thrift honors it. | -| `oauthClientId` + no secret | ✅ (U2M) | ❌ | **Divergence.** Thrift routes to U2M with that id; kernel keys the flow off `oauthClientSecret` presence and throws an M2M "secret required" error. | +| `oauthScopes` | ❌ | ✅ | **Thrift ignores `oauthScopes`** — `createAuthProvider` never threads it into `DatabricksOAuth`, so `authenticate()` always falls back to `defaultOAuthScopes` (`['sql','offline_access']`). Only the kernel honors a custom `oauthScopes`; its defaults happen to match Thrift's fallback (U2M = `['sql','offline_access']`, M2M = `['all-apis']`). | +| `oauthClientId` (U2M) | ✅ | ✅ | The kernel adapter (`buildKernelConnectionOptions`) forwards a custom `oauthClientId` verbatim on the U2M arm; when it is absent the napi binding applies its own default `client_id`. Whether the native binding then honors or rejects a custom id is not observable from this repo — the TypeScript layer neither hardcodes an id nor rejects one. | +| `oauthClientId` + no secret | ✅ (U2M) | ✅ (U2M) | **Parity.** The kernel keys flow selection off `oauthClientSecret` presence exactly like Thrift, so `oauthClientId` + no secret routes to **U2M** (with the id forwarded) — it does **not** throw an M2M "secret required" error. | | `azureTenantId` / `useDatabricksOAuthInAzure` | ✅ | ❌ | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | | `persistence` (custom OAuth token store) | ✅ | ❌ | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | | `authType: 'custom'` (`provider`) | ✅ | ❌ | **Thrift-only.** Kernel supports only `access-token` and `databricks-oauth`. | @@ -105,7 +105,7 @@ column. | Option | Thrift | Kernel | Gap | | ----------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `preserveBigNumericPrecision` | ✅ | ✅ | DECIMAL → exact string, BIGINT → `bigint` on both. | -| `enableMetricViewMetadata` | ✅ | ❌ | **Thrift-only in the connector.** Thrift auto-injects `spark.sql.thriftserver.metadata.metricview.enabled=true` (`ThriftBackend.ts`); `KernelBackend` does **not** auto-inject it. The kernel core _can_ accept the raw conf key via `session_conf`, so a caller could pass it manually in `OpenSessionRequest.configuration`. | +| `enableMetricViewMetadata` | ✅ | ⚠️ | **Auto-injected for both backends** in `DBSQLClient.openSession`, which sets `spark.sql.thriftserver.metadata.metricview.enabled=true` on `request.configuration` before dispatch. `KernelBackend` folds that into `sessionOptions.sessionConf`, so the conf **does** reach the kernel session config. (`ThriftBackend.ts` performs a second, redundant injection on the Thrift path.) The kernel-side gap is that the key is a non-allowlisted session conf, so it is likely dropped by the kernel's case-insensitive allowlist (see "Session defaults") — not that it is never injected. | ## Session defaults (`openSession(request)`) @@ -147,14 +147,15 @@ regardless of `useKernel`. ### Supported on Thrift, missing / ignored on Kernel -1. `enableMetricViewMetadata` — no auto-injection on the kernel path. +1. `enableMetricViewMetadata` — auto-injected for both backends in + `DBSQLClient.openSession`, but the conf key is likely dropped by the + kernel's session-conf allowlist, so it has no effect on the kernel path. 2. Auth types `custom`, `token-provider`, `external-token`, `static-token`, plus `enableTokenFederation` / `federationClientId`. 3. `azureTenantId` / `useDatabricksOAuthInAzure` (Azure-direct OAuth). 4. `persistence` (custom OAuth token store). -5. Custom `oauthClientId` on the U2M flow (and `oauthClientId` + no secret). -6. SOCKS proxies. -7. Per-statement `useCloudFetch`, `useLZ4Compression`, +5. SOCKS proxies. +6. Per-statement `useCloudFetch`, `useLZ4Compression`, `stagingAllowedLocalPath`. ### Supported on Kernel, no Thrift public equivalent @@ -167,7 +168,9 @@ regardless of `useKernel`. ### Behavioral divergences to watch - **U2M flow selection** keys off `oauthClientSecret` presence on the kernel - path but is honored differently on Thrift. + path, matching Thrift: no secret ⇒ U2M, secret present ⇒ M2M. A custom + `oauthClientId` (with no secret) is forwarded on the U2M arm rather than + triggering an M2M "secret required" error. - **`socketTimeout: 0`** means "indefinite" on Thrift but is dropped on the kernel path (kernel default kept). - **`configuration`** is allowlist-filtered on the kernel path but forwarded diff --git a/lib/kernel/KernelAuth.ts b/lib/kernel/KernelAuth.ts index 295fbb9b..2e4fbf4d 100644 --- a/lib/kernel/KernelAuth.ts +++ b/lib/kernel/KernelAuth.ts @@ -474,21 +474,18 @@ export function buildKernelHttpOptions(options: ConnectionOptions): KernelHttpOp * a browser, listens on localhost:8030, exchanges the code, persists * to `~/.config/databricks-sql-kernel/oauth/{sha256}.json`). * - * **Flow selection — DELIBERATE DIVERGENCE FROM THRIFT.** Thrift's + * **Flow selection — MIRRORS THRIFT.** Thrift's * `DBSQLClient.createAuthProvider` (`DBSQLClient.ts:216`) keys off the * *secret* (`oauthClientSecret === undefined ? U2M : M2M`), so a custom - * `oauthClientId` with no secret runs U2M with that id. kernel instead keys - * off `oauthClientId` *presence* (id present → M2M, absent → U2M). The - * trade-off: keying off the id means a caller who set an id but - * typoed/forgot the secret gets the actionable M2M "secret is required" - * error instead of being silently routed to U2M (which would hide their - * intent). The cost is two real behavioural gaps vs Thrift: - * 1. `oauthClientId` + no secret → Thrift runs U2M; kernel throws - * `AuthenticationError` (M2M secret required). - * 2. kernel U2M has NO custom-client-id support — the kernel hardcodes - * `client_id = "databricks-cli"`, and kernel rejects any `oauthClientId` - * on the U2M arm. Thrift U2M honours a custom `clientId`. - * Both are documented limitations of the M0 kernel OAuth surface, not bugs. + * `oauthClientId` with no secret runs U2M with that id. This adapter keys + * off the same signal: `oauthClientSecret === undefined` ⇒ U2M, else M2M + * (see `buildKernelConnectionOptions` below). A custom `oauthClientId` is + * forwarded verbatim on the U2M arm; when absent the napi binding applies + * its own default `client_id`. The adapter therefore does NOT throw an M2M + * "secret required" error for `oauthClientId` + no secret, and does NOT + * reject a custom `oauthClientId` on U2M — those decisions, if the native + * binding makes them, happen below the TypeScript layer and are not + * observable from this repo. * * Out of scope on the OAuth paths (rejected with a clear error): * - `azureTenantId` / `useDatabricksOAuthInAzure` → Microsoft Entra diff --git a/lib/kernel/KernelBackend.ts b/lib/kernel/KernelBackend.ts index e0ee420f..221e7beb 100644 --- a/lib/kernel/KernelBackend.ts +++ b/lib/kernel/KernelBackend.ts @@ -57,7 +57,8 @@ export interface KernelBackendOptions { * (slash-prepended httpPath, AuthenticationError on missing token or * blank OAuth credentials, HiveDriverError on unsupported authType / * Azure-direct / ambiguous credential combinations). M2M and U2M - * routing key off `oauthClientId` presence; see KernelAuth.ts. + * routing key off `oauthClientSecret` presence (mirroring Thrift); see + * KernelAuth.ts. * * **Why we don't use IClientContext's connectionProvider here:** that * provider is the Thrift HTTP transport. The kernel owns its own From 532dad022bd77daaecdcfa9fbdb850f49c84f819 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Tue, 28 Jul 2026 06:03:52 +0000 Subject: [PATCH 05/12] ai: apply changes for #457 (1 review thread) Addresses: - #3663168408 at CONNECTION_PARAMETERS.md:77 Signed-off-by: peco-engineer-bot[bot] --- CONNECTION_PARAMETERS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index dcc925c1..98fc4a7d 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -74,7 +74,7 @@ column. | ---------------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `socketTimeout` | ✅ | ⚠️ | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | | `proxy` (`{protocol, host, port, auth}`) | ✅ | ⚠️ | Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | -| `noProxy` (internal) | ✅ | ✅ | Forwarded to the kernel as `bypassHosts`. | +| `noProxy` (internal) | ❌ | ✅ | **Thrift ignores `noProxy`** — `getConnectionOptions` never threads it, and `createProxyAgent` installs `getProxyForUrl: () => proxyUrl` (the proxy is returned for every URL, so there is no bypass-list logic). Only the kernel honors it, forwarded as `bypassHosts`. | | `customHeaders` | ✅ | ✅ | Kernel drops reserved `Authorization` / `x-databricks-org-id`, rejects CR/LF/NUL, and appends the connector `User-Agent` last. | | `retryMaxAttempts` | ✅ | ✅ | Total-attempt semantics on both; kernel converts to retries-after-first. | | `retriesTimeout` | ✅ | ✅ | Kernel converts ms → whole seconds. | From 768f7b6d6c1033da2c05e7eb1d4167afb3b568ee Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Mon, 17 Aug 2026 22:11:49 +0000 Subject: [PATCH 06/12] ai: apply changes for #457 (1 review thread) Addresses: - #3799169490 at CONNECTION_PARAMETERS.md:88 Signed-off-by: peco-engineer-bot[bot] Signed-off-by: eric-wang-1990 --- CONNECTION_PARAMETERS.md | 42 +++++++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 98fc4a7d..8d442dcd 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -84,21 +84,25 @@ column. ## TLS / SSL -> **Important Thrift caveat.** The public `connect()` surface on the Thrift -> backend does **not** expose any TLS customization. `getConnectionOptions` -> (`lib/DBSQLClient.ts`) maps only `host` / `port` / `path` / `socketTimeout` / -> `proxy` / `User-Agent`; the internal `IConnectionOptions.ca/cert/key` fields -> are never populated from a public option, and `HttpConnection` hardcodes -> `rejectUnauthorized: false`. All TLS-verification and custom-CA / mTLS -> controls below are therefore **kernel-only** in practice. - -| Option | Thrift | Kernel | Gap | -| -------------------------------------- | :----: | :----: | ----------------------------------------------------------------------------------------------- | -| `checkServerCertificate` | ❌ | ✅ | **Kernel-only.** Master verify toggle (secure-by-default; set `false` for accept-anything). | -| `checkServerCertificateHostname` | ❌ | ✅ | **Kernel-only.** Independent hostname-vs-SNI check; no-op when `checkServerCertificate: false`. | -| `customCaCert` (PEM string / `Buffer`) | ❌ | ✅ | **Kernel-only.** Added on top of system roots. | -| `clientCertPem` (mTLS) | ❌ | ✅ | **Kernel-only.** Must be paired with `clientKeyPem`; supplying one alone is rejected. | -| `clientKeyPem` (mTLS) | ❌ | ✅ | **Kernel-only.** PKCS#8 recommended. | +> **Thrift TLS is public and secure-by-default.** `checkServerCertificate`, +> `customCaCert`, `clientCert`, and `clientKey` are declared on the public +> `ConnectionOptions` (`lib/contracts/IDBSQLClient.ts`) and honored on the +> Thrift (default) backend. `getConnectionOptions` (`lib/DBSQLClient.ts`) maps +> `customCaCert` → `ca` (**additively**, on top of `tls.rootCertificates` + +> `NODE_EXTRA_CA_CERTS`), `clientCert` → `cert`, and `clientKey` → `key`, with a +> both-or-neither mTLS guard and `normalizePemBytes` PEM validation. It sets +> `rejectUnauthorized: options.checkServerCertificate ?? true`, and +> `HttpConnection` reads `this.options.rejectUnauthorized ?? true` — i.e. +> **verification is on by default**; you must pass `checkServerCertificate: +> false` to accept any certificate. + +| Option | Thrift | Kernel | Gap | +| -------------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------- | +| `checkServerCertificate` | ✅ | ✅ | Master verify toggle on both, secure-by-default: Thrift uses `options.checkServerCertificate ?? true`. Set `false` to accept-anything. | +| `checkServerCertificateHostname` | ❌ | ✅ | **Kernel-only.** Independent hostname-vs-SNI check; no-op when `checkServerCertificate: false`. No Thrift equivalent. | +| `customCaCert` (PEM string / `Buffer`) | ✅ | ✅ | Honored on both; added on top of system roots. Thrift maps it to `ca` additively (`tls.rootCertificates` + `NODE_EXTRA_CA_CERTS`). | +| `clientCert` (mTLS) | ✅ | ✅ | Honored on both; maps to `cert`. Must be paired with `clientKey`; supplying one alone throws a client-side error. | +| `clientKey` (mTLS) | ✅ | ✅ | Honored on both; maps to `key`. Must be paired with `clientCert`. PKCS#8 recommended. | ## Results & type rendering @@ -161,9 +165,11 @@ regardless of `useKernel`. ### Supported on Kernel, no Thrift public equivalent 1. `maxConnections` (connection-pool sizing). -2. TLS controls: `checkServerCertificate`, `checkServerCertificateHostname`, - `customCaCert`, `clientCertPem`, `clientKeyPem`. The Thrift backend exposes - **no** public TLS options and hardcodes `rejectUnauthorized: false`. +2. `checkServerCertificateHostname` — the independent hostname-vs-SNI check has + no public Thrift equivalent. (The other TLS controls — + `checkServerCertificate`, `customCaCert`, `clientCert`, `clientKey` — **are** + public and honored on the Thrift backend, which verifies certificates by + default via `checkServerCertificate ?? true`; see the TLS / SSL section.) ### Behavioral divergences to watch From 702627dab82ef718f8d4e2f6cf23d1302bc3f4d3 Mon Sep 17 00:00:00 2001 From: eric-wang-1990 Date: Mon, 17 Aug 2026 15:18:41 -0700 Subject: [PATCH 07/12] docs: restructure connection-parameter tables to Option/Type/Thrift/Kernel/Default/Note Add `Type` and `Default Value` columns to every parameter table and rename the `Gap` column to `Note`. Types come from the public `ConnectionOptions` / `ExecuteStatementOptions` shapes (`IDBSQLClient.ts`, `IDBSQLSession.ts`); defaults come from `DBSQLClientDefaults` (`DBSQLClient.ts`) and `DEFAULT_TELEMETRY_CONFIG` (`telemetry/types.ts`). The telemetry section's prose "Aspect" table is replaced with a full per-option table (one row per `telemetry*` knob with its default), and the events caveat is kept as a note. Formatted with `prettier --write` so `prettier . --check` passes (the file previously failed the check). Co-authored-by: Isaac Signed-off-by: eric-wang-1990 --- CONNECTION_PARAMETERS.md | 166 +++++++++++++++++++++------------------ 1 file changed, 89 insertions(+), 77 deletions(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 8d442dcd..1430db92 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -6,7 +6,7 @@ whether each parameter is honored on the **Thrift** backend (the default), the **SEA / Kernel** backend (opt-in via `useKernel: true`), or both. The goal is to make protocol gaps explicit: a parameter honored on one -backend but ignored (or rejected) on the other is called out in the **Gap** +backend but ignored (or rejected) on the other is called out in the **Note** column. > **Backend selection.** The connector defaults to Thrift. The SEA backend is @@ -17,17 +17,20 @@ column. ## Legend -| Symbol | Meaning | -| ------ | ------------------------------------------------------------------ | -| ✅ | Honored — the option is read and forwarded to the backend. | -| ❌ | Ignored or rejected — see the Gap column. | -| ⚠️ | Partially supported or behaves differently from the other backend. | -| — | Not applicable / no public equivalent on this backend. | +| Symbol | Meaning | +| ------ | ------------------------------------------------------------------- | +| ✅ | Honored — the option is read and forwarded to the backend. | +| ❌ | Ignored or rejected — see the Note column. | +| ⚠️ | Partially supported or behaves differently from the other backend. | +| — | Not applicable / no public equivalent / no default on this backend. | ## Sources of truth - Public option shape ← `lib/contracts/IDBSQLClient.ts` (`ConnectionOptions`, - `AuthOptions`, `OpenSessionRequest`). + `AuthOptions`, `OpenSessionRequest`) and `lib/contracts/IDBSQLSession.ts` + (`ExecuteStatementOptions`). +- Default values ← `DBSQLClientDefaults` (`lib/DBSQLClient.ts`) and + `DEFAULT_TELEMETRY_CONFIG` (`lib/telemetry/types.ts`). - Internal / kernel-only flags ← `lib/contracts/InternalConnectionOptions.ts`. - Thrift wiring ← `lib/DBSQLClient.ts` (`getConnectionOptions`, `createAuthProvider`), `lib/thrift-backend/ThriftBackend.ts`, @@ -43,44 +46,44 @@ column. ## Connection identity -| Option | Type | Thrift | Kernel | Gap | -| ---------------- | -------- | :----: | :----: | ---------------------------------------------------------------------------------------------------------------------------- | -| `host` | `string` | ✅ | ✅ | Required on both. | -| `path` | `string` | ✅ | ✅ | HTTP path; on the kernel path the org id is auto-parsed from a `?o=` query param and sent as `x-databricks-org-id`. | -| `port` | `number` | ✅ | ⚠️ | Thrift defaults to `443`. The kernel derives host/port from `host` + `path`; a standalone `port` is not separately threaded. | -| `userAgentEntry` | `string` | ✅ | ✅ | Folded into the composed `User-Agent` on both. | +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------- | -------- | :----: | :----: | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `host` | `string` | ✅ | ✅ | — (required) | Required on both. | +| `path` | `string` | ✅ | ✅ | — (required) | HTTP path; on the kernel path the org id is auto-parsed from a `?o=` query param and sent as `x-databricks-org-id`. | +| `port` | `number` | ✅ | ⚠️ | `443` | Thrift defaults to `443` (`options.port \|\| 443`). The kernel derives host/port from `host` + `path`; a standalone `port` is not separately threaded. | +| `userAgentEntry` | `string` | ✅ | ✅ | — | Folded into the composed `User-Agent` on both. | ## Authentication -| Option | Thrift | Kernel | Gap | -| ---------------------------------------------------------------------------- | :------: | :----: | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `authType: 'access-token'` + `token` (PAT) | ✅ | ✅ | | -| `authType: 'databricks-oauth'` — M2M (`oauthClientId` + `oauthClientSecret`) | ✅ | ✅ | Kernel runs OIDC discovery + client-credentials internally. | -| `authType: 'databricks-oauth'` — U2M (browser) | ✅ | ⚠️ | See U2M gaps below. | -| `oauthScopes` | ❌ | ✅ | **Thrift ignores `oauthScopes`** — `createAuthProvider` never threads it into `DatabricksOAuth`, so `authenticate()` always falls back to `defaultOAuthScopes` (`['sql','offline_access']`). Only the kernel honors a custom `oauthScopes`; its defaults happen to match Thrift's fallback (U2M = `['sql','offline_access']`, M2M = `['all-apis']`). | -| `oauthClientId` (U2M) | ✅ | ✅ | The kernel adapter (`buildKernelConnectionOptions`) forwards a custom `oauthClientId` verbatim on the U2M arm; when it is absent the napi binding applies its own default `client_id`. Whether the native binding then honors or rejects a custom id is not observable from this repo — the TypeScript layer neither hardcodes an id nor rejects one. | -| `oauthClientId` + no secret | ✅ (U2M) | ✅ (U2M) | **Parity.** The kernel keys flow selection off `oauthClientSecret` presence exactly like Thrift, so `oauthClientId` + no secret routes to **U2M** (with the id forwarded) — it does **not** throw an M2M "secret required" error. | -| `azureTenantId` / `useDatabricksOAuthInAzure` | ✅ | ❌ | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | -| `persistence` (custom OAuth token store) | ✅ | ❌ | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | -| `authType: 'custom'` (`provider`) | ✅ | ❌ | **Thrift-only.** Kernel supports only `access-token` and `databricks-oauth`. | -| `authType: 'token-provider'` (`tokenProvider`) | ✅ | ❌ | **Thrift-only.** | -| `authType: 'external-token'` (`getToken`) | ✅ | ❌ | **Thrift-only.** | -| `authType: 'static-token'` (`staticToken`) | ✅ | ❌ | **Thrift-only.** | -| `enableTokenFederation` / `federationClientId` | ✅ | ❌ | **Thrift-only** (available on the token-provider / external-token / static-token arms, none of which the kernel supports). | +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------------------------------------------------------------------- | -------------------- | :------: | :------: | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `authType: 'access-token'` + `token` (PAT) | `token: string` | ✅ | ✅ | `authType` ⇒ `'access-token'` | Default auth mode when `authType` is omitted. | +| `authType: 'databricks-oauth'` — M2M (`oauthClientId` + `oauthClientSecret`) | `string` + `string` | ✅ | ✅ | — | Kernel runs OIDC discovery + client-credentials internally. | +| `authType: 'databricks-oauth'` — U2M (browser) | — | ✅ | ⚠️ | — | See U2M rows below. | +| `oauthScopes` | `Array` | ❌ | ✅ | U2M `['sql','offline_access']`, M2M `['all-apis']` | **Thrift ignores `oauthScopes`** — `createAuthProvider` never threads it into `DatabricksOAuth`, so `authenticate()` always falls back to `defaultOAuthScopes` (`['sql','offline_access']`). Only the kernel honors a custom `oauthScopes`; its defaults happen to match Thrift's fallback. | +| `oauthClientId` (U2M) | `string` | ✅ | ✅ | napi default `client_id` when absent | The kernel adapter (`buildKernelConnectionOptions`) forwards a custom `oauthClientId` verbatim on the U2M arm; when it is absent the napi binding applies its own default `client_id`. Whether the native binding then honors or rejects a custom id is not observable from this repo — the TypeScript layer neither hardcodes an id nor rejects one. | +| `oauthClientId` + no secret | `string` | ✅ (U2M) | ✅ (U2M) | — | **Parity.** The kernel keys flow selection off `oauthClientSecret` presence exactly like Thrift, so `oauthClientId` + no secret routes to **U2M** (with the id forwarded) — it does **not** throw an M2M "secret required" error. | +| `azureTenantId` / `useDatabricksOAuthInAzure` | `string` / `boolean` | ✅ | ❌ | — | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | +| `persistence` (custom OAuth token store) | `OAuthPersistence` | ✅ | ❌ | — | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | +| `authType: 'custom'` (`provider`) | `IAuthentication` | ✅ | ❌ | — | **Thrift-only.** Kernel supports only `access-token` and `databricks-oauth`. | +| `authType: 'token-provider'` (`tokenProvider`) | `ITokenProvider` | ✅ | ❌ | — | **Thrift-only.** | +| `authType: 'external-token'` (`getToken`) | `TokenCallback` | ✅ | ❌ | — | **Thrift-only.** | +| `authType: 'static-token'` (`staticToken`) | `string` | ✅ | ❌ | — | **Thrift-only.** | +| `enableTokenFederation` / `federationClientId` | `boolean` / `string` | ✅ | ❌ | `false` / — | **Thrift-only** (available on the token-provider / external-token / static-token arms, none of which the kernel supports). | ## HTTP client, proxy, retries -| Option | Thrift | Kernel | Gap | -| ---------------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `socketTimeout` | ✅ | ⚠️ | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | -| `proxy` (`{protocol, host, port, auth}`) | ✅ | ⚠️ | Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | -| `noProxy` (internal) | ❌ | ✅ | **Thrift ignores `noProxy`** — `getConnectionOptions` never threads it, and `createProxyAgent` installs `getProxyForUrl: () => proxyUrl` (the proxy is returned for every URL, so there is no bypass-list logic). Only the kernel honors it, forwarded as `bypassHosts`. | -| `customHeaders` | ✅ | ✅ | Kernel drops reserved `Authorization` / `x-databricks-org-id`, rejects CR/LF/NUL, and appends the connector `User-Agent` last. | -| `retryMaxAttempts` | ✅ | ✅ | Total-attempt semantics on both; kernel converts to retries-after-first. | -| `retriesTimeout` | ✅ | ✅ | Kernel converts ms → whole seconds. | -| `retryDelayMin` | ✅ | ✅ | Kernel converts ms → seconds. | -| `retryDelayMax` | ✅ | ✅ | Kernel converts ms → seconds. | -| `maxConnections` (pool size) | ❌ | ✅ | **Kernel-only** (`InternalConnectionOptions`). Thrift has no connection pool. | +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------------------- | ------------------------ | :----: | :----: | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `socketTimeout` | `number` (ms) | ✅ | ⚠️ | `900000` (15 min) | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | +| `proxy` | `ProxyOptions` | ✅ | ⚠️ | — | `{protocol, host, port, auth}`. Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | +| `noProxy` (internal) | `string` | ❌ | ✅ | — | **Thrift ignores `noProxy`** — `getConnectionOptions` never threads it, and `getProxyForUrl: () => proxyUrl` returns the proxy for every URL (no bypass-list logic). Only the kernel honors it, forwarded as `bypassHosts`. | +| `customHeaders` | `Record` | ✅ | ✅ | — | Kernel drops reserved `Authorization` / `x-databricks-org-id`, rejects CR/LF/NUL, and appends the connector `User-Agent` last. | +| `retryMaxAttempts` | `number` | ✅ | ✅ | `5` | Total-attempt semantics on both; kernel converts to retries-after-first. | +| `retriesTimeout` | `number` (ms) | ✅ | ✅ | `900000` (15 min) | Kernel converts ms → whole seconds. | +| `retryDelayMin` | `number` (ms) | ✅ | ✅ | `1000` (1 s) | Kernel converts ms → seconds. | +| `retryDelayMax` | `number` (ms) | ✅ | ✅ | `60000` (60 s) | Kernel converts ms → seconds. | +| `maxConnections` (pool size) | `number` (internal) | ❌ | ✅ | kernel default | **Kernel-only** (`InternalConnectionOptions`). Thrift has no connection pool. | ## TLS / SSL @@ -94,56 +97,65 @@ column. > `rejectUnauthorized: options.checkServerCertificate ?? true`, and > `HttpConnection` reads `this.options.rejectUnauthorized ?? true` — i.e. > **verification is on by default**; you must pass `checkServerCertificate: -> false` to accept any certificate. +false` to accept any certificate. -| Option | Thrift | Kernel | Gap | -| -------------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------- | -| `checkServerCertificate` | ✅ | ✅ | Master verify toggle on both, secure-by-default: Thrift uses `options.checkServerCertificate ?? true`. Set `false` to accept-anything. | -| `checkServerCertificateHostname` | ❌ | ✅ | **Kernel-only.** Independent hostname-vs-SNI check; no-op when `checkServerCertificate: false`. No Thrift equivalent. | -| `customCaCert` (PEM string / `Buffer`) | ✅ | ✅ | Honored on both; added on top of system roots. Thrift maps it to `ca` additively (`tls.rootCertificates` + `NODE_EXTRA_CA_CERTS`). | -| `clientCert` (mTLS) | ✅ | ✅ | Honored on both; maps to `cert`. Must be paired with `clientKey`; supplying one alone throws a client-side error. | -| `clientKey` (mTLS) | ✅ | ✅ | Honored on both; maps to `key`. Must be paired with `clientCert`. PKCS#8 recommended. | +| Option | Type | Thrift | Kernel | Default Value | Note | +| -------------------------------- | ------------------ | :----: | :----: | ------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| `checkServerCertificate` | `boolean` | ✅ | ✅ | `true` | Master verify toggle on both, secure-by-default: Thrift uses `options.checkServerCertificate ?? true`. Set `false` to accept-anything. | +| `checkServerCertificateHostname` | `boolean` (kernel) | ❌ | ✅ | `true` | **Kernel-only.** Independent hostname-vs-SNI check; no-op when `checkServerCertificate: false`. No Thrift equivalent. | +| `customCaCert` | `Buffer \| string` | ✅ | ✅ | — | PEM. Honored on both; added on top of system roots. Thrift maps it to `ca` additively (`tls.rootCertificates` + `NODE_EXTRA_CA_CERTS`). | +| `clientCert` (mTLS) | `Buffer \| string` | ✅ | ✅ | — | Honored on both; maps to `cert`. Must be paired with `clientKey`; supplying one alone throws a client-side error. | +| `clientKey` (mTLS) | `Buffer \| string` | ✅ | ✅ | — | Honored on both; maps to `key`. Must be paired with `clientCert`. PKCS#8 recommended. | ## Results & type rendering -| Option | Thrift | Kernel | Gap | -| ----------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `preserveBigNumericPrecision` | ✅ | ✅ | DECIMAL → exact string, BIGINT → `bigint` on both. | -| `enableMetricViewMetadata` | ✅ | ⚠️ | **Auto-injected for both backends** in `DBSQLClient.openSession`, which sets `spark.sql.thriftserver.metadata.metricview.enabled=true` on `request.configuration` before dispatch. `KernelBackend` folds that into `sessionOptions.sessionConf`, so the conf **does** reach the kernel session config. (`ThriftBackend.ts` performs a second, redundant injection on the Thrift path.) The kernel-side gap is that the key is a non-allowlisted session conf, so it is likely dropped by the kernel's case-insensitive allowlist (see "Session defaults") — not that it is never injected. | +| Option | Type | Thrift | Kernel | Default Value | Note | +| ----------------------------- | --------- | :----: | :----: | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `preserveBigNumericPrecision` | `boolean` | ✅ | ✅ | `false` | DECIMAL → exact string, BIGINT → `bigint` on both. | +| `enableMetricViewMetadata` | `boolean` | ✅ | ⚠️ | `false` | **Auto-injected for both backends** in `DBSQLClient.openSession`, which sets `spark.sql.thriftserver.metadata.metricview.enabled=true` on `request.configuration` before dispatch. `KernelBackend` folds that into `sessionOptions.sessionConf`, so the conf **does** reach the kernel session config. (`ThriftBackend.ts` performs a second, redundant injection on the Thrift path.) The kernel-side gap is that the key is a non-allowlisted session conf, so it is likely dropped by the kernel's case-insensitive allowlist (see "Session defaults") — not that it is never injected. | ## Session defaults (`openSession(request)`) -| Option | Thrift | Kernel | Gap | -| ------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `initialCatalog` | ✅ | ✅ | Kernel → `DefaultOpts.catalog` on `CreateSession`. | -| `initialSchema` | ✅ | ✅ | Kernel → `DefaultOpts.schema`. | -| `configuration` (session confs) | ✅ | ⚠️ | Kernel matches keys **case-insensitively against an allowlist** and uppercases them; **non-allowlisted keys are dropped with a warning**. Thrift forwards the map more freely. | -| `queryTags` | ✅ | ✅ | Both serialize into the reserved `QUERY_TAGS` conf; `queryTags` takes precedence over `configuration.QUERY_TAGS`. | +| Option | Type | Thrift | Kernel | Default Value | Note | +| ------------------------------- | --------------------------------------------- | :----: | :----: | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `initialCatalog` | `string` | ✅ | ✅ | — | Kernel → `DefaultOpts.catalog` on `CreateSession`. | +| `initialSchema` | `string` | ✅ | ✅ | — | Kernel → `DefaultOpts.schema`. | +| `configuration` (session confs) | `{ [key: string]: string }` | ✅ | ⚠️ | — | Kernel matches keys **case-insensitively against an allowlist** and uppercases them; **non-allowlisted keys are dropped with a warning**. Thrift forwards the map more freely. | +| `queryTags` | `Record` | ✅ | ✅ | — | Both serialize into the reserved `QUERY_TAGS` conf; `queryTags` takes precedence over `configuration.QUERY_TAGS`. | ## Telemetry -All `telemetry*` options (`telemetryEnabled`, `telemetryBatchSize`, -`telemetryFlushIntervalMs`, `telemetryMaxRetries`, -`telemetryAuthenticatedExport`, `telemetryCircuitBreakerThreshold`, -`telemetryCircuitBreakerTimeout`, `telemetryCloseTimeoutMs`, -`telemetryMaxStatementMetrics`, `telemetryMaxPendingMetrics`) live in the -driver-layer `ClientConfig`, not in either backend, so they are read -regardless of `useKernel`. - -| Aspect | Thrift | Kernel | Gap | -| ----------------------------------------- | :----: | :----: | ---------------------------------------------------------------------------------------------------------------------- | -| Telemetry config knobs | ✅ | ✅ | Backend-agnostic (driver layer). | -| Statement / CloudFetch telemetry _events_ | ✅ | ⚠️ | The kernel owns result fetching internally, so it emits fewer per-statement / cloud-fetch events than the Thrift path. | +All `telemetry*` options live in the driver-layer `ClientConfig`, not in either +backend, so they are read regardless of `useKernel`. Defaults are sourced from +`DEFAULT_TELEMETRY_CONFIG` (`lib/telemetry/types.ts`). + +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------------------------- | ------------- | :----: | :----: | ------------- | --------------------------------------------------- | +| `telemetryEnabled` | `boolean` | ✅ | ✅ | `true` | Enabled by default, gated by a server feature flag. | +| `telemetryBatchSize` | `number` | ✅ | ✅ | `100` | Metrics per export batch. | +| `telemetryFlushIntervalMs` | `number` (ms) | ✅ | ✅ | `5000` | Periodic flush interval. | +| `telemetryMaxRetries` | `number` | ✅ | ✅ | `3` | Export retry attempts. | +| `telemetryAuthenticatedExport` | `boolean` | ✅ | ✅ | `true` | Export via the authenticated endpoint. | +| `telemetryCircuitBreakerThreshold` | `number` | ✅ | ✅ | `5` | Consecutive failures before the breaker opens. | +| `telemetryCircuitBreakerTimeout` | `number` (ms) | ✅ | ✅ | `60000` | Breaker open duration. | +| `telemetryCloseTimeoutMs` | `number` (ms) | ✅ | ✅ | `2000` | Caps `client.close()` shutdown latency. | +| `telemetryMaxStatementMetrics` | `number` | ✅ | ✅ | `5000` | Hard cap for the per-statement aggregation map. | +| `telemetryMaxPendingMetrics` | `number` | ✅ | ✅ | `500` | Cap on buffered, not-yet-exported metrics. | + +> **Telemetry _events_ differ by backend.** The config knobs above are +> backend-agnostic (driver layer), but the kernel owns result fetching +> internally, so it emits fewer per-statement / CloudFetch telemetry events than +> the Thrift path. ## Per-statement options (`session.executeStatement(sql, options)`) -| Option | Thrift | Kernel | Gap | -| -------------------------------------- | :----: | :----: | ------------------------------------------------------------------------------------------------------------------------------------------ | -| `maxRows` | ✅ | ✅ | Kernel applies it at fetch time in the facade rather than on the request. | -| `useCloudFetch` | ✅ | ❌ | **Thrift-only.** Kernel ignores it (logs a no-op warning); CloudFetch is governed by the kernel's result configuration, not per-statement. | -| `useLZ4Compression` | ✅ | ❌ | **Thrift-only.** Kernel ignores it (no-op warning); the kernel auto-detects and decompresses `LZ4_FRAME` from the server result manifest. | -| `stagingAllowedLocalPath` (volume ops) | ✅ | ❌ | **Thrift-only.** Not supported on the kernel path. | -| `runAsync` | ✅ | ⚠️ | Deprecated; not threaded on the kernel path. | +| Option | Type | Thrift | Kernel | Default Value | Note | +| -------------------------------------- | ----------------------------------- | :----: | :----: | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| `maxRows` | `number \| bigint \| Int64 \| null` | ✅ | ✅ | `100000` | Kernel applies it at fetch time in the facade rather than on the request. | +| `useCloudFetch` | `boolean` | ✅ | ❌ | `true` | **Thrift-only.** Kernel ignores it (logs a no-op warning); CloudFetch is governed by the kernel's result configuration, not per-statement. | +| `useLZ4Compression` | `boolean` | ✅ | ❌ | `true` | **Thrift-only.** Kernel ignores it (no-op warning); the kernel auto-detects and decompresses `LZ4_FRAME` from the server result manifest. | +| `stagingAllowedLocalPath` (volume ops) | `string \| string[]` | ✅ | ❌ | — | **Thrift-only.** Not supported on the kernel path. | +| `runAsync` | `boolean` | ✅ | ⚠️ | `false` | Deprecated; not threaded on the kernel path. | --- From 6c290ec14e90632762e6721611f884fbc4ff7b3d Mon Sep 17 00:00:00 2001 From: eric-wang-1990 Date: Mon, 17 Aug 2026 15:33:52 -0700 Subject: [PATCH 08/12] docs: fix runAsync backend semantics (Thrift no-op, kernel honors it) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `runAsync` row said "not threaded on the kernel path", but `IDBSQLSession.ts` documents the reverse: on Thrift `runAsync` is a no-op (the path always submits async and polls; the option is not read), while the kernel reads it to select sync direct-results (default) vs. submit-and-poll. Flip the verdicts (Thrift ⚠️, kernel ✅) and rewrite the note to match. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 --- CONNECTION_PARAMETERS.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 1430db92..ee11d174 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -149,13 +149,13 @@ backend, so they are read regardless of `useKernel`. Defaults are sourced from ## Per-statement options (`session.executeStatement(sql, options)`) -| Option | Type | Thrift | Kernel | Default Value | Note | -| -------------------------------------- | ----------------------------------- | :----: | :----: | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | -| `maxRows` | `number \| bigint \| Int64 \| null` | ✅ | ✅ | `100000` | Kernel applies it at fetch time in the facade rather than on the request. | -| `useCloudFetch` | `boolean` | ✅ | ❌ | `true` | **Thrift-only.** Kernel ignores it (logs a no-op warning); CloudFetch is governed by the kernel's result configuration, not per-statement. | -| `useLZ4Compression` | `boolean` | ✅ | ❌ | `true` | **Thrift-only.** Kernel ignores it (no-op warning); the kernel auto-detects and decompresses `LZ4_FRAME` from the server result manifest. | -| `stagingAllowedLocalPath` (volume ops) | `string \| string[]` | ✅ | ❌ | — | **Thrift-only.** Not supported on the kernel path. | -| `runAsync` | `boolean` | ✅ | ⚠️ | `false` | Deprecated; not threaded on the kernel path. | +| Option | Type | Thrift | Kernel | Default Value | Note | +| -------------------------------------- | ----------------------------------- | :----: | :----: | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `maxRows` | `number \| bigint \| Int64 \| null` | ✅ | ✅ | `100000` | Kernel applies it at fetch time in the facade rather than on the request. | +| `useCloudFetch` | `boolean` | ✅ | ❌ | `true` | **Thrift-only.** Kernel ignores it (logs a no-op warning); CloudFetch is governed by the kernel's result configuration, not per-statement. | +| `useLZ4Compression` | `boolean` | ✅ | ❌ | `true` | **Thrift-only.** Kernel ignores it (no-op warning); the kernel auto-detects and decompresses `LZ4_FRAME` from the server result manifest. | +| `stagingAllowedLocalPath` (volume ops) | `string \| string[]` | ✅ | ❌ | — | **Thrift-only.** Not supported on the kernel path. | +| `runAsync` | `boolean` | ⚠️ | ✅ | `false` | Deprecated. **Thrift:** no-op — the path always submits async (`runAsync: true` on the wire) and polls during fetch; the option is not read. **Kernel:** selects the execution path — `false`/unset (default) runs the blocking direct-results path (cancellable mid-compute); `true` submits and polls (returns a pending handle). | --- From fe47621ec5373bd9ad87aa554bde1060a20ae342 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Mon, 17 Aug 2026 22:41:30 +0000 Subject: [PATCH 09/12] ai: apply changes for #457 (1 review thread) Addresses: - #3799315347 at CONNECTION_PARAMETERS.md:60 Signed-off-by: peco-engineer-bot[bot] Signed-off-by: eric-wang-1990 --- CONNECTION_PARAMETERS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index ee11d174..167f59a8 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -57,7 +57,7 @@ column. | Option | Type | Thrift | Kernel | Default Value | Note | | ---------------------------------------------------------------------------- | -------------------- | :------: | :------: | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `authType: 'access-token'` + `token` (PAT) | `token: string` | ✅ | ✅ | `authType` ⇒ `'access-token'` | Default auth mode when `authType` is omitted. | +| `authType: 'access-token'` + `token` (PAT) | `token: string` | ✅ | ✅ | `authType` ⇒ `'access-token'` | `authType` is an enum accepting one of `'access-token'`, `'databricks-oauth'`, `'custom'`, `'token-provider'`, `'external-token'`, or `'static-token'`; each row below names the value. This row is the default auth mode, used when `authType` is omitted. | | `authType: 'databricks-oauth'` — M2M (`oauthClientId` + `oauthClientSecret`) | `string` + `string` | ✅ | ✅ | — | Kernel runs OIDC discovery + client-credentials internally. | | `authType: 'databricks-oauth'` — U2M (browser) | — | ✅ | ⚠️ | — | See U2M rows below. | | `oauthScopes` | `Array` | ❌ | ✅ | U2M `['sql','offline_access']`, M2M `['all-apis']` | **Thrift ignores `oauthScopes`** — `createAuthProvider` never threads it into `DatabricksOAuth`, so `authenticate()` always falls back to `defaultOAuthScopes` (`['sql','offline_access']`). Only the kernel honors a custom `oauthScopes`; its defaults happen to match Thrift's fallback. | From ec26f9a6a0e507d116fdd7dda784822e411a7b28 Mon Sep 17 00:00:00 2001 From: eric-wang-1990 Date: Mon, 17 Aug 2026 16:42:38 -0700 Subject: [PATCH 10/12] docs: consolidate authType rows (both-backends vs Thrift-only) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per review feedback (PR #457 discussion r3799323450): collapse the seven per-authType rows into two — one for the modes both backends accept (`access-token`, `databricks-oauth`) and one for the Thrift-only modes (`custom`, `token-provider`, `external-token`, `static-token`). OAuth sub-option rows are unchanged. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 --- CONNECTION_PARAMETERS.md | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 167f59a8..448b6585 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -55,21 +55,16 @@ column. ## Authentication -| Option | Type | Thrift | Kernel | Default Value | Note | -| ---------------------------------------------------------------------------- | -------------------- | :------: | :------: | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `authType: 'access-token'` + `token` (PAT) | `token: string` | ✅ | ✅ | `authType` ⇒ `'access-token'` | `authType` is an enum accepting one of `'access-token'`, `'databricks-oauth'`, `'custom'`, `'token-provider'`, `'external-token'`, or `'static-token'`; each row below names the value. This row is the default auth mode, used when `authType` is omitted. | -| `authType: 'databricks-oauth'` — M2M (`oauthClientId` + `oauthClientSecret`) | `string` + `string` | ✅ | ✅ | — | Kernel runs OIDC discovery + client-credentials internally. | -| `authType: 'databricks-oauth'` — U2M (browser) | — | ✅ | ⚠️ | — | See U2M rows below. | -| `oauthScopes` | `Array` | ❌ | ✅ | U2M `['sql','offline_access']`, M2M `['all-apis']` | **Thrift ignores `oauthScopes`** — `createAuthProvider` never threads it into `DatabricksOAuth`, so `authenticate()` always falls back to `defaultOAuthScopes` (`['sql','offline_access']`). Only the kernel honors a custom `oauthScopes`; its defaults happen to match Thrift's fallback. | -| `oauthClientId` (U2M) | `string` | ✅ | ✅ | napi default `client_id` when absent | The kernel adapter (`buildKernelConnectionOptions`) forwards a custom `oauthClientId` verbatim on the U2M arm; when it is absent the napi binding applies its own default `client_id`. Whether the native binding then honors or rejects a custom id is not observable from this repo — the TypeScript layer neither hardcodes an id nor rejects one. | -| `oauthClientId` + no secret | `string` | ✅ (U2M) | ✅ (U2M) | — | **Parity.** The kernel keys flow selection off `oauthClientSecret` presence exactly like Thrift, so `oauthClientId` + no secret routes to **U2M** (with the id forwarded) — it does **not** throw an M2M "secret required" error. | -| `azureTenantId` / `useDatabricksOAuthInAzure` | `string` / `boolean` | ✅ | ❌ | — | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | -| `persistence` (custom OAuth token store) | `OAuthPersistence` | ✅ | ❌ | — | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | -| `authType: 'custom'` (`provider`) | `IAuthentication` | ✅ | ❌ | — | **Thrift-only.** Kernel supports only `access-token` and `databricks-oauth`. | -| `authType: 'token-provider'` (`tokenProvider`) | `ITokenProvider` | ✅ | ❌ | — | **Thrift-only.** | -| `authType: 'external-token'` (`getToken`) | `TokenCallback` | ✅ | ❌ | — | **Thrift-only.** | -| `authType: 'static-token'` (`staticToken`) | `string` | ✅ | ❌ | — | **Thrift-only.** | -| `enableTokenFederation` / `federationClientId` | `boolean` / `string` | ✅ | ❌ | `false` / — | **Thrift-only** (available on the token-provider / external-token / static-token arms, none of which the kernel supports). | +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------------------------------------- | -------------------------------------------------------------------------- | :------: | :------: | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `authType` — supported on both backends | `'access-token'` \| `'databricks-oauth'` | ✅ | ✅ | `'access-token'` | The two auth modes both backends accept. `access-token` uses `token` (PAT) and is the default when `authType` is omitted. `databricks-oauth` covers M2M (`oauthClientId` + `oauthClientSecret`; kernel runs OIDC discovery + client-credentials internally) and U2M (browser; no secret — kernel U2M differs slightly, see the OAuth sub-option rows below). | +| `authType` — Thrift-only | `'custom'` \| `'token-provider'` \| `'external-token'` \| `'static-token'` | ✅ | ❌ | — | **Thrift-only.** `custom` (`provider: IAuthentication`), `token-provider` (`tokenProvider: ITokenProvider`), `external-token` (`getToken: TokenCallback`), `static-token` (`staticToken`). The kernel throws `unsupported auth mode` for all four — it supports only the two modes above. | +| `oauthScopes` | `Array` | ❌ | ✅ | U2M `['sql','offline_access']`, M2M `['all-apis']` | **Thrift ignores `oauthScopes`** — `createAuthProvider` never threads it into `DatabricksOAuth`, so `authenticate()` always falls back to `defaultOAuthScopes` (`['sql','offline_access']`). Only the kernel honors a custom `oauthScopes`; its defaults happen to match Thrift's fallback. | +| `oauthClientId` (U2M) | `string` | ✅ | ✅ | napi default `client_id` when absent | The kernel adapter (`buildKernelConnectionOptions`) forwards a custom `oauthClientId` verbatim on the U2M arm; when it is absent the napi binding applies its own default `client_id`. Whether the native binding then honors or rejects a custom id is not observable from this repo — the TypeScript layer neither hardcodes an id nor rejects one. | +| `oauthClientId` + no secret | `string` | ✅ (U2M) | ✅ (U2M) | — | **Parity.** The kernel keys flow selection off `oauthClientSecret` presence exactly like Thrift, so `oauthClientId` + no secret routes to **U2M** (with the id forwarded) — it does **not** throw an M2M "secret required" error. | +| `azureTenantId` / `useDatabricksOAuthInAzure` | `string` / `boolean` | ✅ | ❌ | — | **Thrift-only.** Kernel rejects Azure-direct (Entra) OAuth; workspace-OIDC discovery covers Azure workspaces without it. | +| `persistence` (custom OAuth token store) | `OAuthPersistence` | ✅ | ❌ | — | **Thrift-only.** Kernel throws; it auto-persists U2M tokens to `~/.config/databricks-sql-kernel/oauth/` and does not cache M2M. | +| `enableTokenFederation` / `federationClientId` | `boolean` / `string` | ✅ | ❌ | `false` / — | **Thrift-only** (available on the token-provider / external-token / static-token arms, none of which the kernel supports). | ## HTTP client, proxy, retries From 7f4a9724bdfafefee6a4619cbae134ae2ced0639 Mon Sep 17 00:00:00 2001 From: "peco-engineer-bot[bot]" Date: Mon, 17 Aug 2026 23:49:53 +0000 Subject: [PATCH 11/12] ai: apply changes for #457 (1 review thread) Addresses: - #3799637110 at CONNECTION_PARAMETERS.md:76 Signed-off-by: peco-engineer-bot[bot] --- CONNECTION_PARAMETERS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 448b6585..1c6748e6 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -73,7 +73,7 @@ column. | `socketTimeout` | `number` (ms) | ✅ | ⚠️ | `900000` (15 min) | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | | `proxy` | `ProxyOptions` | ✅ | ⚠️ | — | `{protocol, host, port, auth}`. Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | | `noProxy` (internal) | `string` | ❌ | ✅ | — | **Thrift ignores `noProxy`** — `getConnectionOptions` never threads it, and `getProxyForUrl: () => proxyUrl` returns the proxy for every URL (no bypass-list logic). Only the kernel honors it, forwarded as `bypassHosts`. | -| `customHeaders` | `Record` | ✅ | ✅ | — | Kernel drops reserved `Authorization` / `x-databricks-org-id`, rejects CR/LF/NUL, and appends the connector `User-Agent` last. | +| `customHeaders` | `Record` | ⚠️ | ✅ | — | **Thrift: out-of-band requests only.** `getConnectionOptions` never threads `customHeaders` into the Thrift query transport (it sets only `User-Agent`); the map is consumed by `buildCustomHeaders` for driver-owned telemetry POSTs / feature-flag GETs and SPOG `x-databricks-org-id` injection, not the primary transport or OAuth/OIDC token requests. Kernel applies them to every request (dropping reserved `Authorization` / `x-databricks-org-id`, rejecting CR/LF/NUL, appending the connector `User-Agent` last). | | `retryMaxAttempts` | `number` | ✅ | ✅ | `5` | Total-attempt semantics on both; kernel converts to retries-after-first. | | `retriesTimeout` | `number` (ms) | ✅ | ✅ | `900000` (15 min) | Kernel converts ms → whole seconds. | | `retryDelayMin` | `number` (ms) | ✅ | ✅ | `1000` (1 s) | Kernel converts ms → seconds. | From 5c067fb8efd25a4eb1fea6c571b0eac09a53954f Mon Sep 17 00:00:00 2001 From: eric-wang-1990 Date: Mon, 17 Aug 2026 16:52:33 -0700 Subject: [PATCH 12/12] docs: run prettier on connection-parameters table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit edited the customHeaders row without re-aligning the table; prettier requires the column padding to match the widest cell. No content change — realignment only. Fixes the lint / "Check code style" job. Co-authored-by: Isaac Signed-off-by: eric-wang-1990 --- CONNECTION_PARAMETERS.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/CONNECTION_PARAMETERS.md b/CONNECTION_PARAMETERS.md index 1c6748e6..30853562 100644 --- a/CONNECTION_PARAMETERS.md +++ b/CONNECTION_PARAMETERS.md @@ -68,17 +68,17 @@ column. ## HTTP client, proxy, retries -| Option | Type | Thrift | Kernel | Default Value | Note | -| ---------------------------- | ------------------------ | :----: | :----: | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `socketTimeout` | `number` (ms) | ✅ | ⚠️ | `900000` (15 min) | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | -| `proxy` | `ProxyOptions` | ✅ | ⚠️ | — | `{protocol, host, port, auth}`. Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | -| `noProxy` (internal) | `string` | ❌ | ✅ | — | **Thrift ignores `noProxy`** — `getConnectionOptions` never threads it, and `getProxyForUrl: () => proxyUrl` returns the proxy for every URL (no bypass-list logic). Only the kernel honors it, forwarded as `bypassHosts`. | -| `customHeaders` | `Record` | ⚠️ | ✅ | — | **Thrift: out-of-band requests only.** `getConnectionOptions` never threads `customHeaders` into the Thrift query transport (it sets only `User-Agent`); the map is consumed by `buildCustomHeaders` for driver-owned telemetry POSTs / feature-flag GETs and SPOG `x-databricks-org-id` injection, not the primary transport or OAuth/OIDC token requests. Kernel applies them to every request (dropping reserved `Authorization` / `x-databricks-org-id`, rejecting CR/LF/NUL, appending the connector `User-Agent` last). | -| `retryMaxAttempts` | `number` | ✅ | ✅ | `5` | Total-attempt semantics on both; kernel converts to retries-after-first. | -| `retriesTimeout` | `number` (ms) | ✅ | ✅ | `900000` (15 min) | Kernel converts ms → whole seconds. | -| `retryDelayMin` | `number` (ms) | ✅ | ✅ | `1000` (1 s) | Kernel converts ms → seconds. | -| `retryDelayMax` | `number` (ms) | ✅ | ✅ | `60000` (60 s) | Kernel converts ms → seconds. | -| `maxConnections` (pool size) | `number` (internal) | ❌ | ✅ | kernel default | **Kernel-only** (`InternalConnectionOptions`). Thrift has no connection pool. | +| Option | Type | Thrift | Kernel | Default Value | Note | +| ---------------------------- | ------------------------ | :----: | :----: | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `socketTimeout` | `number` (ms) | ✅ | ⚠️ | `900000` (15 min) | Kernel maps it to the request timeout but forwards **only positive** values — `socketTimeout: 0` (Thrift "wait indefinitely") is omitted so the kernel keeps its large default. | +| `proxy` | `ProxyOptions` | ✅ | ⚠️ | — | `{protocol, host, port, auth}`. Kernel accepts **`http://` / `https://` only**; a SOCKS `protocol` surfaces a kernel connect error (Thrift supports SOCKS variants). | +| `noProxy` (internal) | `string` | ❌ | ✅ | — | **Thrift ignores `noProxy`** — `getConnectionOptions` never threads it, and `getProxyForUrl: () => proxyUrl` returns the proxy for every URL (no bypass-list logic). Only the kernel honors it, forwarded as `bypassHosts`. | +| `customHeaders` | `Record` | ⚠️ | ✅ | — | **Thrift: out-of-band requests only.** `getConnectionOptions` never threads `customHeaders` into the Thrift query transport (it sets only `User-Agent`); the map is consumed by `buildCustomHeaders` for driver-owned telemetry POSTs / feature-flag GETs and SPOG `x-databricks-org-id` injection, not the primary transport or OAuth/OIDC token requests. Kernel applies them to every request (dropping reserved `Authorization` / `x-databricks-org-id`, rejecting CR/LF/NUL, appending the connector `User-Agent` last). | +| `retryMaxAttempts` | `number` | ✅ | ✅ | `5` | Total-attempt semantics on both; kernel converts to retries-after-first. | +| `retriesTimeout` | `number` (ms) | ✅ | ✅ | `900000` (15 min) | Kernel converts ms → whole seconds. | +| `retryDelayMin` | `number` (ms) | ✅ | ✅ | `1000` (1 s) | Kernel converts ms → seconds. | +| `retryDelayMax` | `number` (ms) | ✅ | ✅ | `60000` (60 s) | Kernel converts ms → seconds. | +| `maxConnections` (pool size) | `number` (internal) | ❌ | ✅ | kernel default | **Kernel-only** (`InternalConnectionOptions`). Thrift has no connection pool. | ## TLS / SSL