Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

## Unreleased

- Kernel backend (`useKernel: true`): preserve qualified `INTERVAL MONTH` and
Comment thread
cathleeny marked this conversation as resolved.
Comment thread
cathleeny marked this conversation as resolved.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — This entry advertises the fix as a user-facing behavior change but omits the published-native-package caveat that the sibling kernel entry directly below it carries. The PR moves all parameter binding off positionalParams/namedParams and onto the new rawParams surface (StatementSpec::param_raw, RawParameterInput), which is a brand-new field added to the native contract in this PR (via the KERNEL_REV bump to bb4a077). The customer-installed native packages remain pinned at 0.2.0 (package.json:94-101).

If 0.2.0 predates the param_raw surface, this is worse than the getTypeInfo case: napi-rs deserializes only the object fields it knows, so an unrecognized rawParams would be silently ignored — every bound parameter would drop on the wire for customer npm installs, changing query results with no error. That is precisely the failure mode the code comment in KernelSessionBackend.executeStatement says must never be a no-op ("a dropped param would silently change results").

Please either confirm 0.2.0 already exposes param_raw (in which case the source-build routing is safe for customers), or add the same "requires a follow-up published-native-package bump" caveat + kernel PR / ticket reference this entry currently lacks.

`INTERVAL DAY` parameter types on the SEA wire by using the kernel raw-parameter
path.
- Kernel backend source builds (`useKernel: true`, built from `KERNEL_REV`): `getTypeInfo()` now matches the Thrift backend's canonical 18-column, 20-row type-info result. Customer-facing npm installs require a follow-up bump to a published native package containing this Kernel change. ([databricks-sql-kernel#291](https://github.com/databricks/databricks-sql-kernel/pull/291), PECOBLR-4166)
- Kernel backend (`useKernel: true`): **Azure Entra (Azure AD) auth is now threaded through the kernel path.** On `authType: 'databricks-oauth'`: **U2M** (no secret) always routes to `OAuthU2m` — the kernel runs one cloud-blind in-house workspace-federated browser flow (it uses the workspace's OIDC-discovered authorize endpoint verbatim), which works against Azure workspaces, so Azure U2M forwards the in-house app (`databricks-sql-connector`) + `sql offline_access` scopes exactly like AWS/GCP, regardless of `useDatabricksOAuthInAzure` (verified E2E against a live Azure workspace). **M2M** (secret): `useDatabricksOAuthInAzure: true` (or non-Azure) → `OAuthM2m` (workspace-OIDC client-credentials); an Azure host with `useDatabricksOAuthInAzure` absent/`false` → the Entra-direct Azure service-principal M2M (`AzureSpM2m`, the Entra SP creds ride `oauthClientId`/`oauthClientSecret`, `azureTenantId` optional and auto-discovered when omitted). On a non-Azure host `useDatabricksOAuthInAzure` is inert. The `AzureSpM2m` path requires a `databricks-sql-kernel` native module that exposes the Azure SP surface — landed on `main` via [databricks-sql-kernel#282](https://github.com/databricks/databricks-sql-kernel/pull/282) (which the pinned `KERNEL_REV` `ef1a6f2` carries; the surface was originally proposed in [#280](https://github.com/databricks/databricks-sql-kernel/pull/280), which never reached `main`); U2M works on any kernel build. (PECOBLR-4141 / PECOBLR-4120)

Expand Down
2 changes: 1 addition & 1 deletion KERNEL_REV
Original file line number Diff line number Diff line change
@@ -1 +1 @@
628abd6f5045897efcadb38ec77a1e9e0c23544e
Comment thread
cathleeny marked this conversation as resolved.
bb4a0770673926201d386d1d9295da8d4bf459aa
25 changes: 23 additions & 2 deletions bin/build-native.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,24 @@ set -euo pipefail
driver_repo=$(pwd)
kernel_repo=${DATABRICKS_SQL_KERNEL_REPO:-../../databricks-sql-kernel}
napi_dir="${kernel_repo}/napi"
kernel_package_version=$(
node -e '
const { optionalDependencies = {} } = require(process.argv[1]);
const versions = [...new Set(
Object.entries(optionalDependencies)
.filter(([name]) => name.startsWith("@databricks/databricks-sql-kernel-"))
.map(([, version]) => version),
)];

if (versions.length !== 1) {
throw new Error(
`Expected one pinned kernel package version, found: ${versions.join(", ") || "none"}`,
);
}

process.stdout.write(versions[0]);
' "${driver_repo}/package.json"
)
napi_major=$(
cargo metadata --format-version 1 --locked --manifest-path "${napi_dir}/Cargo.toml" |
node -e '
Expand Down Expand Up @@ -34,10 +52,13 @@ esac
build_profile=${BUILD_PROFILE---release}

cd "${napi_dir}"
# napi-rs normally embeds the kernel source manifest version in index.js. The
# source can advance before its native packages are published, so generate the
# loader guard from the version the driver actually installs instead.
if [[ -n "${build_profile//[[:space:]]/}" ]]; then
read -r -a build_profile_args <<< "${build_profile}"
"${cli[@]}" build --platform "${build_profile_args[@]}"
npm_new_version="${kernel_package_version}" "${cli[@]}" build --platform "${build_profile_args[@]}"
else
"${cli[@]}" build --platform
npm_new_version="${kernel_package_version}" "${cli[@]}" build --platform
fi
cp index.* "${driver_repo}/native/kernel/"
14 changes: 4 additions & 10 deletions lib/kernel/KernelNativeLoader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,7 @@ import type {
ArrowBatch as NativeArrowBatch,
ArrowSchema as NativeArrowSchema,
ExecuteOptions as NativeExecuteOptions,
TypedValueInput as NativeTypedValueInput,
NamedTypedValueInput as NativeNamedTypedValueInput,
RawParameterInput as NativeRawParameterInput,
AsyncStatement as NativeAsyncStatement,
AsyncResultHandle as NativeAsyncResultHandle,
CancellableExecution as NativeCancellableExecution,
Expand All @@ -53,15 +52,10 @@ export type KernelArrowSchema = NativeArrowSchema;
export type KernelConnection = NativeConnection;
export type KernelStatement = NativeStatement;

// Per-statement execution options and bound-parameter inputs are kernel
// concerns: the napi binding generates the canonical shapes (`positionalParams`
// / `namedParams` as `TypedValueInput` / `NamedTypedValueInput`, plus
// `rowLimit`, `statementConf`, `queryTags`). We re-export
// rather than re-declare so the driver-side param codec can never drift from
// the kernel contract.
// Per-statement execution options and raw-parameter inputs come directly from
// the generated kernel contract so the driver-side codec cannot drift.
export type KernelNativeExecuteOptions = NativeExecuteOptions;
export type KernelNativeTypedValueInput = NativeTypedValueInput;
export type KernelNativeNamedTypedValueInput = NativeNamedTypedValueInput;
export type KernelNativeRawParameterInput = NativeRawParameterInput;

// Async-submit surface: `Connection.submitStatement` returns an
// `AsyncStatement` (status / awaitResult / cancel / close); `awaitResult`
Expand Down
31 changes: 14 additions & 17 deletions lib/kernel/KernelPositionalParams.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

import { DBSQLParameter, DBSQLParameterValue } from '../DBSQLParameter';
import ParameterError from '../errors/ParameterError';
import { KernelNativeTypedValueInput, KernelNativeNamedTypedValueInput } from './KernelNativeLoader';
import { KernelNativeRawParameterInput } from './KernelNativeLoader';
import assertBindableValue from './KernelInputValidation';

/**
Expand Down Expand Up @@ -58,15 +58,14 @@ function decimalPrecisionScale(v: string): string {

/**
* Reduce a `DBSQLParameter | DBSQLParameterValue` to the napi
* `TypedValueInput` (`{ sqlType, value? }`) the kernel's positional-param
* codec (`parse_typed_value`) accepts. Reuses `DBSQLParameter.toSparkParameter`
* — the same type-inference + value-stringification the Thrift backend uses —
* then adapts the type name to the codec's expectations:
* `RawParameterInput` (`{ name?, sqlType, value? }`) accepted by the kernel's
* raw-parameter path. Reuses `DBSQLParameter.toSparkParameter` — the same
* type-inference + value-stringification the Thrift backend uses — then adapts
* the type name where required:
* - DECIMAL → `DECIMAL(p,s)` (parenthesised form required)
* - INTERVAL * → `INTERVAL` (the codec's single interval type name)
* - a missing value ⇒ SQL NULL (`parse_typed_value` maps `value: None` to NULL).
* - a missing value ⇒ SQL NULL.
*/
function toTypedValueInput(value: DBSQLParameter | DBSQLParameterValue): KernelNativeTypedValueInput {
function toRawParameterInput(value: DBSQLParameter | DBSQLParameterValue): KernelNativeRawParameterInput {
const param = value instanceof DBSQLParameter ? value : new DBSQLParameter({ value });
const spark = param.toSparkParameter();
const stringValue = spark.value?.stringValue ?? undefined;
Expand All @@ -81,44 +80,42 @@ function toTypedValueInput(value: DBSQLParameter | DBSQLParameterValue): KernelN
const upper = sqlType.toUpperCase();
if (upper === 'DECIMAL') {
sqlType = `DECIMAL(${decimalPrecisionScale(stringValue)})`;
} else if (upper.startsWith('INTERVAL')) {
sqlType = 'INTERVAL';
}
return { sqlType, value: stringValue };
}

/**
* Convert the public `ordinalParameters` option into the napi
* `positionalParams` array (1-based `?` placeholders). Returns `undefined`
* `rawParams` array (1-based `?` placeholders). Returns `undefined`
* when none were supplied, so the caller can keep the minimal no-options
* call shape.
*/
export function buildKernelPositionalParams(
ordinalParameters?: Array<DBSQLParameter | DBSQLParameterValue>,
): Array<KernelNativeTypedValueInput> | undefined {
): Array<KernelNativeRawParameterInput> | undefined {
if (ordinalParameters === undefined || ordinalParameters.length === 0) {
return undefined;
}
return ordinalParameters.map((value, i) => {
assertBindableValue(value, `ordinalParameters[${i}]`);
return toTypedValueInput(value);
return toRawParameterInput(value);
});
}

/**
* Convert the public `namedParameters` option (`Record<name, value>`) into
* the napi `namedParams` array (`:name` placeholders). Each value reuses the
* same `toTypedValueInput` mapping (DECIMAL → DECIMAL(p,s), NULL → VOID, …),
* the napi `rawParams` array (`:name` placeholders). Each value reuses the
* same `toRawParameterInput` mapping (DECIMAL → DECIMAL(p,s), NULL → VOID, …),
* then carries its name. Returns `undefined` when none were supplied.
*/
export function buildKernelNamedParams(
namedParameters?: Record<string, DBSQLParameter | DBSQLParameterValue>,
): Array<KernelNativeNamedTypedValueInput> | undefined {
): Array<KernelNativeRawParameterInput> | undefined {
if (namedParameters === undefined || Object.keys(namedParameters).length === 0) {
return undefined;
}
return Object.keys(namedParameters).map((name) => {
assertBindableValue(namedParameters[name], `namedParameters[${name}]`);
return { name, ...toTypedValueInput(namedParameters[name]) };
return { name, ...toRawParameterInput(namedParameters[name]) };
});
}
9 changes: 4 additions & 5 deletions lib/kernel/KernelSessionBackend.ts
Original file line number Diff line number Diff line change
Expand Up @@ -298,11 +298,10 @@ export default class KernelSessionBackend implements ISessionBackend {
}

const execOptions: KernelNativeExecuteOptions = {};
if (positionalParams !== undefined) {
execOptions.positionalParams = positionalParams;
}
if (namedParams !== undefined) {
execOptions.namedParams = namedParams;
// Raw binding preserves qualified SQL types such as INTERVAL MONTH.
const rawParams = positionalParams ?? namedParams;
if (rawParams !== undefined) {
Comment thread
cathleeny marked this conversation as resolved.
Comment thread
cathleeny marked this conversation as resolved.
execOptions.rawParams = rawParams;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The driver now forwards parameters only through execOptions.rawParams and no longer sets positionalParams / namedParams. napi-rs deserializes object arguments field-by-field and silently ignores unknown keys — so if a consumer runs against a native binding that predates rawParams support (i.e. a published @databricks/databricks-sql-kernel-* optional dependency rather than the freshly-bumped KERNEL_REV source build), every bound parameter is silently dropped and the query executes with unbound placeholders (wrong results, not an error).

The loader comment itself acknowledges rawParams is "available in the source-pinned kernel before its published types." The CHANGELOG entry for this PR does not carry the customer-install caveat that the earlier getTypeInfo() entry did ("Customer-facing npm installs require a follow-up bump to a published native package…"). Consider either (a) adding the equivalent CHANGELOG caveat here, or (b) keeping a fallback that still sets positionalParams/namedParams when rawParams is unsupported, so older bindings don't silently drop parameters.

}
// NB: `queryTimeout` is intentionally NOT forwarded — it is a no-op on kernel
// (SQL Warehouses use `STATEMENT_TIMEOUT`; mapping it to `wait_timeout` would
Expand Down
27 changes: 27 additions & 0 deletions native/kernel/index.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

115 changes: 114 additions & 1 deletion tests/e2e/kernel/execution-e2e.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
// limitations under the License.

import { expect } from 'chai';
import { DBSQLClient } from '../../../lib';
import { DBSQLClient, DBSQLParameter, DBSQLParameterType } from '../../../lib';
import { ConnectionOptions } from '../../../lib/contracts/IDBSQLClient';
import { InternalConnectionOptions } from '../../../lib/contracts/InternalConnectionOptions';

Expand Down Expand Up @@ -121,4 +121,117 @@ describe('kernel execution end-to-end', function e2eSuite() {
await session.close();
await client.close();
});

it('binds ordinary positional parameters through rawParams', async () => {
const client = new DBSQLClient();

await client.connect({
host: hostName as string,
path: httpPath as string,
token: token as string,
useKernel: true,
} as ConnectionOptions & InternalConnectionOptions);

const session = await client.openSession({ initialCatalog: 'main' });
let operation;
try {
operation = await session.executeStatement('SELECT ? AS p_int, ? AS p_string, ? AS p_bool', {
ordinalParameters: [42, 'hello', true],
});
expect(await operation.fetchAll()).to.deep.equal([{ p_int: 42, p_string: 'hello', p_bool: true }]);
} finally {
await operation?.close();
await session.close();
await client.close();
}
});

it('binds named NULL and empty string through rawParams on the async path', async () => {
const client = new DBSQLClient();

await client.connect({
host: hostName as string,
path: httpPath as string,
token: token as string,
useKernel: true,
} as ConnectionOptions & InternalConnectionOptions);

const session = await client.openSession({ initialCatalog: 'main' });
let operation;
try {
operation = await session.executeStatement('SELECT :null_value AS null_value, :empty_value AS empty_value', {
namedParameters: { null_value: null, empty_value: '' },
runAsync: true,
});
expect(await operation.fetchAll()).to.deep.equal([{ null_value: null, empty_value: '' }]);
} finally {
await operation?.close();
await session.close();
await client.close();
}
});

it('binds a valid INTERVAL MONTH on the SEA wire', async () => {
const client = new DBSQLClient();

await client.connect({
host: hostName as string,
path: httpPath as string,
token: token as string,
useKernel: true,
} as ConnectionOptions & InternalConnectionOptions);

const session = await client.openSession({ initialCatalog: 'main' });
let operation;
try {
operation = await session.executeStatement("SELECT ? = INTERVAL '13' MONTH AS matches", {
ordinalParameters: [
new DBSQLParameter({
type: DBSQLParameterType.INTERVALMONTH,
value: '13',
}),
],
});
expect(await operation.fetchAll()).to.deep.equal([{ matches: true }]);
} finally {
await operation?.close();
await session.close();
await client.close();
}
});

it('preserves INTERVAL MONTH on the SEA wire', async () => {
const client = new DBSQLClient();

await client.connect({
host: hostName as string,
path: httpPath as string,
token: token as string,
useKernel: true,
} as ConnectionOptions & InternalConnectionOptions);

const session = await client.openSession({ initialCatalog: 'main' });
let operation;
let caught: unknown;
try {
operation = await session.executeStatement('SELECT ?', {
ordinalParameters: [
new DBSQLParameter({
type: DBSQLParameterType.INTERVALMONTH,
value: '2-6',
Comment thread
cathleeny marked this conversation as resolved.
}),
],
});
await operation.fetchAll();
} catch (error) {
caught = error;
} finally {
await operation?.close();
await session.close();
await client.close();
}

// "2-6" is valid YEAR TO MONTH syntax, but invalid for INTERVAL MONTH.
expect(caught).to.be.instanceOf(Error);
});
});
Loading
Loading