feat(kernel): thread Azure Entra auth (U2M + SP M2M) through the kernel path

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Release History
 
+## Unreleased
+
+- Kernel backend (`useKernel: true`): **Azure Entra (Azure AD) auth is now threaded through the kernel path.** On `authType: 'databricks-oauth'`: **U2M** (no secret) always routes to `OAuthU2m` — the kernel runs one cloud-blind in-house workspace-federated browser flow (it uses the workspace's OIDC-discovered authorize endpoint verbatim), which works against Azure workspaces, so Azure U2M forwards the in-house app (`databricks-sql-connector`) + `sql offline_access` scopes exactly like AWS/GCP, regardless of `useDatabricksOAuthInAzure` (verified E2E against a live Azure workspace). **M2M** (secret): `useDatabricksOAuthInAzure: true` (or non-Azure) → `OAuthM2m` (workspace-OIDC client-credentials); an Azure host with `useDatabricksOAuthInAzure` absent/`false` → the Entra-direct Azure service-principal M2M (`AzureSpM2m`, the Entra SP creds ride `oauthClientId`/`oauthClientSecret`, `azureTenantId` optional and auto-discovered when omitted). On a non-Azure host `useDatabricksOAuthInAzure` is inert. The `AzureSpM2m` path requires a `databricks-sql-kernel` native module that exposes the Azure SP surface — landed on `main` via [databricks-sql-kernel#282](https://github.com/databricks/databricks-sql-kernel/pull/282) (which the pinned `KERNEL_REV` `5e5dea9` carries; the surface was originally proposed in [#280](https://github.com/databricks/databricks-sql-kernel/pull/280), which never reached `main`); U2M works on any kernel build. (PECOBLR-4141 / PECOBLR-4120)
+
 ## 2.0.0
 
 **Breaking changes — completes the security cleanup that 1.17.0 could not do without breaking changes.**

KERNEL_REV

@@ -1 +1 @@
-eff8950428f4e6cc9975c663ec919f334962f7d0
+5e5dea91ebc17df49d63665e1f933bacb3072c65

lib/kernel/KernelAuth.ts

@@ -252,6 +252,14 @@ export type KernelNativeConnectionOptions = KernelSessionDefaults &
         oauthScopes?: Array<string>;
         oauthClientId?: string;
       }
+    | {
+        hostName: string;
+        httpPath: string;
+        authMode: 'AzureSpM2m';
+        azureClientId: string;
+        azureClientSecret: string;
+        azureTenantId?: string;
+      }
   );
 
 function prependSlash(str: string): string {
@@ -261,6 +269,34 @@ function prependSlash(str: string): string {
   return str;
 }
 
+/**
+ * Azure Databricks host suffixes — the superset the Thrift driver's
+ * `OAuthManager.getManager` recognises (`.azuredatabricks.net`,
+ * `.databricks.azure.us`, `.databricks.azure.cn`). Used to decide whether an
+ * OAuth connection is on Azure and therefore subject to the in-house-vs-
+ * Entra-direct split.
+ */
+const AZURE_HOST_SUFFIXES = ['.azuredatabricks.net', '.databricks.azure.us', '.databricks.azure.cn'];
+
+/**
+ * True when `host` is an Azure Databricks workspace host. Normalises the input
+ * more aggressively than the Thrift driver's `getManager` (which only
+ * lowercases and strips a leading `https://`): here we also trim surrounding
+ * whitespace, strip either scheme, then drop any path and explicit `:port`, so
+ * a caller passing a bare host, a padded string, or a full URL with a port is
+ * treated identically. The suffix set matches `getManager`, so routing stays a
+ * superset — not a byte-for-byte match — of Thrift's.
+ */
+function isAzureHost(host: string): boolean {
+  const normalized = host
+    .trim()
+    .toLowerCase()
+    .replace(/^https?:\/\//, '')
+    .split('/')[0]
+    .split(':')[0];
+  return AZURE_HOST_SUFFIXES.some((suffix) => normalized.endsWith(suffix));
+}
+
 /**
  * Reject inputs that pass `typeof === 'string' && length > 0` but are
  * structurally useless as credentials: whitespace-only strings, and the
@@ -481,11 +517,25 @@ export function buildKernelHttpOptions(options: ConnectionOptions): KernelHttpOp
  *     binding makes them, happen below the TypeScript layer and are not
  *     observable from this repo.
  *
+ * Azure (Entra) on the OAuth path. The kernel runs a single, cloud-blind
+ * in-house U2M flow and workspace-OIDC M2M; only Entra-direct **M2M** gets a
+ * dedicated kernel mode:
+ *   - **U2M (no secret), any cloud, any `useDatabricksOAuthInAzure`** →
+ *     `OAuthU2m`. The kernel uses the workspace's OIDC-discovered authorize
+ *     endpoint (`{host}/oidc/v1/authorize`) verbatim; that in-house
+ *     workspace-federated flow works against Azure workspaces too (they federate
+ *     the browser login to Entra server-side — verified E2E). So Azure U2M is
+ *     NOT special-cased and NOT rejected — it forwards the in-house app
+ *     (`databricks-sql-connector`) + `sql offline_access`, exactly like AWS/GCP.
+ *   - **M2M (secret) with `useDatabricksOAuthInAzure: true`** (or non-Azure) →
+ *     `OAuthM2m` (workspace-OIDC client-credentials).
+ *   - **M2M (secret) on an Azure host with `useDatabricksOAuthInAzure` absent/
+ *     `false`** (Entra-direct) → Azure service-principal M2M (`AzureSpM2m`); the
+ *     Entra SP creds ride `oauthClientId`/`oauthClientSecret`, `azureTenantId`
+ *     optional (kernel auto-discovers).
+ *   - On a non-Azure host `useDatabricksOAuthInAzure` is inert.
+ *
  * Out of scope on the OAuth paths (rejected with a clear error):
- *   - `azureTenantId` / `useDatabricksOAuthInAzure` → Microsoft Entra
- *     direct flow. The kernel uses workspace-OIDC discovery (which works
- *     against Azure workspaces too — they serve `/oidc/.well-known/...`)
- *     and does not implement the Entra-direct scope-rewrite path.
  *   - `persistence` on M2M → M2M tokens are not cached (re-issuing is
  *     cheap; no refresh token).
  *   - `persistence` on U2M → custom token store is a parity gap;
@@ -499,7 +549,7 @@ export function buildKernelHttpOptions(options: ConnectionOptions): KernelHttpOp
  *
  * Throws:
  *   - `AuthenticationError` for missing/blank required credentials.
- *   - `HiveDriverError` for unsupported auth modes / Azure-direct /
+ *   - `HiveDriverError` for unsupported auth modes /
  *     custom persistence / ambiguous combinations.
  */
 /**
@@ -667,12 +717,91 @@ export function buildKernelConnectionOptions(options: ConnectionOptions): Kernel
       );
     }
 
-    if (oauth.azureTenantId !== undefined || oauth.useDatabricksOAuthInAzure === true) {
-      throw new HiveDriverError(
-        'kernel backend: Azure-direct OAuth (azureTenantId / useDatabricksOAuthInAzure) ' +
-          'is not supported. The workspace-OIDC discovery path handles Azure workspaces ' +
-          'today without these options.',
-      );
+    // Azure Entra-direct **M2M** → the kernel's dedicated azure-sp-m2m. Closely
+    // mirroring the Thrift driver's `OAuthManager.getManager`, an Azure host with
+    // `useDatabricksOAuthInAzure` NOT set to true (the Entra-direct default) plus a
+    // secret is an Entra service-principal client-credentials flow: the Entra SP
+    // credentials ride the generic `oauthClientId` / `oauthClientSecret` (Thrift
+    // convention); forward them as `azureClientId` / `azureClientSecret`.
+    // `azureTenantId` is optional — the kernel auto-discovers it from the workspace
+    // `/aad/auth` redirect when omitted.
+    //
+    // Azure **U2M** is deliberately NOT special-cased and NOT rejected. The kernel
+    // runs a single, cloud-blind in-house U2M flow: it uses the workspace's
+    // OIDC-discovered authorize endpoint (`{host}/oidc/v1/authorize`) verbatim, and
+    // that in-house workspace-federated flow works against Azure workspaces (the
+    // workspace federates the browser login to Entra server-side; verified E2E). So
+    // ALL U2M — including Azure, with or without `useDatabricksOAuthInAzure` — falls
+    // through to the standard `OAuthU2m` path below, which forwards the in-house app
+    // (`databricks-sql-connector`) + `sql offline_access` scopes, exactly like
+    // AWS/GCP. Handing the kernel the Thrift Azure Entra-direct app / scope instead
+    // would derail its in-house flow to a broken AAD authorize URL.
+    //
+    // One deliberate divergence from Thrift: `isAzureHost` uses the full suffix
+    // superset (incl. `.databricks.azure.us`) for every branch, whereas Thrift's
+    // `useDatabricksOAuthInAzure`-true arm omits `.databricks.azure.us` and so
+    // throws `OAuth is not supported` for a US-gov host in that mode. Here such a
+    // host falls through to the in-house flow (accepted) instead — intentional,
+    // since the kernel's in-house flow is cloud-blind and reachable everywhere.
+    // The `oauthClientSecret !== undefined` check is inline so TypeScript narrows
+    // the field to `string` inside the branch (for the AzureSpM2m literal below).
+    if (
+      isAzureHost(options.host) &&
+      oauth.useDatabricksOAuthInAzure !== true &&
+      oauth.oauthClientSecret !== undefined
+    ) {
+      // Entra-direct SP M2M is a client-credentials flow (no refresh token), so
+      // `persistence` is rejected here for parity with the workspace-OIDC M2M and
+      // U2M arms below (and matching the contract docblock's "persistence on M2M
+      // → rejected" note). Otherwise a caller's hook would be silently dropped.
+      if (oauth.persistence !== undefined) {
+        throw new HiveDriverError(
+          'kernel backend: `persistence` is not supported on Azure service-principal M2M ' +
+            '(M2M tokens have no refresh token; the kernel re-issues on expiry).',
+        );
+      }
+      // Reject a present-but-degenerate secret (`''`, whitespace, or the reserved
+      // `'undefined'`/`'null'` shell-export strings) up front. Unlike the generic
+      // `OAuthM2m` arm below — which forwards such values verbatim for byte-for-byte
+      // Thrift parity — this Azure arm has no parity contract (it already rejects a
+      // missing id outright), so a blank credential is as unusable as a missing one
+      // and would only surface an opaque Entra `invalid_client` downstream.
+      const azureClientSecret = oauth.oauthClientSecret;
+      if (isBlankOrReserved(azureClientSecret)) {
+        throw new HiveDriverError(
+          'kernel backend: Azure service-principal M2M requires a non-blank `oauthClientSecret` ' +
+            '(the Entra app-registration client secret).',
+        );
+      }
+      const azureClientId = oauth.oauthClientId;
+      if (typeof azureClientId !== 'string' || isBlankOrReserved(azureClientId)) {
+        throw new HiveDriverError(
+          'kernel backend: Azure service-principal M2M requires `oauthClientId` (the Entra ' +
+            'app-registration client id) alongside `oauthClientSecret`.',
+        );
+      }
+      // `oauthScopes` is intentionally NOT forwarded here (and the `AzureSpM2m`
+      // union member has no such field), unlike the generic `OAuthM2m` arm below
+      // which honors an override. Entra service-principal tokens use a fixed
+      // `<resource>/.default` scope that the kernel derives from the Azure app id;
+      // a caller-supplied scope override is meaningless to that flow, so it is
+      // dropped by design rather than plumbed through.
+      const azure = {
+        ...base,
+        authMode: 'AzureSpM2m' as const,
+        azureClientId,
+        azureClientSecret,
+      };
+      // Forward `azureTenantId` only when it's a real value. A blank/reserved
+      // string (`''`, whitespace, `'undefined'`/`'null'` shell-export artifacts)
+      // is treated as omitted so the kernel auto-discovers the tenant from the
+      // workspace `/aad/auth` redirect, rather than being handed a degenerate
+      // tenant that suppresses discovery and yields a malformed AAD URL. Matches
+      // this arm's `oauthClientId`/`oauthClientSecret` hygiene above and the
+      // Thrift `AzureOAuthManager` empty-tenant fallback.
+      return oauth.azureTenantId !== undefined && !isBlankOrReserved(oauth.azureTenantId)
+        ? { ...azure, azureTenantId: oauth.azureTenantId }
+        : azure;
     }
 
     // Flow selector + client-id resolution mirror the Thrift driver EXACTLY