feat(kernel): thread Azure Entra auth (U2M + SP M2M) through the kernel path

[eric-wang-1990]

What

Threads Azure Entra (Azure AD) auth — both U2M and service-principal M2M — through the useKernel: true path, replacing the previous blanket Azure-OAuth rejection. This is the NodeJS analog of the Python connector's kernel Azure bridge, and it consumes the kernel Azure surface — napi AuthMode::AzureSpM2m — now on kernel main via databricks-sql-kernel#282 (which re-landed the napi + C-ABI surface after #280 was mis-merged into an orphaned branch).

How

lib/kernel/KernelAuth.ts — on authType: 'databricks-oauth', useDatabricksOAuthInAzure selects the flavour on an Azure host, mirroring the Thrift OAuthManager.getManager:

host	useDatabricksOAuthInAzure	secret?	kernel routing
any	any	no (U2M)	OAuthU2m — in-house workspace-federated browser flow (in-house app + sql offline_access); works on Azure too
Azure	true (in-house)	yes	OAuthM2m (workspace-OIDC client-credentials)
Azure	absent/false (Entra-direct)	yes	AzureSpM2m (Entra SP M2M)
non-Azure	(inert)	yes	existing OAuthM2m

U2M is never rejected and never special-cased by cloud. The kernel runs a single, cloud-blind in-house U2M flow (it uses the workspace's OIDC-discovered {host}/oidc/v1/authorize verbatim), and that flow works against Azure workspaces — the workspace federates the browser login to Entra server-side. Feeding the kernel the Thrift Azure Entra-direct app / user_impersonation scope would derail it to a broken AAD authorize URL, so the kernel path forwards the in-house app (databricks-sql-connector) + sql offline_access for all clouds. Only Entra-direct M2M (Azure host + secret + not useDatabricksOAuthInAzure) uses the dedicated AzureSpM2m.

The kernel runs the in-house / workspace-federated flow natively (workspace-OIDC discovery works against Azure workspaces), and it is cloud-blind — so U2M always uses it (the in-house app + sql offline_access), for every cloud, never special-cased or rejected. The only Entra-direct flow the kernel implements is SP M2M (azure-sp-m2m, direct Entra client-credentials + tenant auto-discovery). For AzureSpM2m the Entra SP creds ride the generic oauthClientId/oauthClientSecret (Thrift convention); azureTenantId is forwarded when set and auto-discovered by the kernel otherwise.

New: an AzureSpM2m variant on the native-options union and an isAzureHost helper. azureWorkspaceResourceId (the management-token / RBAC-only-SP path) is not exposed here yet — a follow-up; the data token authenticates the common workspace-member SP.

Tests

Unit (tests/unit/kernel/auth-m2m.test.ts, auth-u2m.test.ts): Entra-direct M2M → AzureSpM2m (with/without tenant); in-house M2M under useDatabricksOAuthInAzure: true → OAuthM2m; Azure U2M (with and without useDatabricksOAuthInAzure) → in-house OAuthU2m. All 315 kernel unit tests pass; prettier --check + eslint clean.

E2E (live Azure workspace, native module built from kernel main @ 5e5dea9, which includes #282):

• Azure U2M (useDatabricksOAuthInAzure: true, no secret) — full browser authorization-code flow against {host}/oidc/v1/authorize, returned { one: 1, who: 'e.wang@databricks.com' }. ✅
• Azure SP M2M (Entra-direct, secret) — { one: 1, who: '<sp-app-id>' }. ✅

Depends on

• napi AzureSpM2m surface landed on kernel main via databricks-sql-kernel#282 (merged; it re-landed Create DBSQLOperation from IDBSQLSession #280's napi + C-ABI changes, which had been mis-merged into the already-merged Upgrade version of apache-arrow for security #263 branch and never reached main). This PR's KERNEL_REV now points at merged kernel main (5e5dea9), so kernel-e2e builds the native module from a main that carries the Azure SP surface. Re-verified E2E against that build (M2M who=c2108ab0…, U2M who=e.wang@…).
• Still pending (customer-facing): the npm @databricks/databricks-sql-kernel-* @ 0.2.0 pins in package.json must bump to a published kernel release that ships Give warning for deprecated clientId param #282 before end users get the AzureSpM2m path — 0.2.0 lacks it. That release is tracked by PECOBLR-4194 (the kernel 1.0.0 cut); the pin bump waits on it. Azure U2M rides the existing OAuthU2m and needs neither the KERNEL_REV bump nor a new release.
• JIRA: PECOBLR-4141 (SP M2M), PECOBLR-4120 (U2M).

This pull request and its description were written by Isaac.

[peco-review-bot]

Verdict: 2 Low

Looks good — the Entra-direct routing (useDatabricksOAuthInAzure splitting in-house vs Entra-direct, SP M2M → AzureSpM2m, Entra-direct U2M rejected) is correct and matches the described table and the updated tests. Two low-severity consistency notes: persistence is silently accepted on the new AzureSpM2m branch (rejected everywhere else on the M2M/U2M paths, and contradicts the contract docblock), and the oauthClientId-required rejection for Azure SP M2M lacks a unit test. The bundled bot-infra changes (.bot/config.yaml, learning-log, workflow) are unrelated to the auth feature but self-contained.

[Copilot]

Pull request overview

This PR extends the NodeJS kernel backend (useKernel: true) to support Azure Entra (Azure AD) authentication routing for authType: 'databricks-oauth', including Entra-direct service-principal M2M (AzureSpM2m) and in-house/workspace-federated U2M/M2M flows, and updates tests accordingly. It also adds an “engineer-bot learning” daily workflow and supporting configuration/log scaffolding.

Changes:

• Add AzureSpM2m to the kernel native auth options union and implement Azure-host routing logic in buildKernelConnectionOptions.
• Update kernel unit tests to reflect the new Azure routing and the Entra-direct U2M rejection behavior.
• Add engineer-bot retrospective learning workflow + config/log plumbing, and bump the shared bot engine pin.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
lib/kernel/KernelAuth.ts	Adds AzureSpM2m native options and implements Azure host routing for in-house vs Entra-direct flows.
tests/unit/kernel/auth-u2m.test.ts	Updates U2M kernel tests for Azure routing and Entra-direct U2M rejection.
tests/unit/kernel/auth-m2m.test.ts	Updates M2M kernel tests to cover Azure Entra-direct SP M2M (AzureSpM2m) and in-house routing.
CHANGELOG.md	Documents the new Azure Entra auth routing behavior on the kernel backend.
.github/workflows/engineer-bot-learning.yml	Adds a daily cron workflow to extract retrospective learnings and open/update a rolling PR.
.github/actions/bot-prelude/action.yml	Updates the pinned bot-engine commit SHA used by bot workflows.
.claude/knowledge/learning-log.md	Seeds the shared learning log file consumed by the author flow and appended by retrospective.
.bot/config.yaml	Wires the author phase to read the learning log and enables the retrospective configuration block.

Suppressed comments (1)

lib/kernel/KernelAuth.ts:688

• Azure routing diverges from the Thrift OAuthManager.getManager behavior for .databricks.azure.us: Thrift only treats that suffix as Azure when useDatabricksOAuthInAzure is false (Entra-direct), but the kernel path will currently allow useDatabricksOAuthInAzure: true and route it to in-house OAuthU2m/OAuthM2m. If the goal is to mirror Thrift routing (as described in the PR), add an explicit guard for .databricks.azure.us when useDatabricksOAuthInAzure:true to avoid a silent behavior mismatch.

      staticToken?: string;
      federationClientId?: string;

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[peco-review-bot]

Verdict: 1 Medium

Looks solid overall — the Azure routing mirrors Thrift's getManager split cleanly and is well-tested. One medium concern: the new AzureSpM2m branch bypasses the M2M persistence rejection, silently dropping an option that every other OAuth path rejects loudly.

[peco-review-bot]

Verdict: 2 Medium · 2 Low

Solid, well-documented change that faithfully mirrors the Thrift OAuthManager.getManager split for the kernel path; routing logic and tests look correct. Main concerns: persistence is silently ignored on the new Azure SP M2M branch (rejected everywhere else), a stale Throws docstring line, the temporary KERNEL_REV pin to an unmerged kernel branch + un-bumped npm pins, and a missing test for the new oauthClientId-required throw.

[peco-review-bot]

Verdict: 1 Medium

Looks good overall — the AzureSpM2m routing, TS narrowing, and persistence/clientId guards in KernelAuth.ts are correct and well-tested. One medium concern: the emitted AzureSpM2m authMode outruns the still-pinned 0.2.0 npm kernel, so end users hitting the Azure Entra-direct M2M combo get an opaque native error instead of the prior clean rejection until the pins are bumped.

Other findings

• 🟡 Medium — The new routing emits authMode: 'AzureSpM2m' for the Azure-host + secret + useDatabricksOAuthInAzure !== true combination, but the published native-module pins here are still 0.2.0, which (per the PR description) does not expose the Azure SP surface. KERNEL_REV is bumped for kernel-e2e, but that file does not govern what end users npm install.

Net effect for a user on the published package who hits this combination: before this PR they got a clear HiveDriverError ("Azure-direct OAuth ... is not supported"); after this PR they instead get an opaque native failure from a 0.2.0 kernel that doesn't recognize AzureSpM2m. That's a regression in error clarity that ships the moment this merges, independent of the follow-up kernel release.

Consider gating the AzureSpM2m emission behind a kernel-capability/version check (falling back to the previous clear rejection), or holding this branch until the npm pins can be bumped in the same release, so end users never route to an authMode the installed kernel can't accept.

[peco-review-bot]

Verdict: 1 Low

Looks good — a clean, well-documented, well-tested change that threads Azure Entra auth (U2M + SP M2M) through the kernel path and correctly mirrors the Thrift OAuthManager.getManager routing (including the intentional .databricks.azure.us divergence, which I verified against lib/connection/auth/DatabricksOAuth/OAuthManager.ts). One low-severity note about missing test coverage for the new host-normalization logic that gates auth routing. The native/kernel/* changes are NAPI-RS regenerations tracking the new KERNEL_REV.

[peco-review-bot]

Verdict: 1 Low

Looks good — the Azure Entra SP M2M / U2M threading through the kernel path is well-structured, closely mirrors Thrift's OAuthManager.getManager, and is thoroughly unit-tested (routing, host normalization, blank/reserved credential hygiene, tenant auto-discovery fallback). Note: on an Azure host with a secret and no useDatabricksOAuthInAzure, M2M now routes to Entra-direct AzureSpM2m instead of the previous kernel behavior (generic workspace-OIDC OAuthM2m) — this is an intentional, CHANGELOG-documented alignment with Thrift, but is a silent routing change for existing kernel-on-Azure M2M setups. Only one low doc-accuracy nit inline. The large native/kernel/index.* churn is regenerated napi-rs output (CI asserts it matches KERNEL_REV), not a hand-edit.

[peco-review-bot]

Verdict: 1 Low

Looks good — the Azure Entra (U2M + SP M2M) kernel-path threading is correct, mirrors the Thrift OAuthManager.getManager routing, and is well covered by the new/updated unit tests (routing, host normalization, blank/reserved credential hygiene, tenant auto-discovery fallback). Comments match the code and the contract docblock/CHANGELOG were kept in sync. One low-severity note: the regenerated native/kernel/index.js/index.d.ts come from an apparently older napi-rs codegen than before — worth confirming the toolchain downgrade is intentional.

[peco-review-bot]

✅ No issues identified by the review bot.