feat(kernel): JWT private-key M2M auth on use_kernel=True (#921)

* feat(kernel): support JWT private-key M2M auth on use_kernel=True

Route JWT private-key client-assertion auth (RFC 7523) through the kernel
backend. When the caller passes `oauth_jwt_key_file` (+ `oauth_client_id`
and `oauth_jwt_kid`, optional `oauth_jwt_passphrase` / `oauth_jwt_algorithm`
/ `oauth_scopes` / `token_url`), the bridge forwards them to the kernel's
`auth_type="oauth-m2m-jwt"`, which signs a short-lived assertion with the
private key instead of sending a client secret and owns the token lifecycle.

- auth_bridge.py: new JWT branch (checked before shared-secret M2M and PAT,
  since a private-key file is unambiguous JWT M2M intent); mutually exclusive
  with oauth_client_secret / credentials_provider; requires client_id + kid.
- session.py: forward the new oauth_jwt_* / token_url kwargs into the kernel
  auth options.
- tests: 9 unit tests covering routing, precedence, validation, and
  ambiguity guards.

Verified end-to-end: `SELECT 1` via use_kernel=True against an Azure
Databricks warehouse, authenticated by Entra ID against the service
principal's registered public certificate.

Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>

* fix(kernel): annotate kwargs at first assignment to satisfy mypy

The JWT branch introduced an earlier untyped `kwargs =`, so mypy flagged
the M2M branch's `kwargs: Dict[str, Any]` as a redefinition. Move the
annotation to the first (JWT) assignment.

Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>

* fix(kernel): guard JWT M2M + databricks-oauth auth_type; renumber comments

Address peco-review-bot review on #921:
- Medium: add the missing ambiguity guard for oauth_jwt_key_file +
  auth_type="databricks-oauth" (U2M intent), mirroring the existing
  shared-secret M2M + U2M guard. Fails loudly rather than silently
  resolving to one flow. Covered by a new unit test.
- Low: renumber the inline resolution-order comments (PAT→3, U2M→4,
  creds→5, else→6) to match the docstring after the JWT branch insert.

Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>

* docs: document JWT private-key M2M params in CONNECTION_PARAMETERS.md

Address Eric's review on #921: add rows for oauth_jwt_key_file /
oauth_jwt_kid / oauth_jwt_passphrase / oauth_jwt_algorithm / token_url
to the connection-parameter reference (all kernel-only).

Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>

* security: suppress unreachable cryptography PKCS#7 advisory (osv-scanner)

Security Scan flagged cryptography@49.0.0 CVE-2026-69247
(GHSA-g6cj-pr64-35w5 / PYSEC-2026-3552, CVSS 8.2) — a Bleichenbacher-style
oracle in PKCS#7 EnvelopedData / S-MIME decryption. `cryptography` is a
transitive dep (PyJWT[crypto] / oauthlib) used only for OAuth token
signing/verification; the connector never decrypts PKCS#7 / S-MIME, so the
vulnerable path is never invoked. Pre-existing on main (transitive, not
introduced by this PR). Suppress both OSV ids with justification; a clean
bump awaits PyJWT/oauthlib floors pulling cryptography>=50.

Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>

* fix(kernel): forward token_url on shared-secret M2M too (JDBC parity)

Address peco-review-bot: the CONNECTION_PARAMETERS.md row said token_url
"applies to shared-secret M2M too", but the bridge only forwarded it on
the JWT branch — so an oauth_client_id + oauth_client_secret + token_url
connection silently dropped token_url.

token_url is an auth-method-agnostic token-endpoint override: JDBC's
OAuth2ConnAuthTokenEndpoint is consumed by the client-secret M2M provider,
the JWT provider, and the refresh provider alike; the Node driver also
forwards it on both branches; and the kernel's oauth-m2m auth_type accepts
it (pyo3). Forward it on the shared-secret M2M branch so code matches the
doc and the other drivers. Adds a unit test.

Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>

---------

Signed-off-by: Rahul Singhal <rahul.singhal@databricks.com>

Author: rahuls-db

CHANGELOG.md

@@ -1,6 +1,7 @@
 # Release History
 
 # Unreleased
+- Kernel backend (`use_kernel=True`): OAuth **M2M with a JWT private-key client assertion** (RFC 7523) is now supported. Pass `oauth_client_id` + `oauth_jwt_key_file` + `oauth_jwt_kid` (with optional `oauth_jwt_passphrase` for an encrypted PKCS#8 key, `oauth_jwt_algorithm` defaulting to `RS256`, `oauth_scopes`, and `token_url` for the IdP token endpoint) and the connector routes them to the kernel's `auth_type="oauth-m2m-jwt"`, which signs a short-lived assertion with the private key instead of sending a client secret. The kernel owns the token lifecycle. A private-key file is treated as unambiguous JWT M2M intent and is mutually exclusive with `oauth_client_secret` / `credentials_provider` (both raise `NotSupportedError`). Verified end-to-end against an Azure Databricks workspace with the service principal's public certificate registered on its Entra ID app registration. Requires `databricks-sql-kernel >= 0.2.0` with JWT support.
 - Kernel backend (`use_kernel=True`): OAuth U2M with `auth_type="databricks-oauth"` now forwards the connector's `databricks-sql-python` OAuth-app bundle (`client_id` + `sql offline_access` scopes + redirect port) into the kernel, so a bare U2M connection authenticates as `databricks-sql-python` — parity with the Thrift path — instead of inheriting the kernel's own `databricks-sql-connector` default. A caller-supplied `oauth_client_id` (with its coupled `oauth_redirect_port`) is honored, as is a caller-supplied `oauth_scopes`; absent one, the connector default (`sql offline_access`) is forwarded. Note: the kernel binds a single U2M redirect port, so unlike the Thrift path (which tries the full `8020..8024` range) the kernel path uses only one port and does not fall back to the next port if it is already bound — pass `oauth_redirect_port` (with `oauth_client_id`) to pick a free one on a port collision. `auth_type="azure-oauth"` (Azure AD) is not yet supported on the kernel path and raises `NotSupportedError` — use the Thrift backend for it (PECOBLR-4040; Azure tracked by PECOBLR-4120)
 
 # 4.4.0 (2026-07-22)

CONNECTION_PARAMETERS.md

@@ -73,6 +73,11 @@ to change without notice.
 | `oauth_client_id` (U2M)                             | `str`                |   ✅   |   ✅   | built-in client id        | Custom U2M client id. Forwarded on both; when absent, each path applies its own built-in default.                                                              |
 | `oauth_redirect_port` (U2M)                         | `int`                |   ✅   |   ✅   | `None`                    | Localhost redirect port for the browser flow. On **both** backends it is only honored when a custom `oauth_client_id` is also supplied — then that single port becomes the redirect URI. With the built-in client id (or when omitted) the connector uses the full registered range 8020–8024 and binds the first free port, so a bare `oauth_redirect_port` has no effect. (Thrift: `auth.py` `oauth_redirect_port_range`; Kernel: same logic, forwarded as `redirect_ports`.) |
 | `oauth_client_secret` (OAuth M2M)                   | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only in practice.** The Thrift auth path never reads `oauth_client_secret`; use `credentials_provider` or an Azure service principal for M2M on Thrift. |
+| `oauth_jwt_key_file` (OAuth M2M, JWT private key)   | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Path to the PEM private key for JWT private-key M2M (RFC 7523 client assertion). Supplying it selects the JWT flow: the kernel signs a short-lived assertion with the key instead of sending a client secret. Requires `oauth_client_id` + `oauth_jwt_kid`; mutually exclusive with `oauth_client_secret` / `credentials_provider`. |
+| `oauth_jwt_kid` (OAuth M2M, JWT private key)        | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Key id written into the JWT header so the IdP can select the registered public key. Required with `oauth_jwt_key_file`. (For Entra ID this is the certificate's `x5t` thumbprint.) |
+| `oauth_jwt_passphrase` (OAuth M2M, JWT private key) | `str`                |   ❌   |   ✅   | `None`                    | **Kernel-only.** Passphrase for an encrypted PKCS#8 private key; omit for an unencrypted key.                                                                    |
+| `oauth_jwt_algorithm` (OAuth M2M, JWT private key)  | `str`                |   ❌   |   ✅   | `RS256`                   | **Kernel-only.** JWT signing algorithm (`RS256`/`384`/`512`, `PS256`/`384`/`512`, `ES256`, `ES384`).                                                            |
+| `token_url` (OAuth M2M)                             | `str`                |   ❌   |   ✅   | `None` ⇒ OIDC discovery   | **Kernel-only.** OAuth IdP token endpoint override. Required for JWT M2M against an external-IdP-backed workspace (e.g. Entra ID for Azure Databricks), since Databricks-native OIDC does not advertise the `private_key_jwt` method. Applies to shared-secret M2M too. |
 | `oauth_scopes`                                      | `List[str]`          |   ❌   |   ✅   | `["sql","offline_access"]`| **Thrift ignores custom scopes** — it always uses the built-in scope set. Only the kernel honors a custom `oauth_scopes`.                                       |
 | `credentials_provider`                              | `CredentialsProvider`|   ✅   |   ❌   | `None`                    | Custom external credentials provider. **Rejected on the kernel path** (`NotSupportedError`) — it is an opaque token source, so the kernel cannot own the token lifecycle; use `oauth_client_id` + `oauth_client_secret` for M2M, or the Thrift backend. |
 | `identity_federation_client_id`                     | `str`                |   ✅   |   ✅   | `None`                    | Workload identity / token-federation client id (kernel support added in #910).                                                                                 |

osv-scanner.toml

@@ -16,3 +16,26 @@
 # This file starts empty -- populate iteratively as the first scan run
 # surfaces real false positives. Do not pre-populate with speculative
 # suppressions.
+
+[[IgnoredVulns]]
+id = "GHSA-g6cj-pr64-35w5"
+# CVE-2026-69247 / PYSEC-2026-3552 -- a Bleichenbacher-style timing/error
+# oracle in cryptography's PKCS#7 *EnvelopedData* decryption
+# (pkcs7_decrypt_der/pem/smime), exploitable only against an endpoint that
+# auto-decrypts attacker-supplied S/MIME EnvelopedData and responds
+# adaptively. Affects cryptography >=44.0.0,<50.0.0; fixed in 50.0.0.
+#
+# Not reachable here: `cryptography` is a transitive dependency (via
+# PyJWT[crypto] / oauthlib, for OAuth token signing/verification). The
+# connector never decrypts PKCS#7 / S/MIME EnvelopedData -- the vulnerable
+# code path is never invoked. Pre-existing on main (transitive, not
+# introduced by any connector change); a clean bump awaits PyJWT/oauthlib
+# floors that pull cryptography>=50. Re-evaluate when the dependency tree
+# resolves to a fixed cryptography.
+
+[[IgnoredVulns]]
+# Same advisory under its PYSEC alias -- OSV reports both ids for this
+# finding, so suppress both to be safe (see GHSA-g6cj-pr64-35w5 above for
+# the full justification: unreachable PKCS#7/S-MIME decryption oracle in a
+# transitive dependency).
+id = "PYSEC-2026-3552"

src/databricks/sql/backend/kernel/auth_bridge.py

@@ -156,11 +156,16 @@ def kernel_auth_kwargs(
 
     (``azure-oauth`` is rejected as unsupported before these guards —
     PECOBLR-4120.)
-    1. **OAuth M2M** — ``oauth_client_id`` + ``oauth_client_secret``
+    1. **OAuth M2M (JWT private key)** — ``oauth_jwt_key_file`` present →
+       forward the private-key + ``oauth_client_id`` + ``oauth_jwt_kid``
+       to the kernel's ``oauth-m2m-jwt`` (RFC 7523 client assertion). The
+       kernel signs the assertion and owns the token lifecycle. Checked
+       first because a private-key file is unambiguous JWT M2M intent.
+    2. **OAuth M2M** — ``oauth_client_id`` + ``oauth_client_secret``
        both present → forward raw creds to the kernel's ``oauth-m2m``.
-    2. **PAT** — the built provider is (or wraps) an
+    3. **PAT** — the built provider is (or wraps) an
        ``AccessTokenAuthProvider`` → extract the bearer token.
-    3. **OAuth U2M** — ``auth_type`` is ``databricks-oauth`` → forward the
+    4. **OAuth U2M** — ``auth_type`` is ``databricks-oauth`` → forward the
        connector's coupled ``databricks-sql-python`` bundle (``client_id``
        + ``redirect_ports`` list, defaulting scopes to ``PYSQL_OAUTH_SCOPES``
        when the caller supplies none) to the kernel's ``oauth-u2m``, so a
@@ -169,9 +174,9 @@ def kernel_auth_kwargs(
        ``databricks-sql-connector`` default (PECOBLR-4039/4040). Unlike the
        Thrift path, a caller-supplied ``oauth_scopes`` is honored here.
        ``azure-oauth`` is rejected as unsupported (PECOBLR-4120).
-    4. **Custom credentials_provider** → ``NotSupportedError`` (opaque
+    5. **Custom credentials_provider** → ``NotSupportedError`` (opaque
        token source; no raw creds for the kernel to own).
-    5. Anything else → ``NotSupportedError``.
+    6. Anything else → ``NotSupportedError``.
 
     M2M is checked before PAT so that a workload passing both an
     access token *and* M2M creds resolves to the (refreshing) M2M path
@@ -186,7 +191,12 @@ def kernel_auth_kwargs(
     client_secret = opts.get("oauth_client_secret")
     federation_client_id = opts.get("identity_federation_client_id")
     auth_type = opts.get("auth_type")
+    jwt_key_file = opts.get("oauth_jwt_key_file")
     has_m2m = bool(client_id and client_secret)
+    # A private-key file is unambiguous JWT client-assertion M2M intent
+    # (RFC 7523): the kernel signs a short-lived assertion with the key
+    # rather than sending a client secret.
+    has_jwt_m2m = bool(jwt_key_file)
 
     # azure-oauth (Azure AD U2M) is not yet supported on the kernel path.
     # Reject it up front — before any M2M/U2M routing — so ANY azure-oauth
@@ -223,22 +233,96 @@ def kernel_auth_kwargs(
             "(machine-to-machine). Drop oauth_client_secret for U2M, or drop "
             "auth_type for M2M."
         )
+    if has_jwt_m2m and client_secret:
+        raise NotSupportedError(
+            "Ambiguous auth on use_kernel=True: both oauth_jwt_key_file "
+            "(JWT private-key M2M) and oauth_client_secret (shared-secret "
+            "M2M) were provided. Pass exactly one — a private key for "
+            "JWT client-assertion M2M, or a client secret for shared-secret M2M."
+        )
+    if has_jwt_m2m and opts.get("credentials_provider") is not None:
+        raise NotSupportedError(
+            "Ambiguous auth on use_kernel=True: both a custom "
+            "credentials_provider and oauth_jwt_key_file were provided. "
+            "Pass exactly one — oauth_client_id + oauth_jwt_key_file for "
+            "kernel-managed JWT private-key M2M, or use the Thrift backend "
+            "(default) for credentials_provider."
+        )
+    if has_jwt_m2m and auth_type == "databricks-oauth":
+        raise NotSupportedError(
+            f"Ambiguous auth on use_kernel=True: auth_type={auth_type!r} selects "
+            "the U2M browser flow, but oauth_jwt_key_file was also provided "
+            "(JWT private-key M2M). Drop oauth_jwt_key_file for U2M, or drop "
+            "auth_type for JWT M2M."
+        )
 
-    # 1. OAuth M2M — raw client-credentials pair forwarded to the kernel.
-    if has_m2m:
+    # 1. OAuth M2M (JWT private-key client assertion) — the kernel signs a
+    #    short-lived assertion with the private key and runs the
+    #    client-credentials grant. Checked before shared-secret M2M and PAT
+    #    because a private-key file is unambiguous JWT M2M intent. Requires
+    #    oauth_client_id (the service principal / OAuth client) and
+    #    oauth_jwt_kid (the key id the IdP uses to select the registered
+    #    public key). Optional oauth_jwt_passphrase / oauth_jwt_algorithm /
+    #    oauth_scopes / token_url are forwarded when present; the kernel
+    #    fills defaults (RS256 algorithm, all-apis scope, OIDC discovery)
+    #    for any omitted.
+    if has_jwt_m2m:
+        if not client_id:
+            raise ProgrammingError(
+                "use_kernel=True JWT private-key M2M (oauth_jwt_key_file) "
+                "requires oauth_client_id (the service principal / OAuth "
+                "client id used as the assertion issuer and subject)."
+            )
+        jwt_kid = opts.get("oauth_jwt_kid")
+        if not jwt_kid:
+            raise ProgrammingError(
+                "use_kernel=True JWT private-key M2M (oauth_jwt_key_file) "
+                "requires oauth_jwt_kid (the key id written into the JWT "
+                "header so the IdP can select the registered public key)."
+            )
         kwargs: Dict[str, Any] = {
+            "auth_type": "oauth-m2m-jwt",
+            "client_id": client_id,
+            "jwt_key_file": jwt_key_file,
+            "jwt_kid": jwt_kid,
+        }
+        jwt_passphrase = opts.get("oauth_jwt_passphrase")
+        if jwt_passphrase:
+            kwargs["jwt_passphrase"] = jwt_passphrase
+        jwt_algorithm = opts.get("oauth_jwt_algorithm")
+        if jwt_algorithm:
+            kwargs["jwt_algorithm"] = jwt_algorithm
+        token_url = opts.get("token_url")
+        if token_url:
+            kwargs["token_url"] = token_url
+        scopes = _normalize_scopes(opts.get("oauth_scopes"))
+        if scopes is not None:
+            kwargs["oauth_scopes"] = scopes
+        if federation_client_id:
+            kwargs["identity_federation_client_id"] = federation_client_id
+        return kwargs
+
+    # 2. OAuth M2M — raw client-credentials pair forwarded to the kernel.
+    if has_m2m:
+        kwargs = {
             "auth_type": "oauth-m2m",
             "client_id": client_id,
             "client_secret": client_secret,
         }
         scopes = _normalize_scopes(opts.get("oauth_scopes"))
         if scopes is not None:
             kwargs["oauth_scopes"] = scopes
+        # token_url is an auth-method-agnostic token-endpoint override (JDBC's
+        # OAuth2ConnAuthTokenEndpoint applies it to client-secret M2M too), so
+        # forward it here as well as on the JWT path.
+        token_url = opts.get("token_url")
+        if token_url:
+            kwargs["token_url"] = token_url
         if federation_client_id:
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs
 
-    # 2. PAT (including TokenFederationProvider-wrapped PAT).
+    # 3. PAT (including TokenFederationProvider-wrapped PAT).
     if _is_pat(auth_provider):
         token = _extract_bearer_token(auth_provider)
         if not token:
@@ -251,7 +335,7 @@ def kernel_auth_kwargs(
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs
 
-    # 3. OAuth U2M — browser authorization-code flow; the kernel runs it.
+    # 4. OAuth U2M — browser authorization-code flow; the kernel runs it.
     #    Only databricks-oauth reaches here (azure-oauth rejected up front).
     #    Forward the connector's own databricks-sql-python bundle instead of
     #    the kernel's databricks-sql-connector default, for parity with the
@@ -283,7 +367,7 @@ def kernel_auth_kwargs(
             kwargs["identity_federation_client_id"] = federation_client_id
         return kwargs
 
-    # 4. Custom credentials_provider — the connector's primary M2M path
+    # 5. Custom credentials_provider — the connector's primary M2M path
     #    on Thrift/SEA, but unusable on the kernel: it's an opaque token
     #    source with no extractable client_id/secret, so the kernel
     #    can't own the token lifecycle. Point the caller at the raw
@@ -297,7 +381,7 @@ def kernel_auth_kwargs(
             "credentials_provider."
         )
 
-    # 5. Everything else (including no usable credentials at all —
+    # 6. Everything else (including no usable credentials at all —
     #    ``auth_provider`` is None on the kernel path when no access
     #    token was supplied and no OAuth kwargs resolved above).
     provider_desc = (

src/databricks/sql/session.py

@@ -173,6 +173,15 @@ def _create_backend(
                 "oauth_client_secret": kwargs.get("oauth_client_secret"),
                 "oauth_redirect_port": kwargs.get("oauth_redirect_port"),
                 "oauth_scopes": kwargs.get("oauth_scopes"),
+                # JWT private-key M2M (RFC 7523 client assertion): the kernel
+                # signs a short-lived assertion with the private key instead
+                # of sending a client secret. token_url points the assertion
+                # at the workspace's OAuth IdP token endpoint (e.g. Entra ID).
+                "oauth_jwt_key_file": kwargs.get("oauth_jwt_key_file"),
+                "oauth_jwt_kid": kwargs.get("oauth_jwt_kid"),
+                "oauth_jwt_passphrase": kwargs.get("oauth_jwt_passphrase"),
+                "oauth_jwt_algorithm": kwargs.get("oauth_jwt_algorithm"),
+                "token_url": kwargs.get("token_url"),
                 "credentials_provider": kwargs.get("credentials_provider"),
                 "identity_federation_client_id": kwargs.get(
                     "identity_federation_client_id"