[DCV-3894] Add Azure Key Vault secrets backend how-to docs

docs/how-tos/airflow/use-aws-secrets-manager.mdx

@@ -6,7 +6,7 @@ sidebar_position: 36
 ---
 # How to use AWS Secrets Manager in Airflow
 
-Datacoves integrates with the Airflow Secrets Backend Interface, offering support for both its native Datacoves Secrets Backend and AWS Secrets Manager. For other Airflow-compatible Secrets Managers, please reach out to us.
+Datacoves integrates with the Airflow Secrets Backend Interface, offering support for its native Datacoves Secrets Backend, AWS Secrets Manager, and Azure Key Vault. For other Airflow-compatible Secrets Managers, please reach out to us.
 
 Secrets backends can be configured at the project level, at the environment level, or both. See [configure your AWS Secrets Manager](/docs/how-tos/datacoves/how_to_projects/how_to_configure_aws_secrets_manager) for details.
 

docs/how-tos/airflow/use-azure-key-vault.mdx

@@ -0,0 +1,175 @@
+---
+title: Use Azure Key Vault for Airflow in Datacoves
+sidebar_label: Secrets - Azure Key Vault
+description: "Configure Azure Key Vault as the secrets backend for Apache Airflow in Datacoves to securely store and retrieve connections and variables using Managed Identity."
+sidebar_position: 37
+---
+# How to use Azure Key Vault in Airflow
+
+Datacoves integrates with the Airflow Secrets Backend Interface, offering support for its native Datacoves Secrets Backend, AWS Secrets Manager, and Azure Key Vault. For other Airflow-compatible Secrets Managers, please reach out to us.
+
+Secrets backends can be configured at the project level, at the environment level, or both. See [configure your Azure Key Vault](/docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault) for details.
+
+When Datacoves runs in your Azure subscription, Airflow can authenticate to Key Vault with the cluster's Managed Identity, so no credentials need to be stored anywhere. Alternatively, a service principal with a client secret can be used on any Datacoves deployment.
+
+## Read variable from Azure Key Vault
+
+Airflow's `Variable.get` searches multiple places:
+
+1. Azure Key Vault (if configured)
+2. Datacoves Secrets Manager
+3. Airflow variables and environment variables
+
+Once a variable is found, Airflow stops searching.
+
+### Secret naming
+
+Variable keys and connection ids must start with `datacoves-` for the lookup to be sent to Azure Key Vault. The Key Vault secret name is the variable key with the `airflow-variables-` prefix (or `airflow-connections-` for connections), and underscores are translated to dashes because Key Vault secret names only allow letters, numbers and dashes:
+
+| In your DAG | Key Vault secret name |
+| --- | --- |
+| `Variable.get("datacoves-my-secret")` | `airflow-variables-datacoves-my-secret` |
+| `Variable.get("datacoves-my_secret")` | `airflow-variables-datacoves-my-secret` |
+| Connection `datacoves-warehouse` | `airflow-connections-datacoves-warehouse` |
+
+### Best practices
+
+1. Call `Variable.get` from within an Airflow/Datacoves decorator to fetch at runtime only. Fetching at the top level of a DAG file would query Key Vault on every DAG parse.
+2. Keep the `datacoves-` prefix on everything you store in Key Vault for Airflow; lookups without it never reach the vault.
+
+### Example DAG using Azure Key Vault
+
+```python
+try:
+    # Airflow 3
+    from airflow.sdk import Variable, dag, task
+except ImportError:
+    # Airflow 2
+    from airflow.decorators import dag, task
+    from airflow.models import Variable
+
+from pendulum import datetime
+
+@dag(
+    catchup=False,
+    default_args={
+        "start_date": datetime(2024, 1, 1),
+        "owner": "Mayra Pena",
+        "email": "mayra@example.com",
+        "email_on_failure": True,
+    },
+    tags=["version_1"],
+    description="Read a variable from Azure Key Vault",
+    schedule="0 0 1 */12 *",
+)
+def azure_key_vault_example():
+
+    @task
+    def read_secret_from_key_vault():
+        # Fetch at runtime (inside the task), never at the top level of the
+        # DAG file, so Key Vault is only called when the task runs.
+        my_var = Variable.get("datacoves-my-secret")
+        print(f"Fetched a {len(my_var)} character value from Azure Key Vault")
+
+    read_secret_from_key_vault()
+
+dag = azure_key_vault_example()
+```
+
+:::tip
+To auto mask your secret you can use `secret` or `password` in the variable name since this will honor `hide_sensitive_var_conn_fields`. eg `datacoves-my-password`. Please see [this documentation](https://www.astronomer.io/docs/learn/airflow-variables#hide-sensitive-information-in-airflow-variables) for a full list of masking words.
+:::
+
+## Using Azure Key Vault directly from Airflow
+
+While not recommended, you can bypass the Datacoves secrets manager integration by configuring an Airflow connection and reading secrets with the Azure Key Vault SDK. The SDK (`azure-identity` and `azure-keyvault-secrets`) is already installed in Datacoves Airflow images as part of the Microsoft Azure provider.
+
+When reading secrets this way, the [secret naming](#secret-naming) rules above do not apply: you fetch any Key Vault secret by its exact name, with no `airflow-variables-` or `datacoves-` prefix required.
+
+### Configure an Airflow Connection
+
+Create a new Airflow Connection with the service principal credentials:
+
+Connection Id: `azure_key_vault`  
+Connection Type: Generic  
+Login: `<client id>`  
+Password: `<client secret>`  
+
+Extra:
+
+```json
+{
+  "tenant_id": "<tenant id>",
+  "vault_url": "https://<your-vault>.vault.azure.net/"
+}
+```
+
+### Example DAG reading Key Vault directly
+
+```python
+try:
+    # Airflow 3
+    from airflow.sdk import dag, task
+except ImportError:
+    # Airflow 2
+    from airflow.decorators import dag, task
+
+from airflow.hooks.base import BaseHook
+from pendulum import datetime
+
+@dag(
+    catchup=False,
+    default_args={
+        "start_date": datetime(2024, 1, 1),
+        "owner": "Noel Gomez",
+        "email": "noel@example.com",
+        "email_on_failure": True,
+    },
+    tags=["sample"],
+    description="Read a secret directly from Azure Key Vault",
+    schedule="0 0 1 */12 *",
+)
+def key_vault_direct_usage():
+
+    @task
+    def azure_secret():
+        from azure.identity import ClientSecretCredential
+        from azure.keyvault.secrets import SecretClient
+
+        conn = BaseHook.get_connection("azure_key_vault")
+        credential = ClientSecretCredential(
+            tenant_id=conn.extra_dejson["tenant_id"],
+            client_id=conn.login,
+            client_secret=conn.password,
+        )
+        client = SecretClient(
+            vault_url=conn.extra_dejson["vault_url"], credential=credential
+        )
+        secret = client.get_secret("my-secret-name")
+        print(f"Fetched a {len(secret.value)} character value from Azure Key Vault")
+
+    azure_secret()
+
+dag = key_vault_direct_usage()
+```
+
+:::tip
+When Datacoves runs in your Azure subscription with Managed Identity, no connection or credentials are needed at all: replace `ClientSecretCredential` with `DefaultAzureCredential()` from `azure.identity` and pass your vault URL to `SecretClient` directly.
+:::
+
+## Check when a secret is being fetched from Azure
+
+It is a good idea to verify that secrets are only being fetched when expected. To do this, enable diagnostic logging on your Key Vault:
+
+1. In the Azure Portal, go to your Key Vault
+2. Click `Diagnostic settings` and send the `Audit` (AuditEvent) category to a Log Analytics workspace
+3. In Log Analytics, query for `SecretGet` operations:
+
+```kusto
+AzureDiagnostics
+| where ResourceType == "VAULTS" and OperationName == "SecretGet"
+| project TimeGenerated, requestUri_s, identity_claim_appid_g, ResultSignature
+| order by TimeGenerated desc
+```
+
+Review the request URI (which contains the secret name) and the timestamp. Note: it may take a few minutes for events to show up in Log Analytics.

docs/how-tos/datacoves/how_to_projects/how_to_configure_azure_key_vault.md

@@ -0,0 +1,123 @@
+---
+title: "Add Azure Key Vault as a Datacoves Backend"
+sidebar_label: "Configure Azure Key Vault"
+description: "Connect Azure Key Vault to Datacoves as a project-level secrets backend so Airflow reads variables and connections directly from your vault, using Managed Identity or a service principal."
+sidebar_position: 52
+---
+# Configuring Azure Key Vault
+
+Datacoves can chain Azure Key Vault behind the Datacoves Secrets Backend so that Airflow fetches variables and connections directly from your vault at runtime. Secret values never pass through or get stored in Datacoves; Airflow talks to Azure Key Vault directly from within your environment.
+
+## Table of Contents
+- [Prereqs](#prereqs)
+- [Choose an authentication method](#choose-an-authentication-method)
+- [Create your Secret in Azure Key Vault](#create-your-secret-in-azure-key-vault)
+- [Configure your Secrets Backend](#configure-your-secrets-backend)
+  - [Project-level configuration](#project-level-configuration)
+  - [Environment-level configuration](#environment-level-configuration)
+
+## Prereqs
+
+1. An Azure Key Vault.
+2. An identity that Airflow can use to read secrets from the vault: either the Managed Identity already used by your Datacoves cluster (recommended when Datacoves runs in your Azure subscription) or a service principal with a client secret.
+3. That identity needs permission to read secrets on the vault. With Azure RBAC, assign the `Key Vault Secrets User` role scoped to the vault. If your vault uses access policies instead, grant the `Get` and `List` secret permissions.
+
+## Choose an authentication method
+
+### Managed Identity (recommended on Azure-hosted Datacoves)
+
+When your Datacoves cluster runs on AKS and Airflow is configured with an Azure identity (Workload Identity or a node-attached Managed Identity), no credentials are needed at all. The Airflow pods already carry the identity, so the backend configuration is just the vault URL:
+
+```json
+{
+  "vault_url": "https://<your-vault-name>.vault.azure.net/"
+}
+```
+
+Grant that Managed Identity the `Key Vault Secrets User` role on your vault and you are done. If you are not sure whether your cluster is set up this way, or which identity it uses, contact us at support@datacoves.com.
+
+:::note
+If your cluster uses a node-attached Managed Identity and the node carries more than one user-assigned identity, add `"managed_identity_client_id": "<client-id>"` to the configuration so the Azure SDK selects the right one.
+:::
+
+### Service principal with a client secret
+
+This works on any Datacoves deployment, including clusters that do not run on Azure.
+
+**Step 1:** In the Azure Portal, go to **Microsoft Entra ID** -> **App registrations** -> **New registration** and register an application (any name works, e.g. `datacoves-airflow-secrets`). No redirect URI is needed.
+
+**Step 2:** On the app's **Overview** page, copy the **Application (client) ID** and the **Directory (tenant) ID**.
+
+**Step 3:** Go to **Certificates & secrets** -> **New client secret**. Copy the secret's **Value** immediately after creating it (not the Secret ID); it is only shown once.
+
+**Step 4:** On your Key Vault, go to **Access control (IAM)** -> **Add role assignment**, pick the `Key Vault Secrets User` role, and under **Members** select **User, group, or service principal** and search for the app you registered.
+
+:::note
+If this role assignment is missing (or was created moments ago and has not propagated yet), secret lookups fail with `Forbidden: Caller is not authorized to perform action on resource` even though authentication itself succeeded. The same applies to the Managed Identity variant. Role assignments can take a few minutes to become effective.
+:::
+
+The backend configuration is:
+
+```json
+{
+  "vault_url": "https://<your-vault-name>.vault.azure.net/",
+  "tenant_id": "<directory-tenant-id>",
+  "client_id": "<application-client-id>",
+  "client_secret": "<client-secret-value>"
+}
+```
+
+## Create your Secret in Azure Key Vault
+
+:::note
+With the (recommended) Azure RBAC permission model, data plane roles are separate from management roles: even the subscription Owner cannot create or view secrets until they assign themselves a data plane role. If you see "The operation is not allowed by RBAC" or "You are unauthorized to view these contents" on the vault's Secrets page, assign yourself the `Key Vault Secrets Officer` role under **Access control (IAM)** and wait a few minutes for the role assignment to propagate. Airflow's identity only needs the read-only `Key Vault Secrets User` role, not Officer. Also watch out for the similarly named certificate roles when searching: `Key Vault Certificates Officer` and `Key Vault Certificate User` grant access to certificates, not secrets, and picking them by mistake leads to generic "An error occurred while creating the secret" failures.
+:::
+
+Airflow maps variables and connections to Key Vault secret names using a prefix:
+
+- Variable `<key>` is read from the secret named `airflow-variables-<key>`
+- Connection `<conn_id>` is read from the secret named `airflow-connections-<conn_id>`
+
+### Things to note:
+
+1. Variable keys and connection ids must start with `datacoves-`. The Datacoves Secrets Backend only forwards lookups with that prefix to the additional backend; anything else is resolved from Airflow's own variables and connections. For example, to use `Variable.get("datacoves-my-secret")` in a DAG, create a Key Vault secret named `airflow-variables-datacoves-my-secret`.
+2. Azure Key Vault secret names only allow letters, numbers and dashes. Underscores in a variable key or connection id are automatically translated to dashes when building the secret name, so `Variable.get("datacoves-my_secret")` also reads the secret `airflow-variables-datacoves-my-secret`.
+3. Store the value as plain text. For connections, store either an Airflow connection URI (for example `snowflake://user:password@account/`) or a JSON object with the connection fields.
+
+## Configure your Secrets Backend
+
+Azure Key Vault can be configured at the project level, at the environment level, or both. When configured at the project level, all environments under that project will use it by default. Individual environments can have their own configuration that takes precedence, or they can be set to inherit the project-level settings.
+
+### Project-level configuration
+
+This configuration applies to all environments under the project unless overridden at the environment level.
+
+**Step 1:** Navigate to the Projects Admin page and click on the edit icon for the desired project.
+
+**Step 2:** Scroll down to the `Secrets` section and select `Azure Key Vault` from the `Additional Secrets Backend` dropdown.
+
+**Step 3:** Paste the JSON configuration for the authentication method you chose above.
+
+:::tip
+See the [Azure Key Vault secrets backend documentation](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/secrets-backends/azure-key-vault.html) for more customization options, such as `connections_prefix`, `variables_prefix` and `sep`.
+:::
+
+:::tip
+For security purposes, once this has been saved you will not be able to view the values. To modify the Secrets backend you will need to set the Secrets backend to `None` and save the changes. Then start the setup again.
+:::
+
+### Environment-level configuration
+
+Azure Key Vault can also be configured directly at the environment level, independently of the project settings. This is useful when only specific environments should use Azure Key Vault, or when different environments need different vaults (for example, one vault for development and another for production).
+
+**Step 1:** Navigate to the Environments Admin page and click on the edit icon for the desired environment.
+
+**Step 2:** Go to **Services Configuration**, then select **Airflow settings**.
+
+**Step 3:** Scroll down to the **Additional Secrets Backend** section. Select `Azure Key Vault` to configure it for this environment. If a project-level configuration exists and you want this environment to use it, leave the field set to `Use Project Settings`.
+
+:::note
+The configuration fields available at the environment level are the same as those at the project level. Any values entered here will take precedence over the project settings for this environment only.
+:::
+
+To learn how to read a variable from Azure Key Vault in a DAG, check out [How to use Azure Key Vault in Airflow](/docs/how-tos/airflow/use-azure-key-vault).