From 80358df27385fc4e06732d3ff0e08e593fdbb7de Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 21 Oct 2026 12:03:04 -0700 Subject: [PATCH 1/3] fix(finops-hub): Grant Data Factory Fabric admin and warn on storage firewall Deploy-FinOpsHub now grants the Data Factory identity admin on the Ingestion and Hub databases when -FabricQueryUri is set, and warns that Fabric ingestion cannot pass the storage firewall in vnet/private modes (#2061). Informed by zhshah/Azure-FinOps-Automation-Deployment-Utility (MIT) by Zahir Hussain Shah. Co-Authored-By: Claude Sonnet 5.5 --- docs-mslearn/toolkit/changelog.md | 1 + docs-mslearn/toolkit/hubs/deploy.md | 7 ++- .../Private/Grant-FinOpsHubFabricAccess.ps1 | 56 +++++++++++++++++++ src/powershell/Public/Deploy-FinOpsHub.ps1 | 15 ++++- .../Grant-FinOpsHubFabricAccess.Tests.ps1 | 35 ++++++++++++ 5 files changed, 111 insertions(+), 3 deletions(-) create mode 100644 src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 create mode 100644 src/powershell/Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index 7be0356d5..9723129ab 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -35,6 +35,7 @@ The following section lists features and enhancements that are currently in deve - Replaced whole-term `contains` matches with `has` across hub KQL and the query catalog (resource ID paths, licensing phrases, SKU description terms) and added a per-row operator-equivalence regression harness with unit test coverage ([#2220](https://github.com/microsoft/finops-toolkit/pull/2220)). - Documented how the hub schema's columns nest in the hub database guide - the organization chain and the two parallel service chains - and brought the `Costs()` column reference back in line with the deployed schema by adding 27 missing columns, correcting three columns still listed under the names hubs v12 renamed away from, and fixing one column typed as `long` instead of `int` ([#2325](https://github.com/microsoft/finops-toolkit/issues/2325), [#2329](https://github.com/microsoft/finops-toolkit/issues/2329)). - **Fixed** + - Fixed Fabric eventhouse deployments leaving the Data Factory identity without access to the Ingestion and Hub databases. `Deploy-FinOpsHub -FabricQueryUri` now grants the admin role automatically, and warns that Fabric ingestion cannot pass the storage firewall in VNet or private modes. Approach informed by [Zahir Hussain Shah](https://github.com/zhshah/Azure-FinOps-Automation-Deployment-Utility) (MIT) ([#2061](https://github.com/microsoft/finops-toolkit/issues/2061)). - Fixed ADF schedule trigger deployment failures (`InvalidWorkflowTriggerRecurrence`) in regions not covered by the region-to-time-zone map (for example, Sweden Central), where the trigger's `startTime` was missing the required `Z` suffix for the `UTC` fallback time zone ([#2157](https://github.com/microsoft/finops-toolkit/issues/2157), [#2291](https://github.com/microsoft/finops-toolkit/pull/2291)). - Fixed managed exports failing with an `Unauthorized` error because the Role Based Access Control Administrator role was never assigned to the Data Factory identity. Roles requested by a hub app are now assigned on the publisher storage account even when the app doesn't create the storage account itself ([#2253](https://github.com/microsoft/finops-toolkit/issues/2253)). - Fixed private-network deployments that Azure Policy blocked when `defaultOutboundAccess` was omitted. Private mode subnets now set it to `false`, while an Azure Files private endpoint supports deployment-script storage and the NAT Gateway provides required container egress ([#2258](https://github.com/microsoft/finops-toolkit/issues/2258), [#2259](https://github.com/microsoft/finops-toolkit/pull/2259)). diff --git a/docs-mslearn/toolkit/hubs/deploy.md b/docs-mslearn/toolkit/hubs/deploy.md index 3a7e853ab..536cf9540 100644 --- a/docs-mslearn/toolkit/hubs/deploy.md +++ b/docs-mslearn/toolkit/hubs/deploy.md @@ -3,7 +3,7 @@ title: How to create and update FinOps hubs description: This tutorial helps you create a new or update an existing FinOps hubs instance in Azure or Microsoft Fabric. author: flanakin ms.author: micflan -ms.date: 08/19/2026 +ms.date: 10/21/2026 ms.topic: tutorial ms.service: finops ms.subservice: finops-toolkit @@ -210,7 +210,10 @@ For other parameters, see [Deploy-FinOpsHub](../powershell/hubs/Deploy-FinOpsHub ## Optional: Configure Fabric access -If you set up Microsoft Fabric as a primary data store, configure access for Data Factory and the Fabric eventhouse. +If you set up Microsoft Fabric as a primary data store, configure access for Data Factory and the Fabric eventhouse. `Deploy-FinOpsHub -FabricQueryUri` grants the Data Factory access in step 2 for you; follow these steps if you deploy the template another way. + +> [!WARNING] +> Fabric eventhouse ingestion pulls files from the storage account and cannot pass the storage firewall (trusted services bypass doesn't apply). With private or VNet networking, data stays in storage until the firewall allows Fabric access. 1. Get the Data Factory identity: 1. From the Azure portal, open the FinOps hub resource group. diff --git a/src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 b/src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 new file mode 100644 index 000000000..aac622ffb --- /dev/null +++ b/src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 @@ -0,0 +1,56 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +<# + .SYNOPSIS + Grants the FinOps hub Data Factory identity admin access to the Fabric eventhouse Ingestion and Hub databases. + + .PARAMETER QueryUri + Required. Fabric eventhouse query URI. + + .PARAMETER PrincipalId + Required. Object ID of the Data Factory managed identity. + + .PARAMETER TenantId + Required. Entra tenant ID. + + .EXAMPLE + Grant-FinOpsHubFabricAccess -QueryUri 'https://xxx.kusto.fabric.microsoft.com' -PrincipalId $adfId -TenantId $tenantId +#> +function Grant-FinOpsHubFabricAccess +{ + [CmdletBinding(SupportsShouldProcess)] + param + ( + [Parameter(Mandatory = $true)] + [string] + $QueryUri, + + [Parameter(Mandatory = $true)] + [string] + $PrincipalId, + + [Parameter(Mandatory = $true)] + [string] + $TenantId + ) + + foreach ($database in 'Ingestion', 'Hub') + { + if (-not $PSCmdlet.ShouldProcess($database, 'Add Data Factory identity as database admin')) + { + continue + } + + try + { + $token = (Get-AzAccessToken -ResourceUrl 'https://kusto.kusto.windows.net').Token + $body = @{ db = $database; csl = ".add database $database admins ('aadapp=$PrincipalId;$TenantId')" } | ConvertTo-Json + Invoke-RestMethod -Method Post -Uri "$($QueryUri.TrimEnd('/'))/v1/rest/mgmt" -Headers @{ Authorization = "Bearer $token" } -ContentType 'application/json' -Body $body | Out-Null + } + catch + { + Write-Warning "Could not grant the Data Factory identity admin on the $database database. Run the .add database command manually (see https://aka.ms/finops/hubs/deploy#optional-configure-fabric-access). $($_.Exception.Message)" + } + } +} diff --git a/src/powershell/Public/Deploy-FinOpsHub.ps1 b/src/powershell/Public/Deploy-FinOpsHub.ps1 index 6140e0af8..45aba7e01 100644 --- a/src/powershell/Public/Deploy-FinOpsHub.ps1 +++ b/src/powershell/Public/Deploy-FinOpsHub.ps1 @@ -259,6 +259,11 @@ function Deploy-FinOpsHub # Resolve network mode. -NetworkMode wins; -DisablePublicAccess (deprecated) maps to 'vnet'. $effectiveNetworkMode = if ($NetworkMode) { $NetworkMode } elseif ($DisablePublicAccess) { 'vnet' } else { 'public' } + + if ($FabricQueryUri -and $effectiveNetworkMode -ne 'public') + { + Write-Warning 'Fabric eventhouse ingestion pulls files from storage and cannot pass the storage firewall, so data stays in storage until the firewall allows Fabric. See https://github.com/microsoft/finops-toolkit/issues/2061.' + } if (-not $NetworkMode -and $DisablePublicAccess) { Write-Warning "-DisablePublicAccess is deprecated; use -NetworkMode 'vnet' or -NetworkMode 'private'." } # Private network mode (NAT Gateway) requires hub template version 15.0 or later @@ -344,7 +349,15 @@ function Deploy-FinOpsHub if (Test-ShouldProcess $PSCmdlet $ResourceGroupName 'DeployFinOpsHub') { Write-Verbose -Message ($LocalizedData.Hub_Deploy_Deploy -f $bicepFile.FullName, $resourceGroupObject.ResourceGroupName) - return New-AzResourceGroupDeployment @parameterSplat -ResourceGroupName $resourceGroupObject.ResourceGroupName + $deployment = New-AzResourceGroupDeployment @parameterSplat -ResourceGroupName $resourceGroupObject.ResourceGroupName + + if ($FabricQueryUri -and $deployment.Outputs -and $deployment.Outputs.dataFactoryName) + { + $adf = Get-AzDataFactoryV2 -ResourceGroupName $resourceGroupObject.ResourceGroupName -Name $deployment.Outputs.dataFactoryName.Value + Grant-FinOpsHubFabricAccess -QueryUri $FabricQueryUri -PrincipalId $adf.Identity.PrincipalId -TenantId $adf.Identity.TenantId -WhatIf:$WhatIfPreference + } + + return $deployment } } catch diff --git a/src/powershell/Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 b/src/powershell/Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 new file mode 100644 index 000000000..e0ead37ff --- /dev/null +++ b/src/powershell/Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 @@ -0,0 +1,35 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. + +& "$PSScriptRoot/../Initialize-Tests.ps1" + +InModuleScope 'FinOpsToolkit' { + Describe 'Grant-FinOpsHubFabricAccess' { + BeforeAll { + function Get-AzAccessToken { param($ResourceUrl, $AsSecureString) } + Mock -CommandName 'Get-AzAccessToken' { [pscustomobject]@{ Token = 'token' } } + Mock -CommandName 'Invoke-RestMethod' { } + } + + It 'Should grant admin on both databases to the Data Factory identity' { + Grant-FinOpsHubFabricAccess -QueryUri 'https://x.kusto.fabric.microsoft.com/' -PrincipalId 'p-id' -TenantId 't-id' -Confirm:$false + + Should -Invoke -CommandName 'Invoke-RestMethod' -Times 2 + Should -Invoke -CommandName 'Invoke-RestMethod' -Times 1 -ParameterFilter { + $Uri -eq 'https://x.kusto.fabric.microsoft.com/v1/rest/mgmt' -and $Body -like "*.add database Ingestion admins ('aadapp=p-id;t-id')*" } + Should -Invoke -CommandName 'Invoke-RestMethod' -Times 1 -ParameterFilter { $Body -like "*.add database Hub admins ('aadapp=p-id;t-id')*" } + } + + It 'Should not call the eventhouse with -WhatIf' { + Grant-FinOpsHubFabricAccess -QueryUri 'https://x/' -PrincipalId 'p' -TenantId 't' -WhatIf + + Should -Invoke -CommandName 'Invoke-RestMethod' -Times 0 + } + + It 'Should warn and not throw when the grant fails' { + Mock -CommandName 'Invoke-RestMethod' { throw 'denied' } + + { Grant-FinOpsHubFabricAccess -QueryUri 'https://x/' -PrincipalId 'p' -TenantId 't' -Confirm:$false -WarningVariable w -WarningAction SilentlyContinue } | Should -Not -Throw + } + } +} From 55e24b6dc19c0360d1c3e8c946002e95b2feeb31 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 28 Oct 2026 22:36:52 -0700 Subject: [PATCH 2/3] fix(finops-hub): Handle SecureString token in Fabric access grant Co-Authored-By: Claude Sonnet 5.5 --- src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 | 2 ++ .../Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 | 8 ++++++++ 2 files changed, 10 insertions(+) diff --git a/src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 b/src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 index aac622ffb..64276279e 100644 --- a/src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 +++ b/src/powershell/Private/Grant-FinOpsHubFabricAccess.ps1 @@ -45,6 +45,8 @@ function Grant-FinOpsHubFabricAccess try { $token = (Get-AzAccessToken -ResourceUrl 'https://kusto.kusto.windows.net').Token + # Az.Accounts 14+ returns a SecureString; older versions return plain text + if ($token -is [securestring]) { $token = [System.Net.NetworkCredential]::new('', $token).Password } $body = @{ db = $database; csl = ".add database $database admins ('aadapp=$PrincipalId;$TenantId')" } | ConvertTo-Json Invoke-RestMethod -Method Post -Uri "$($QueryUri.TrimEnd('/'))/v1/rest/mgmt" -Headers @{ Authorization = "Bearer $token" } -ContentType 'application/json' -Body $body | Out-Null } diff --git a/src/powershell/Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 b/src/powershell/Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 index e0ead37ff..d68301e6b 100644 --- a/src/powershell/Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 +++ b/src/powershell/Tests/Unit/Grant-FinOpsHubFabricAccess.Tests.ps1 @@ -20,6 +20,14 @@ InModuleScope 'FinOpsToolkit' { Should -Invoke -CommandName 'Invoke-RestMethod' -Times 1 -ParameterFilter { $Body -like "*.add database Hub admins ('aadapp=p-id;t-id')*" } } + It 'Should send a plain-text bearer token when Get-AzAccessToken returns a SecureString' { + Mock -CommandName 'Get-AzAccessToken' { [pscustomobject]@{ Token = (ConvertTo-SecureString 'secret' -AsPlainText -Force) } } + + Grant-FinOpsHubFabricAccess -QueryUri 'https://x/' -PrincipalId 'p' -TenantId 't' -Confirm:$false + + Should -Invoke -CommandName 'Invoke-RestMethod' -Times 2 -ParameterFilter { $Headers.Authorization -eq 'Bearer secret' } + } + It 'Should not call the eventhouse with -WhatIf' { Grant-FinOpsHubFabricAccess -QueryUri 'https://x/' -PrincipalId 'p' -TenantId 't' -WhatIf From a8a5520b3566bc21938baa5111eee1ecdb92ea23 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Thu, 29 Oct 2026 05:37:39 +0000 Subject: [PATCH 3/3] chore: Update ms.date in docs-mslearn files --- docs-mslearn/toolkit/changelog.md | 2 +- docs-mslearn/toolkit/hubs/deploy.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs-mslearn/toolkit/changelog.md b/docs-mslearn/toolkit/changelog.md index f400655b6..3e5b820e4 100644 --- a/docs-mslearn/toolkit/changelog.md +++ b/docs-mslearn/toolkit/changelog.md @@ -3,7 +3,7 @@ title: FinOps toolkit changelog description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more. author: MSBrett ms.author: brettwil -ms.date: 10/28/2026 +ms.date: 10/29/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit diff --git a/docs-mslearn/toolkit/hubs/deploy.md b/docs-mslearn/toolkit/hubs/deploy.md index 536cf9540..fa6600662 100644 --- a/docs-mslearn/toolkit/hubs/deploy.md +++ b/docs-mslearn/toolkit/hubs/deploy.md @@ -3,7 +3,7 @@ title: How to create and update FinOps hubs description: This tutorial helps you create a new or update an existing FinOps hubs instance in Azure or Microsoft Fabric. author: flanakin ms.author: micflan -ms.date: 10/21/2026 +ms.date: 10/29/2026 ms.topic: tutorial ms.service: finops ms.subservice: finops-toolkit