fix(hubs): stop price transform row fan-out and fix eligibility scoping

docs-mslearn/toolkit/changelog.md

@@ -3,7 +3,7 @@ title: FinOps toolkit changelog
 description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more.
 author: MSBrett
 ms.author: brettwil
-ms.date: 10/28/2026
+ms.date: 10/29/2026
 ms.topic: reference
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -18,14 +18,17 @@ ms.reviewer: brettwil
 
 This article summarizes the features and enhancements in each release of the FinOps toolkit.
 
-<!--
 <br>
 
 ## Unreleased
 
 The following section lists features and enhancements that are currently in development.
 
--->
+### [FinOps hubs](hubs/finops-hubs-overview.md)
+
+- **Fixed**
+  - Fixed the v1_0/v1_2 price transforms creating more rows than were ingested: the savings plan price `lookup` deduplicated its Consumption dimension side with `distinct` over price columns, so a meter with more than one price fanned out every matching savings plan row. The lookup key now includes billing profile and currency so one profile's or currency's price is never applied to another's savings plan rows, and the dimension side uses `summarize take_any(...)` for a guaranteed one-row-per-key result ([#1736](https://github.com/microsoft/finops-toolkit/issues/1736)).
+  - Fixed commitment discount eligibility (`x_CommitmentDiscountSpendEligibility`, `x_CommitmentDiscountUsageEligibility`) being computed incorrectly when a pricesheet export lands as multiple parquet files: because `Prices_transform_v1_0`/`v1_2` are Data Explorer update policy functions, each file triggers a separate invocation that only sees that file's rows of `Prices_raw`, so a meter's Reservation/SavingsPlan row and its Consumption row could each be visible in different invocations and eligibility would be computed against a partial view. Eligibility is now sourced from the [Commitment discount eligibility](open-data.md#commitment-discount-eligibility) open-data table, ingested into a new `CommitmentDiscountEligibility` ADX table, which isn't affected by per-invocation partitioning ([#1625](https://github.com/microsoft/finops-toolkit/issues/1625)). If a meter isn't in the open-data table (for example, Azure Government or China meters, or a newly released meter), the transform falls back to the previous pricesheet-derived answer instead of reporting "Not Eligible". Local FinOps hubs load the table from the open-data release artifact. Fabric users who rerun the versioned script on an existing eventhouse must run the infrastructure script (`IngestionSetup_HubInfra.kql`) first to create the new table. Both sides of the eligibility lookup are normalized to lowercase before joining, since the pricesheet's meter ID casing isn't guaranteed to match the open-data table's.
 
 <br><a name="latest"></a>
 

docs-mslearn/toolkit/hubs/upgrade.md

@@ -3,7 +3,7 @@ title: Upgrade your FinOps hubs
 description: Learn how to upgrade your existing FinOps hub instance to the latest version, including necessary steps and considerations.
 author: flanakin
 ms.author: micflan
-ms.date: 10/28/2026
+ms.date: 10/29/2026
 ms.topic: how-to
 ms.service: finops
 ms.subservice: finops-toolkit
@@ -64,7 +64,7 @@ This step only applies if you are using Microsoft Fabric as a primary data store
 - You are using Azure Storage as your data store.
 - You are using Azure Data Explorer as your data store.
 
-The Microsoft Fabric eventhouse database schema must be manually updated with each release. For details, see [Set up Microsoft Fabric](deploy.md#optional-set-up-microsoft-fabric).
+The Microsoft Fabric eventhouse database schema must be manually updated with each release. For details, see [Set up Microsoft Fabric](deploy.md#optional-set-up-microsoft-fabric). Run the infrastructure script (`IngestionSetup_HubInfra.kql`) before the versioned script so new tables such as `CommitmentDiscountEligibility` exist before functions reference them.
 
 <br>
 

src/powershell/Tests/Unit/HubsPriceTransformLookups.Tests.ps1

@@ -0,0 +1,64 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+<#
+    Regression coverage for #1736 (Prices_transform row fan-out) and #1625 (commitment discount eligibility
+    computed from a per-invocation partial view of Prices_raw).
+    Text-pattern checks, like HubsReservationPriceBackfill.Tests.ps1.
+#>
+
+Describe 'HubsPriceTransformLookups' {
+
+    BeforeDiscovery {
+        $repoRoot = (Resolve-Path "$PSScriptRoot/../../../..").Path
+        $scriptsPath = Join-Path $repoRoot 'src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts'
+        $ingestionFiles = @('IngestionSetup_v1_0.kql', 'IngestionSetup_v1_2.kql') | ForEach-Object {
+            @{ Name = $_; FullName = (Join-Path $scriptsPath $_) }
+        }
+    }
+
+    BeforeAll {
+        $repoRoot = (Resolve-Path "$PSScriptRoot/../../../..").Path
+        $analyticsPath = Join-Path $repoRoot 'src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics'
+    }
+
+    Context 'Savings plan price lookup (#1736)' {
+
+        It 'Should dedupe the Consumption side with take_any, not distinct: <Name>' -ForEach $ingestionFiles {
+            $content = Get-Content -Path $FullName -Raw
+            $content | Should -Match "lookup kind=leftouter \(prices \| where x_SkuPriceType == 'Consumption'[^\n]*summarize take_any\(ListUnitPrice\), take_any\(ContractedUnitPrice\), take_any\(x_BaseUnitPrice\) by tmp_SavingsPlanKey\)" -Because 'distinct over price columns leaves several rows per key and fans out SavingsPlan rows'
+            $content | Should -Not -Match '\| distinct tmp_SavingsPlanKey' -Because 'distinct over price columns is the #1736 fan-out bug'
+        }
+
+        It 'Should keep the savings plan meter prefilter on the dimension side: <Name>' -ForEach $ingestionFiles {
+            $content = Get-Content -Path $FullName -Raw
+            $content | Should -Match "'Consumption' \| where x_SkuMeterId in \(spMeters\) \| summarize take_any" -Because 'limits the aggregation to savings plan meters on the ingestion hot path'
+        }
+
+        It 'Should include billing profile and currency in the savings plan key: <Name>' -ForEach $ingestionFiles {
+            $content = Get-Content -Path $FullName -Raw
+            $content | Should -Match 'tmp_SavingsPlanKey = strcat\([^\n]*x_BillingProfileId[^\n]*Currency' -Because 'one billing profile or currency must not pick up another one''s price'
+        }
+    }
+
+    Context 'Commitment discount eligibility (#1625)' {
+
+        It 'Should source eligibility from the CommitmentDiscountEligibility table: <Name>' -ForEach $ingestionFiles {
+            $content = Get-Content -Path $FullName -Raw
+            $content | Should -Match 'let commitmentEligibility = CommitmentDiscountEligibility' -Because 'deriving eligibility only from the invocation''s partial Prices_raw gives wrong results for multi-file exports'
+            $content | Should -Match 'lookup kind=leftouter \(commitmentEligibility\) on tmp_MeterId'
+        }
+
+        It 'Should fall back to pricesheet-derived eligibility when the meter is missing: <Name>' -ForEach $ingestionFiles {
+            $content = Get-Content -Path $FullName -Raw
+            $content | Should -Match "iff\(x_SkuMeterId in \(spMeters\), 'Eligible', 'Not Eligible'\)" -Because 'Azure Government/China and new meters are not in the open data'
+            $content | Should -Match "iff\(x_SkuMeterId in \(riMeters\) and x_SkuPriceType != 'ReservedInstance', 'Eligible', 'Not Eligible'\)"
+        }
+
+        It 'Should create and populate the table for ADF, Fabric, and local hubs' {
+            Get-Content -Path (Join-Path $analyticsPath 'scripts/IngestionSetup_HubInfra.kql') -Raw | Should -Match '\.create-merge table CommitmentDiscountEligibility'
+            Get-Content -Path (Join-Path $analyticsPath 'scripts/IngestionSetup_OpenDataExternal.kql') -Raw | Should -Match '\.set-or-replace CommitmentDiscountEligibility'
+            Get-Content -Path (Join-Path $analyticsPath 'app.bicep') -Raw | Should -Match '\.set-or-replace CommitmentDiscountEligibility'
+        }
+    }
+}

src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/app.bicep

@@ -1099,12 +1099,43 @@ resource pipeline_InitializeHub 'Microsoft.DataFactory/factories/pipelines@2018-
                       }
                     }
                   }
+                  { // Update CommitmentDiscountEligibility in ADX
+                    name: 'Update CommitmentDiscountEligibility in ADX'
+                    type: 'AzureDataExplorerCommand'
+                    dependsOn: [
+                      {
+                        activity: 'Update Services in ADX'
+                        dependencyConditions: [
+                          'Succeeded'
+                        ]
+                      }
+                    ]
+                    policy: {
+                      timeout: '0.12:00:00'
+                      retry: 2  // CommitmentDiscountEligibility.csv is ~8x larger than the next biggest reference CSV (ResourceTypes); retry transient network failures instead of failing the whole pipeline
+                      retryIntervalInSeconds: 30
+                      secureOutput: false
+                      secureInput: false
+                    }
+                    userProperties: []
+                    typeProperties: {
+                      command: '.set-or-replace CommitmentDiscountEligibility <| externaldata(MeterId: string, x_CommitmentDiscountSpendEligibility: string, x_CommitmentDiscountUsageEligibility: string)[@"${ftkReleaseUri}/CommitmentDiscountEligibility.csv"] with (format="csv", ignoreFirstRecord=true)'
+                      commandTimeout: '00:30:00'
+                    }
+                    linkedServiceName: {
+                      referenceName: linkedService_dataExplorer.name
+                      type: 'LinkedServiceReference'
+                      parameters: {
+                        database: INGESTION_DB  // Do not use dynamic reference since that won't work with Fabric
+                      }
+                    }
+                  }
                   { // Ingestion Complete
                     name: 'Ingestion Complete'
                     type: 'SetVariable'
                     dependsOn: [
                       {
-                        activity: 'Update Services in ADX'
+                        activity: 'Update CommitmentDiscountEligibility in ADX'
                         dependencyConditions: [
                           'Succeeded'
                         ]

src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_HubInfra.kql

@@ -88,6 +88,13 @@ HubScopes()
     x_ServiceModel:           string
 )
 
+// CommitmentDiscountEligibility
+.create-merge table CommitmentDiscountEligibility(
+    MeterId:                               string,
+    x_CommitmentDiscountSpendEligibility:  string,
+    x_CommitmentDiscountUsageEligibility:  string
+)
+
 //----------------------------------------------------------------------------------------------------------------------
 
 // parse_resourceid

src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_OpenDataExternal.kql

@@ -15,3 +15,4 @@
 .set-or-replace Regions <| externaldata(ResourceLocation: string, RegionId: string, RegionName: string)[@'$$openDataPath$$/Regions.csv'] with (format='csv', ignoreFirstRecord=true)
 .set-or-replace ResourceTypes <| externaldata(x_ResourceType: string, SingularDisplayName: string, PluralDisplayName: string, LowerSingularDisplayName: string, LowerPluralDisplayName: string, IsPreview: bool, Description: string, IconUri: string, Links: string)[@'$$openDataPath$$/ResourceTypes.csv'] with (format='csv', ignoreFirstRecord=true) | project-away Links
 .set-or-replace Services <| externaldata(x_ConsumedService: string, x_ResourceType: string, ServiceName: string, ServiceCategory: string, ServiceSubcategory: string, PublisherName: string, x_PublisherCategory: string, x_Environment: string, x_ServiceModel: string)[@'$$openDataPath$$/Services.csv'] with (format='csv', ignoreFirstRecord=true)
+.set-or-replace CommitmentDiscountEligibility <| externaldata(MeterId: string, x_CommitmentDiscountSpendEligibility: string, x_CommitmentDiscountUsageEligibility: string)[@'$$openDataPath$$/CommitmentDiscountEligibility.csv'] with (format='csv', ignoreFirstRecord=true)

src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_0.kql

@@ -72,22 +72,42 @@ Prices_transform_v1_0()
         | extend SkuPriceIdv2 = strcat(case(x_SkuPriceType == 'Consumption', 'OD', x_SkuPriceType == 'ReservedInstance', 'RI', x_SkuPriceType == 'SavingsPlan', 'SP', 'XX'), substring(ChargeCategory, 0, 1), x_SkuTerm, '_', x_SkuProductId, '_', x_SkuId, '_', x_SkuMeterType, '_', x_SkuTier, x_SkuOfferId)
         | extend x_BillingAccountId = iff(BillingAccountId startswith '/', split(BillingAccountId, '/')[-1], coalesce(BillingAccountId, EnrollmentNumber))
         | extend x_BillingProfileId = iff(BillingProfileId startswith '/', split(BillingProfileId, '/')[-1], coalesce(BillingProfileId, EnrollmentNumber))
-        | extend tmp_SavingsPlanKey = strcat(x_SkuMeterId, x_SkuProductId, x_SkuId, x_SkuTier, x_SkuOfferId)
+        | extend tmp_SavingsPlanKey = strcat(x_SkuMeterId, x_SkuProductId, x_SkuId, x_SkuTier, x_SkuOfferId, '|', x_BillingProfileId, '|', coalesce(Currency, CurrencyCode))
         //
         // Get latest ingested row based on the unique ID
         | extend x_IngestionTime = ingestion_time()
     );
     //
-    // Meters for reservations and savings plans to identify commitment eligibility
+    // Commitment discount eligibility per meter, used below. Sourced from the
+    // CommitmentDiscountEligibility open-data table (a snapshot of the Azure Retail Prices API)
+    // rather than derived from `prices` itself: Prices_raw can land as multiple parquet-snappy
+    // files per export, each triggering a SEPARATE update policy invocation (see #1625), so a
+    // meter's Reservation/SavingsPlan row and its Consumption row can be visible in different
+    // invocations. Deriving eligibility from `prices` only sees whichever rows happen to be in the
+    // current invocation; CommitmentDiscountEligibility is ingested as a whole table and doesn't
+    // have that gap. Dedupe with take_any() even though MeterId is expected to be unique in the
+    // source data -- defends against a future duplicate row silently fanning out the lookup below.
+    // MeterId is lowercased by the generator (Update-CommitmentDiscountEligibility.ps1); the
+    // pricesheet's MeterId/MeterID casing isn't documented/guaranteed, and `lookup on` only
+    // supports case-sensitive equality, so both sides are normalized to lowercase for the join.
+    // Fallback only: meters with Reservation/SavingsPlan rows in the current invocation. Used when
+    // a meter is missing from the open-data table (see eligibility step below).
     let riMeters = prices | where x_SkuPriceType == 'ReservedInstance' | distinct x_SkuMeterId;
     let spMeters = prices | where x_SkuPriceType == 'SavingsPlan' | distinct x_SkuMeterId;
+    let commitmentEligibility = CommitmentDiscountEligibility
+        | extend tmp_MeterId = tolower(MeterId)
+        | summarize take_any(x_CommitmentDiscountSpendEligibility), take_any(x_CommitmentDiscountUsageEligibility) by tmp_MeterId;
     //
     // Copy list/base/contracted prices from on-demand SKUs
     prices
     | where x_SkuPriceType == 'SavingsPlan'
     // If we use join, specify the shuffle key
-    // TODO: Compare join vs. lookup perf -- | join kind=leftouter hint.strategy=shuffle (prices | where x_SkuPriceType == 'Consumption' | where x_SkuMeterId in (spMeters) | distinct tmp_SavingsPlanKey, ListUnitPrice, ContractedUnitPrice, x_BaseUnitPrice) on tmp_SavingsPlanKey
-    | lookup kind=leftouter (prices | where x_SkuPriceType == 'Consumption' | where x_SkuMeterId in (spMeters) | distinct tmp_SavingsPlanKey, ListUnitPrice, ContractedUnitPrice, x_BaseUnitPrice) on tmp_SavingsPlanKey
+    // TODO: Compare join vs. lookup perf -- | join kind=leftouter hint.strategy=shuffle (prices | where x_SkuPriceType == 'Consumption' | where x_SkuMeterId in (spMeters) | summarize take_any(ListUnitPrice), take_any(ContractedUnitPrice), take_any(x_BaseUnitPrice) by tmp_SavingsPlanKey) on tmp_SavingsPlanKey
+    // The key includes billing profile and currency so one profile's or currency's prices are never
+    // applied to another's savings plan rows (meter IDs are already region-specific). take_any()
+    // stays as a defence so the dimension side is exactly one row per key (a duplicate would fan
+    // out matching SavingsPlan rows, #1736), and the prefilter limits it to savings plan meters.
+    | lookup kind=leftouter (prices | where x_SkuPriceType == 'Consumption' | where x_SkuMeterId in (spMeters) | summarize take_any(ListUnitPrice), take_any(ContractedUnitPrice), take_any(x_BaseUnitPrice) by tmp_SavingsPlanKey) on tmp_SavingsPlanKey
     | extend ListUnitPrice = coalesce(ListUnitPrice, ListUnitPrice1)
     | extend ContractedUnitPrice = coalesce(ContractedUnitPrice, ContractedUnitPrice1)
     | extend x_BaseUnitPrice = coalesce(x_BaseUnitPrice, x_BaseUnitPrice1)
@@ -98,12 +118,24 @@ Prices_transform_v1_0()
     // "Usage" commitment (committed to a quantity of usage) and a savings plan as a "Spend"
     // commitment (committed to an amount of money), matching the CommitmentDiscountCategory
     // value this transform assigns, so reservation meters set the usage column and savings plan
-    // meters set the spend column.
+    // meters set the spend column. The open-data table follows the same convention.
     // Reservation price rows are excluded from their own eligibility: the column describes what
     // an on-demand charge could be covered by, not the commitment row itself.
+    // The open-data answer wins when it has one. When the meter isn't in the open-data table
+    // (Azure Government/China meters, brand-new meters, or an empty table), fall back to the
+    // pricesheet-derived answer so those hubs keep today's behavior instead of "Not Eligible".
     // TODO: Would a join be faster?
-    | extend x_CommitmentDiscountSpendEligibility = iff(x_SkuMeterId in (spMeters), 'Eligible', 'Not Eligible')
-    | extend x_CommitmentDiscountUsageEligibility = iff(x_SkuMeterId in (riMeters) and x_SkuPriceType != 'ReservedInstance', 'Eligible', 'Not Eligible')
+    | extend tmp_MeterId = tolower(x_SkuMeterId)
+    | lookup kind=leftouter (commitmentEligibility) on tmp_MeterId
+    | extend x_CommitmentDiscountSpendEligibility = coalesce(
+        x_CommitmentDiscountSpendEligibility,
+        iff(x_SkuMeterId in (spMeters), 'Eligible', 'Not Eligible')
+    )
+    | extend x_CommitmentDiscountUsageEligibility = coalesce(
+        iff(isnotempty(x_CommitmentDiscountUsageEligibility) and x_SkuPriceType != 'ReservedInstance', x_CommitmentDiscountUsageEligibility, ''),
+        iff(x_SkuMeterId in (riMeters) and x_SkuPriceType != 'ReservedInstance', 'Eligible', 'Not Eligible')
+    )
+    | project-away tmp_MeterId
     //
     // Add PricingUnit and x_PricingBlockSize
     // TODO: Compare join vs. lookup perf -- | join kind=leftouter (PricingUnits) on x_PricingUnitDescription | project-away x_PricingUnitDescription1