Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs-mslearn/toolkit/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@ This article summarizes the features and enhancements in each release of the Fin

The following section lists features and enhancements that are currently in development.

### [FinOps hubs](hubs/finops-hubs-overview.md)

- **Added**
- Added a draft FOCUS 1.5 SKU Price preview (`SkuPrices_v1_5` and `SkuPricesWide_v1_5`) with validation and scenario scripts under `Analytics/preview/SkuPrices_v1_5`. The preview is not deployed with FinOps hubs; run it manually against a hub ([#2339](https://github.com/microsoft/finops-toolkit/pull/2339)).

-->

<br><a name="latest"></a>
Expand Down
9 changes: 9 additions & 0 deletions docs-mslearn/toolkit/hubs/data-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,15 @@ Columns in the **Prices** managed dataset include:
| x_TotalUnitPriceDiscount | Real | Source: Hubs add-on. |
| x_TotalUnitPriceDiscountPercent | Real | Source: Hubs add-on. |

### FOCUS 1.5 SKU prices preview

A draft of the FOCUS 1.5 SKU Price dataset is available in the `src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/preview/SkuPrices_v1_5` folder of the [FinOps toolkit repository](https://github.com/microsoft/finops-toolkit). It isn't deployed with FinOps hubs and may change before release. The preview builds 2 tables from **Prices_final_v1_2** in the **Ingestion** database:

- **SkuPrices_v1_5** has one row per SKU price and price type (`x_UnitPriceType` of List, Base, Contracted, or Effective) with effective start and end dates.
- **SkuPricesWide_v1_5** has the same prices with each price type in its own column.

To try it, run `Build-SkuPrices.ps1` against your hub, then `Test-SkuPrices.ps1` to validate the results. See the README in that folder for mapping decisions and known gaps.

<br>

## PricingUnits table
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<#
.SYNOPSIS
Rebuilds the FOCUS 1.5 SkuPrices preview tables on a hub from Prices_final_v1_2 and Costs_final_v1_2 using SkuPrices_v1_5.kql.

.DESCRIPTION
1. Creates or updates every function in the KQL file.
2. Builds SkuPrices_v1_5_runs_table (runs of unchanged prices) for each price type and the wide layout.
3. Builds SkuPrices_v1_5 (rows layout) one month and price type at a time, and SkuPricesWide_v1_5 (columns layout) one month at a time.
4. Builds SkuPriceIdv2_map and CostsWithSkuPriceIdv2 (Costs copy with SkuPriceIdv2) one month at a time.
Small steps keep each query within memory on small clusters.
#>
[CmdletBinding(SupportsShouldProcess)]
param(
[string] $Cluster = 'https://ftk-dev.westus.kusto.windows.net',
[string] $Database = 'Ingestion',
[string] $Context = 'fh-dev',
[string] $KqlPath = "$PSScriptRoot/SkuPrices_v1_5.kql",
[switch] $SkipCosts
)

$ErrorActionPreference = 'Stop'

$ctx = Get-AzContext -Name $Context
if (-not $ctx) { throw "Az context '$Context' not found" }
Set-AzContext -Context $ctx -Scope Process -WhatIf:$false | Out-Null # This process only; leaves the saved default alone
$secure = (Get-AzAccessToken -ResourceUrl $Cluster -AsSecureString).Token
$headers = @{ Authorization = "Bearer $([System.Net.NetworkCredential]::new('', $secure).Password)"; 'Content-Type' = 'application/json' }

function Invoke-Kusto {
param([string] $Csl, [string] $Endpoint = 'mgmt')
$body = @{ db = $Database; csl = $Csl; properties = @{ Options = @{ servertimeout = '01:00:00' } } } | ConvertTo-Json -Depth 5
(Invoke-RestMethod -Uri "$Cluster/v1/rest/$Endpoint" -Method Post -Headers $headers -Body $body -TimeoutSec 3600).Tables[0]
}

# Run one command, timing it; the first command for a table replaces it, the rest append
function Invoke-Step([string] $Table, [string] $Query, [ref] $First, [string] $Label) {
$csl = if ($First.Value) { ".set-or-replace $Table with (folder = 'FOCUS 1.5 preview', recreate_schema = true) <| $Query" } else { ".append $Table <| $Query" }
if ($PSCmdlet.ShouldProcess($Table, $csl)) {
$sw = [Diagnostics.Stopwatch]::StartNew()
Invoke-Kusto $csl | Out-Null
Write-Host "$Label done in $([int]$sw.Elapsed.TotalSeconds)s"
}
$First.Value = $false
}

# 1. Functions from the KQL file (every .create-or-alter function block, in order)
$raw = Get-Content -Path $KqlPath -Raw
$pos = 0
while (($start = $raw.IndexOf('.create-or-alter function', $pos)) -ge 0) {
$depth = 0; $end = -1
for ($i = $raw.IndexOf('{', $start); $i -lt $raw.Length; $i++) {
if ($raw[$i] -eq '{') { $depth++ }
elseif ($raw[$i] -eq '}') { $depth--; if ($depth -eq 0) { $end = $i + 1; break } }
}
if ($end -lt 0) { throw "Unbalanced function body in $KqlPath" }
$fn = $raw.Substring($start, $end - $start)
$name = [regex]::Match($fn, '\)\s*(\w+)\s*\(').Groups[1].Value
if ($PSCmdlet.ShouldProcess($name, '.create-or-alter function')) { Invoke-Kusto $fn | Out-Null; Write-Host "Updated $name" }
$pos = $end
}

$types = 'List', 'Base', 'Contracted', 'Effective'

# 2. Runs of unchanged prices, per price type and for the wide layout
$first = $true
$parts = 8 # Hash partitions of keys, to stay within memory
foreach ($type in $types + 'Wide') { for ($p = 0; $p -lt $parts; $p++) { Invoke-Step 'SkuPrices_v1_5_runs_table' "SkuPrices_v1_5_runs_build('$type', $p, $parts)" ([ref]$first) "Runs $type $p" } }

# 3. Rows and columns layouts, one month (and price type) at a time
$months = (Invoke-Kusto 'Prices_final_v1_2 | summarize by M = startofmonth(x_IngestionTime) | order by M asc' 'query').Rows | ForEach-Object { ([datetime]$_[0]).ToString('yyyy-MM-dd') }
$firstRows = $true; $firstWide = $true
foreach ($m in $months) {
foreach ($type in $types) { Invoke-Step 'SkuPrices_v1_5' "SkuPrices_v1_5_build('$type', datetime($m))" ([ref]$firstRows) "Rows $m $type" }
Invoke-Step 'SkuPricesWide_v1_5' "SkuPricesWide_v1_5_build(datetime($m))" ([ref]$firstWide) "Wide $m"
}

# 4. SkuPriceIdv2 map, then a Costs copy with SkuPriceIdv2, one month at a time
if (-not $SkipCosts) {
$first = $true
Invoke-Step 'SkuPriceIdv2_map' 'SkuPriceIdv2_map_build()' ([ref]$first) 'Map'
$costMonths = (Invoke-Kusto 'Costs_final_v1_2 | summarize by M = startofmonth(ChargePeriodStart) | order by M asc' 'query').Rows | ForEach-Object { ([datetime]$_[0]).ToString('yyyy-MM-dd') }
$first = $true
foreach ($m in $costMonths) { Invoke-Step 'CostsWithSkuPriceIdv2' "CostsWithSkuPriceIdv2_build(datetime($m))" ([ref]$first) "Costs $m" }
}

# 5. Verify
foreach ($t in 'SkuPrices_v1_5', 'SkuPricesWide_v1_5') {
$c = (Invoke-Kusto "$t | summarize Rows = count(), Current = countif(isnull(SkuPriceEffectiveEnd))" 'query').Rows[0]
$columns = (Invoke-Kusto "$t | getschema | project ColumnName" 'query').Rows | ForEach-Object { $_[0] }
$sorted = [string[]] $columns.Clone(); [Array]::Sort($sorted, [StringComparer]::Ordinal)
Write-Host "${t}: $($c[0]) rows ($($c[1]) current); $($columns.Count) columns; alphabetical: $(-not (Compare-Object $columns $sorted -SyncWindow 0))"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# FOCUS 1.5 SkuPrices (preview, draft)

Draft FinOps hubs implementation of the FOCUS 1.5 **SKU Price** dataset ([FOCUS PR 2424](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec/pull/2424)), built from `Prices_final_v1_2`, with a comparison layout and scenarios to choose between them. Not wired into `app.bicep` or `.build.config` yet. Run manually against a hub.

## Files

| File | Purpose |
|---|---|
| `SkuPrices_v1_5.kql` | Functions + build commands for `SkuPrices_v1_5`, `SkuPricesWide_v1_5`, `CostsWithSkuPriceIdv2`, and helper tables (`SkuPrices_v1_5_runs_table`, `SkuPriceIdv2_map`) |
| `Build-SkuPrices.ps1` | Creates the functions and builds all tables in small steps (key partitions, months, price types) |
| `Test-SkuPrices.ps1` | Validation: duplicates, overlaps, current rows vs. latest sheet, rows vs. runs, price match, required columns, ranges |
| `SkuPrices_v1_5_scenarios.kql` | 14 scenarios, each with the most efficient query found for each layout (plus today's monthly table for cost joins) |
| `Test-SkuPriceScenarios.ps1` | Runs each query 3x, compares results, scores simplicity, time, CPU, memory, rows scanned, x_ usage, correctness |
| `Render-SkuPriceReport.ps1` + `SkuPrices_v1_5_report.template.html` | Renders the side-by-side HTML report from scenario results and live table sizes |

Scripts default to the `ftk-dev.westus` cluster, `Ingestion` database, and `fh-dev` Az context. Override with `-Cluster`, `-Database`, `-Context`.

```powershell
./Build-SkuPrices.ps1 # ~45 min on ftk-dev
./Test-SkuPrices.ps1
./Test-SkuPriceScenarios.ps1 # writes results JSON to the temp folder
./Render-SkuPriceReport.ps1
```

## Layouts

All three hold the same 14 months of prices.

| | Today `Prices_final_v1_2` | Rows `SkuPrices_v1_5` | Columns `SkuPricesWide_v1_5` |
|---|---|---|---|
| Grain | Price per month | Price type per run of unchanged months | Price per contract per run of unchanged months |
| Prices | Columns | `UnitPrice` + `x_UnitPriceType` (List, Base, Contracted, Effective) | `ListUnitPrice`, `ContractedUnitPrice`, `x_BaseUnitPrice`, `x_EffectiveUnitPrice` |
| Dates | Month | Effective start/end per price type | One start/end per row (new row when any price changes) |
| List price scope | n/a | Public: no contract, global eligibility | Scoped to the contract with the rest of the row |
| Rows | 18.78M | 4.17M (3.89M current) | 1.79M (1.63M current) |
| Extent | 2.34 GB | 0.64 GB | 0.33 GB |

Rows by type (all / current): List 1,348,812 / 1,236,379; Contracted 1,326,353 / 1,236,379 (98% = List); Base 1,056,500 / 1,023,491 (94% = List); Effective 438,723 / 388,900.

## Mapping decisions

* `SkuPriceId` = `SkuPriceIdv2` (unique per price); source `SkuPriceId` kept as `x_SkuPriceIdv1`
* `ContractId` = `/providers/microsoft.billing/billingaccounts/{x_BillingAccountId}` (EA enrollment / MCA billing account, not the profile)
* Savings plan rows carry only the Effective price (list/base/contracted on those rows are copies of the on-demand meter's prices)
* Effective dates are month-level runs of unchanged prices across monthly loads; current prices have no end
* `PricingRegionId` = region name (`x_SkuRegion`); `PricingServiceName` = meter category; `SkuPriceCreated`/`LastUpdated` = first/last ingestion
* `PurchaseDurationType` from `x_SkuTerm`; `PurchasePaymentModel` empty (not in price sheet)
* Non-SKU Price source columns get an `x_` prefix; `CommitmentDiscountCategory` kept as a filter label; columns sorted alphabetically

## Cost and Usage join

Costs `SkuPriceId` doesn't match the price sheet's, and Costs lacks `x_SkuProductId`/`x_SkuMeterType`, so `SkuPriceIdv2` can't be rebuilt from Costs columns. `CostsWithSkuPriceIdv2` is a copy of `Costs_final_v1_2` with `SkuPriceIdv2` looked up from prices by price type + meter + offer + term:

| Cost type | Price used | Rows matched | Cost matched |
|---|---|---|---|
| On-demand usage | On-demand | 100% | 100% |
| Reservation-covered usage | On-demand (no reservation usage price) | 90.2% | 74.7% |
| Savings-plan-covered usage | Savings plan for the term | 83.1% (68 rows have no meter) | 100% |
| Reservation purchase | Reservation for the term (only when unique) | 16/16 | n/a |
| Adjustment | None | 0% | 0% |

Effective-dated prices have no range lookup in KQL. Scenarios spread each price over the months it was in effect and look up on `SkuPriceId` + `ContractId` + `PricingCurrency` + month: no range join and no duplicated cost rows. Today's monthly table needs no spreading.

## Scenarios

| Id | Scenario | Simpler | Faster | CPU | Memory | Rows scanned | FOCUS-only | Correct |
|---|---|---|---|---|---|---|---|---|
| S01 | List price on a date | Tie | Tie | Tie | Tie | Columns | Columns | Tie |
| S02 | Negotiated discount by service | Columns | Columns | Columns | Columns | Columns | Columns | Tie |
| S03 | Base price drift from list | Columns | Columns | Columns | Columns | Columns | Tie | Tie |
| S04 | Reservation price vs. on-demand | Tie | Columns | Columns | Tie | Columns | Columns | Tie |
| S05 | Savings plan rate vs. on-demand | Tie | Tie | Columns | Tie | Columns | Tie | Tie |
| S06 | Price changes since July | Rows | Tie | Tie | Rows | Columns | Rows | Tie |
| S07 | Cheapest region for a VM size | Tie | Tie | Tie | Tie | Columns | Columns | Tie |
| S08 | Verify billed unit prices (+Monthly) | Tie | Tie | Tie | Tie | Columns | Columns | Tie |
| S09 | Savings plan what-if for on-demand usage (+Monthly) | Tie | Monthly | Monthly | Tie | Columns | Tie | Tie |
| S10 | All prices for one SKU | Tie | Tie | Tie | Tie | Columns | Rows | Tie |
| S11 | Estimate a planned workload | Tie | Tie | Tie | Tie | Columns | Columns | Tie |
| S12 | Public prices visible to each account | Tie | Rows | Rows | Tie | Columns | Columns | Rows |
| S13 | Usage rates vs. purchase fees | Rows | Tie | Tie | Tie | Columns | Rows | Tie |
| S14 | Backfill missing prices on cost data (+Monthly) | Monthly | Tie | Tie | Tie | Columns | Columns | Tie |
| | **Wins (Rows / Columns / Monthly)** | 2 / 2 / 1 | 1 / 3 / 1 | 1 / 4 / 1 | 1 / 2 / 0 | 0 / 14 / 0 | 3 / 8 / 0 | 1 / 0 / 0 |

Scores: winner per measure, or Tie within 15% of the runner-up. Simpler = operators + 2x joins + lets; Faster/CPU/Memory/Rows scanned = median of 3 server-measured runs; FOCUS-only = fewer distinct x_ columns referenced; Correct = which layout returns the right answer when they differ (S12: columns hide the public list price from accounts outside the agreement).

Covers 9 of 13 FOCUS PR 2595 queries plus 5 more (S03, S05, S06, S10, S14). Not testable with this data: announced price changes, public vs. negotiated at a quantity, tier resolution, next tier (no future prices or tiers in EA price sheets), negotiated savings plan discount, multiple currencies, consumption currency. Cost scenarios use a price only when its block unit matches the charge's `PricingUnit`.

## Gaps / TODO

* [ ] `PricingUnit` and `x_PricingBlockSize` empty: `PricingUnits` open data table has 0 rows on ftk-dev, so `UnitPrice` is per block (e.g., per 10 hours); scenarios parse the block size from `x_PricingUnitDescription`
* [ ] Savings plan list price ("list effective") is lost on ingestion: `Prices_transform_v1_2` nulls the SP `MarketPrice` and backfills on-demand prices; needed to measure negotiated SP discounts
* [ ] Decide whether to skip Base/Contracted rows equal to List (lookups fall back to List)
* [ ] MCA untested (collapses billing profiles into the billing account; check for profile-specific prices); latest month is global, not per account
* [ ] Decide final layout once FOCUS settles the price type column (planned for 1.6)
* [ ] Add `SkuPriceIdv2` to `Costs_transform_v1_2` (replaces `CostsWithSkuPriceIdv2`); find a reservation-covered usage price
* [ ] Wire into `IngestionSetup_v1_5.kql` / `HubSetup_v1_5.kql`, `app.bicep`, `.build.config`; add `SkuPrices()` hub function, docs, changelog
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
<#
.SYNOPSIS
Renders the SKU Price layout comparison report (HTML) from scenario results and live table sizes.
#>
[CmdletBinding()]
param(
[string] $Cluster = 'https://ftk-dev.westus.kusto.windows.net',
[string] $Database = 'Ingestion',
[string] $Context = 'fh-dev',
[string] $Template = "$PSScriptRoot/SkuPrices_v1_5_report.template.html",
[string] $Results = (Join-Path ([IO.Path]::GetTempPath()) 'SkuPrices_v1_5_scenarios.results.json'),
[string] $OutFile = (Join-Path ([IO.Path]::GetTempPath()) "sku-price-layouts.html")
)

$ErrorActionPreference = 'Stop'
Set-AzContext -Context (Get-AzContext -Name $Context) -Scope Process | Out-Null
$secure = (Get-AzAccessToken -ResourceUrl $Cluster -AsSecureString).Token
$headers = @{ Authorization = "Bearer $([System.Net.NetworkCredential]::new('', $secure).Password)"; 'Content-Type' = 'application/json' }
function Invoke-Kusto([string] $Csl, [string] $Endpoint = 'query') {
$body = @{ db = $Database; csl = $Csl } | ConvertTo-Json
, @((Invoke-RestMethod -Uri "$Cluster/v1/rest/$Endpoint" -Method Post -Headers $headers -Body $body -TimeoutSec 600).Tables[0].Rows)
}

function Get-Size([string] $Table, [string] $Filter = '') {
$r = Invoke-Kusto ".show table $Table extents $Filter | summarize Rows = sum(RowCount), Extent = sum(ExtentSize), Compressed = sum(CompressedSize)" 'mgmt'
$cols = (Invoke-Kusto "$Table | getschema | count")[0][0]
[pscustomobject]@{ Rows = [double]$r[0][0]; Extent = [double]$r[0][1]; Compressed = [double]$r[0][2]; Columns = $cols }
}
function M($v) { '{0:0.00}M' -f ($v / 1e6) }
function G($v) { '{0:0.00}' -f ($v / 1GB) }
function X($a, $b) { '{0:0.0}Γ—' -f ($a / $b) }

$p = Get-Size 'Prices_final_v1_2'
$r = Get-Size 'SkuPrices_v1_5'
$c = Get-Size 'SkuPricesWide_v1_5'
$rCur = (Invoke-Kusto 'SkuPrices_v1_5 | where isnull(SkuPriceEffectiveEnd) | count')[0][0]
$cCur = (Invoke-Kusto 'SkuPricesWide_v1_5 | where isnull(SkuPriceEffectiveEnd) | count')[0][0]

function Card($cls, $label, $table, $s, $ratio, [string[]] $bullets) {
@"
<div class="tcard $cls"><div class="tname">$label <code>$table</code></div>
<div class="stats"><div class="stat"><b>$(M $s.Rows)</b><span>Rows</span></div><div class="stat"><b>$(G $s.Extent)</b><span>GB extent</span></div><div class="stat"><b>$(G $s.Compressed)</b><span>GB compressed</span></div></div>
<div class="ratio">$ratio</div><ul>$(($bullets | ForEach-Object { "<li>$_</li>" }) -join '')</ul></div>
"@
}
$tables = (Card 'mon' 'Today' 'Prices_final_v1_2' $p "$($p.Columns) columns. 14 monthly price sheets." @(
'One row per price per month', 'All price types as columns', 'Month-level dates; joins match on month')) +
(Card 'rows' 'Rows' 'SkuPrices_v1_5' $r "$($r.Columns) columns. $(X $r.Rows $p.Rows) rows, $(X $r.Extent $p.Extent) extent of today. $(M $rCur) current." @(
'One row per price type (<code>x_UnitPriceType</code>) per run of unchanged months', 'List rows are public: no contract, global eligibility', 'Effective start and end per price type')) +
(Card 'cols' 'Columns' 'SkuPricesWide_v1_5' $c "$($c.Columns) columns. $(X $c.Rows $p.Rows) rows, $(X $c.Extent $p.Extent) extent of today. $(M $cCur) current." @(
'One row per price per contract per run; a new row when any price changes', 'List price repeats per contract and is scoped to it', 'One effective start and end for all prices on the row'))

$json = ((Get-Content $Results -Raw | ConvertFrom-Json) | ConvertTo-Json -Depth 8 -Compress).Replace('</', '<\/')
(Get-Content $Template -Raw).Replace('__TABLES__', $tables).Replace('__DATA__', $json).Replace('__GENERATED__', (Get-Date -Format 'yyyy-MM-dd')) | Set-Content $OutFile -NoNewline
Write-Host "Today: $(M $p.Rows) rows $(G $p.Extent) GB | Rows: $(M $r.Rows) ($(M $rCur) current) $(G $r.Extent) GB | Columns: $(M $c.Rows) ($(M $cCur) current) $(G $c.Extent) GB"
Write-Host "Wrote $OutFile"
Loading
Loading