[Michael Flanakin (flanakin)]

Summary

Replaces the manual "open each of the 12 Power BI templates in Power BI Desktop, check for errors, and read the parameter prompt" release check with two repeatable layers.

1. Static layer (any platform): src/powershell/Tests/Unit/PowerBITemplates.Tests.ps1

Opens each built PBIT, detects the encoding of DataModelSchema (UTF-16LE with or without BOM, or UTF-8), and checks:

• 6 .kql and 6 .storage templates exist, one per report in reports.json. The two release ZIP files hold the same six each.
• Tables match reports.json for the report (limited to tables that exist in that model type, like Prices only existing for storage).
• The template prompts for exactly the parameters the shared dataset defines for the report, with the same defaults as the source (expressions.tmdl), except the data source URLs, which must be null. This covers the storage defaults (Number of Months 3, Default Granularity Daily, both FALSE flags), the KQL defaults (Number of Months 13), and Remove Duplicate Resource IDs only where reports.json keeps it. Values are read from the source rather than hard-coded.
• Storage URL, Cluster URL, RangeStart, and RangeEnd are empty, ftk_DemoFilter is the empty version, and no storage account or cluster name is hard-coded.
• No ftkdemo/ftk-demo string appears in any text part of the PBIT.
• Storage templates read gzip exports. The .gz filter ([Extension] = ".gz" plus Binary.Decompress(..., Compression.GZip)) lives in the shared ftk_Storage function, not in the Costs query itself, so the test checks that Costs reads through ftk_Storage("focuscost") and that ftk_Storage has the filter.

When no templates are built (release/pbit and the ZIP files are missing) every test is skipped with the command that builds them, so unit CI is unaffected. When some are built, missing or unreadable ones fail. I did not add a build step to CI, because Build-PowerBI installs the Analysis Services NuGet package; that is a possible follow-up.

2. Windows UI layer: src/scripts/Test-PowerBITemplates.ps1

For each template it launches Power BI Desktop, answers the "Potential security risk" dialog with OK when it shows (KQL), waits for the parameter prompt, fails on any other dialog (for example "Unable to open template"), reads the prompt, and ends only the Power BI Desktop process it started. It checks that the prompt is titled after the template, lists the expected parameters in order with the expected values (derived from the parameters in the PBIT itself), has empty Storage URL, Cluster URL, RangeStart, and RangeEnd fields, and doesn't mention a demo hub. It saves a screenshot per template to release/pbit-test.

Package-PowerBI -ValidateTemplates (off by default, one parameter and one block) runs it before packaging. Save-PowerBIProject.ps1 is unchanged.

Why UI Automation was not enough, and what is used instead. I investigated before falling back to screenshots. Power BI Desktop shows the security dialog and the parameter prompt in an embedded browser control (Internet Explorer_Server). UI Automation lists only the main window for the process; AutomationElement.FromHandle on the dialog gives four unnamed panes and nothing inside the browser control. The dialogs are found from Windows' own window list (the window title is the dialog title), and the page is read through WM_HTML_GETOBJECT and the browser control's HTML document, which is what screen readers use. That gives the real parameter names and values, and buttons are pressed through the page, so no mouse or keyboard input is needed (verified: the DOM click() answers the security dialog). Pass/fail therefore rests on the actual prompt contents, not on screenshots. The screenshots are evidence only. I reused the approach of Save-PowerBIProject.ps1 (Win32 window listing, process-scoped cleanup, screenshot on failure) but did not move its helpers into a shared file: I couldn't safely regression-test that 1200-line script, and the issue asked for its behavior to stay unchanged. Sharing the helpers is a reasonable follow-up.

Why not Playwright. Power BI Desktop is a native WinForms/WPF app with embedded browser controls, not a web page or an Electron/WebView2 app Playwright can attach to.

What it cannot check.

• Anything past the prompt: Power BI Desktop doesn't expose report page tabs to automation, and loading the report needs a data source. It does not click Load or refresh data.
• It does not check that each KQL template shows the security dialog; it answers it when present and reports whether it appeared.
• It reads Power BI Desktop's current dialog markup (class names like o-group-fieldTemplate). A Power BI Desktop update that changes the prompt's markup would make it fail with "prompt did not appear", and the selectors would need updating.
• The KQL Default Granularity parameter's current value is "Monthly" with DefaultValue="Daily". The prompt shows the default (Daily), which is what is checked. I didn't change that.

What was run

• Test-PowerShell -Lint: 3970 of 3970 passed.
• PowerBITemplates.Tests.ps1 and Build-PowerBI.Tests.ps1 with Pester 6.2.0, after building with Build-PowerBI -NoPbip: 169 passed, 0 failed, 6 skipped (the gzip test is skipped for the 6 KQL templates by design).
• The same tests with release moved away: all skipped with the build message (1 passed, 124 skipped). With only the two ZIP files present: pass.
• Negative checks on mutated copies of the built templates, which all failed as intended: a demo Storage URL, a demo cluster, a changed Number of Months default, the gzip filter removed, a missing template.
• Test-PowerBITemplates.ps1 on all 12 built templates, once at the end, on Windows with Power BI Desktop 2.159 from the Microsoft Store: 12 of 12 passed (about 30 seconds each, 5.7 minutes total, security dialog answered for all 6 KQL templates). No Power BI Desktop processes were left running.
• UI negative checks (2 mutated templates): a template with a demo Storage URL fails on the prompt contents, and a template with a corrupted report layout fails on the "Unable to open template" dialog.
• PSScriptAnalyzer: only the warnings the existing scripts also have (Write-Host, ShouldProcess on Stop-Desktop, BOM).

Not run: Package-PowerBI -ValidateTemplates end to end (it needs saved demo PBIX files; the code path is a single call to the script above), the static tests on Linux CI, and a second Power BI Desktop version.

Changelog

No changelog entry. This is a maintainer-facing validation script and tests, not a user-facing change under the changelog standard.

🤖 Generated with Claude Code

[Roland Krummenacher (RolandKrummenacher)]

The static checks look sound, and CI is green. CI only tests the "nothing built" skip path, though; the built path has only run on the author's Windows machine.