Skip to content

Lazy-load open data in the PowerShell module instead of embedding it as generated code #2342

Open

Description

🤔 Problem

Build-OpenData.ps1 turns each src/open-data/*.csv file into a private Get-OpenData* function that embeds every row as a [PSCustomObject] literal. FinOpsToolkit.psm1 dot-sources every file in Private/ when the module is imported.

The pattern worked while the datasets were small. With #2298, Get-OpenDataCommitmentDiscountEligibility.ps1 is 16.2 MB and about 92,900 lines, roughly 10 times larger than any generated file before it. The source CSV is about 5.5 MB. This causes two costs:

  • Module import time: A cold Import-Module FinOpsToolkit takes about 0.7 seconds without that file and 3.6–4.8 seconds with it. Every user of every cmdlet pays this, even if they never query the data, and the file ships to the PowerShell Gallery.
  • Lint time: Tests/Lint/Lint.Tests.ps1 runs Invoke-ScriptAnalyzer once per rule per file. One rule takes about 10.9 seconds on this file, compared with 1.6 seconds on the previous largest generated file. Across all rules, that adds an estimated 14 minutes of lint to every PR that touches src/powershell/**.

🛠️ Proposal

Rework all six Get-OpenData* functions and the generator together, so the pattern stays consistent:

  • Ship the CSV files with the module instead of generating PowerShell literals.
  • Have each Get-OpenData* function load its CSV with Import-Csv on first call and cache the result in a script-scoped variable.
  • Update Build-OpenData.ps1, Build-PsModule.ps1, and the Open Data CI workflow to match.
  • Keep the existing public Get-FinOps* cmdlets and their output unchanged.
  • Keep a unit test per dataset that compares the returned rows with the source CSV.

✅ Done when

  • Import-Module FinOpsToolkit no longer loads any open data.
  • No generated open data .ps1 files remain in src/powershell/Private/.
  • The lint run no longer includes large generated data files.

Follow-up from the review on #2298.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    OKR: 1.3 Perf & scaleIssues that improve performance and scalabilityTool: Open dataDatasets to normalize and enhance FinOps reportsTool: PowerShellPowerShell scripts and automationType: CleanupFixes to clean up any technical debt

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions