Skip to content

fix(power-bi): wire RangeStart/RangeEnd into Costs query for incremental refresh - #2317

Draft
Michael Flanakin (flanakin) wants to merge 2 commits into
devfrom
fix/2314-power-bi-incremental-refresh
Draft

Michael Flanakin (flanakin) wants to merge 2 commits into
devfrom
fix/2314-power-bi-incremental-refresh

Conversation

@flanakin

Copy link
Copy Markdown
Collaborator

Summary

Power BI's Incremental refresh and archive settings dialog couldn't be configured for the Costs table in the storage-based reports because RangeStart/RangeEnd were only referenced inside the shared ftk_Storage function — used there purely as a file-level optimization to skip whole parquet files outside the requested range. Power BI only detects incremental refresh support when a table's own query directly filters rows using Table.SelectRows(..., each [Column] >= RangeStart and [Column] < RangeEnd).

Changes

  • src/power-bi/storage/Shared.Dataset/definition/tables/Costs.tmdl: Added a Table.SelectRows step as the final step of the Costs partition query that filters on [ChargePeriodStart] >= RangeStart and [ChargePeriodStart] < RangeEnd (null-guarded so normal, non-incremental refreshes still load all data by default). This coexists with the existing file-level pruning in ftk_Storage — one skips whole files for performance, the other gives Power BI a row-level filter it can bind incremental refresh to. ChargePeriodStart is compared via DateTime.From(...) since it's typed datetimezone coming out of the earlier Fixes step, while RangeStart/RangeEnd are datetime.
  • src/power-bi/queries/ftk_NormalizeSchema.pq: Changed ChargePeriodStart's cast from type date to type datetime, resolving the // TODO: Should these be datetimezone? comment. FOCUS ChargePeriodStart values are date-only in practice, so datetime (not datetimezone) matches the RangeStart/RangeEnd parameter type without adding unneeded timezone semantics.
  • Checked the other storage-dataset tables (Prices, ReservationDetails, ReservationTransactions, ReservationRecommendations) for the same type mismatch — none have it (their date columns are already cast to datetime), and ReservationRecommendations is explicitly excluded from incremental-refresh support by design in ftk_Storage. No changes were needed there.
  • docs-mslearn/toolkit/power-bi/setup.md: Updated the RangeStart/RangeEnd guidance to explain incremental refresh now works out of the box for the Costs table, with the steps to enable it (Report view → right-click Costs → Incremental refresh and archive settings → configure → publish). No Power Query edits needed first.
  • docs-mslearn/toolkit/changelog.md: Added an "Unreleased" entry under Power BI reports.

Closes #2314

Test plan

This is Power BI TMDL/M code that can't be exercised end-to-end without Power BI Desktop, so validation here is syntactic/structural rather than a live refresh:

  • Verified parenthesis/bracket/brace balance in the modified M query blocks (Costs.tmdl partition, ftk_NormalizeSchema.pq) is unchanged relative to the pre-existing baseline, i.e. the additions are self-balanced.
  • Confirmed RangeStart/RangeEnd are referenced unqualified elsewhere in the same dataset scope from within a table's own partition query (e.g. Costs.tmdl already does PerformDeprecatedOptimizations = #"Deprecated: Perform Extra Query Optimizations"), confirming the same unqualified-reference pattern used here is valid in this project's TMDL structure.
  • Ran markdownlint-cli2 against the two changed docs files — no issues.
  • Not verified: actually opening the Costs table's Incremental refresh and archive settings dialog in Power BI Desktop and confirming it now detects RangeStart/RangeEnd and that a configured incremental refresh policy publishes and runs correctly. This requires Power BI Desktop/Service and real storage data, which isn't available in this environment.

Confidence: Medium-high on syntax/structure and on the type-mismatch fix; the one open risk is whether comparing a datetimezone column (via DateTime.From(...)) against datetime parameters behaves exactly as expected in Power BI's incremental refresh partitioning — this should be spot-checked in Power BI Desktop before merging.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

…tal refresh

RangeStart/RangeEnd were only used inside the shared ftk_Storage function
to skip whole parquet files outside the requested range (a file-level
optimization), so Power BI's Incremental refresh and archive settings
dialog had no row-level filter on the Costs table to bind them to. The
Costs query now also filters rows directly with
Table.SelectRows(..., each [ChargePeriodStart] >= RangeStart and
[ChargePeriodStart] < RangeEnd), guarded for null so normal (non-
incremental) refreshes keep working.

Also fixes ChargePeriodStart being normalized to type date instead of
type datetime in the shared ftk_NormalizeSchema query, which didn't
match the datetime-typed RangeStart/RangeEnd parameters used elsewhere
for incremental refresh.

Closes #2314

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [AI][Claude Code] PR Review

Summary: Fixes the same issue as #2353 (#2314) with a narrower, riskier approach. Recommend closing this PR in favor of #2353 rather than landing it first.

🚫 Blockers (2)

  1. ftk_NormalizeSchema.pq is only used by ftk_LoadData.pq (Cost Management connector reports). The storage Costs query has its own inline Fixes step that already types ChargePeriodStart as datetimezone, so this change doesn't affect the storage reports it targets. It does change ChargePeriodStart from date to datetime for other reports (downstream type date columns like ChargePeriodEnd and Date.ToText keys can change).
  2. DateTime.From([ChargePeriodStart]) on a datetimezone value converts to the local time zone, so rows can land on the wrong day near midnight depending on the machine or service time zone. The row filter also doesn't match the ftk_Storage file pruning, which uses x_IncrementalRefreshDate.

⚠️ Should fix (2)

  1. Overlaps #2353 in Costs.tmdl, setup.md, and changelog.md. Landing both stacks two filters on the same query and conflicts in all three files. #2353 is conflict-free with dev and already approved.
  2. Branch conflicts with dev in changelog.md.

Decision: close in favor of #2353, which filters on x_IncrementalRefreshDate (consistent with ftk_Storage, null-date handling, Date/Time typing, test added).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs: Review 👀 PR that is ready to be reviewed Tool: Power BI Power BI reports

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Power BI storage reports: incremental refresh isn't detectable out of the box

2 participants