Repository navigation
fix(power-bi): wire RangeStart/RangeEnd into Costs query for incremental refresh - #2317
Draft
Michael Flanakin (flanakin) wants to merge 2 commits into
Draft
Michael Flanakin (flanakin) wants to merge 2 commits into
Michael Flanakin (flanakin) wants to merge 2 commits into
Conversation
…tal refresh RangeStart/RangeEnd were only used inside the shared ftk_Storage function to skip whole parquet files outside the requested range (a file-level optimization), so Power BI's Incremental refresh and archive settings dialog had no row-level filter on the Costs table to bind them to. The Costs query now also filters rows directly with Table.SelectRows(..., each [ChargePeriodStart] >= RangeStart and [ChargePeriodStart] < RangeEnd), guarded for null so normal (non- incremental) refreshes keep working. Also fixes ChargePeriodStart being normalized to type date instead of type datetime in the shared ftk_NormalizeSchema query, which didn't match the datetime-typed RangeStart/RangeEnd parameters used elsewhere for incremental refresh. Closes #2314
microsoft-github-policy-service
Bot
requested a review
from Brett Wilson (MSBrett)
September 9, 2026 20:20
Michael Flanakin (flanakin)
left a comment
Collaborator
Author
There was a problem hiding this comment.
🤖 [AI][Claude Code] PR Review
Summary: Fixes the same issue as #2353 (#2314) with a narrower, riskier approach. Recommend closing this PR in favor of #2353 rather than landing it first.
🚫 Blockers (2)
ftk_NormalizeSchema.pqis only used byftk_LoadData.pq(Cost Management connector reports). The storageCostsquery has its own inlineFixesstep that already typesChargePeriodStartasdatetimezone, so this change doesn't affect the storage reports it targets. It does changeChargePeriodStartfromdatetodatetimefor other reports (downstreamtype datecolumns likeChargePeriodEndandDate.ToTextkeys can change).DateTime.From([ChargePeriodStart])on adatetimezonevalue converts to the local time zone, so rows can land on the wrong day near midnight depending on the machine or service time zone. The row filter also doesn't match theftk_Storagefile pruning, which usesx_IncrementalRefreshDate.
⚠️ Should fix (2)
- Overlaps #2353 in
Costs.tmdl,setup.md, andchangelog.md. Landing both stacks two filters on the same query and conflicts in all three files. #2353 is conflict-free withdevand already approved. - Branch conflicts with
devinchangelog.md.
Decision: close in favor of #2353, which filters on x_IncrementalRefreshDate (consistent with ftk_Storage, null-date handling, Date/Time typing, test added).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Power BI's Incremental refresh and archive settings dialog couldn't be configured for the Costs table in the storage-based reports because
RangeStart/RangeEndwere only referenced inside the sharedftk_Storagefunction — used there purely as a file-level optimization to skip whole parquet files outside the requested range. Power BI only detects incremental refresh support when a table's own query directly filters rows usingTable.SelectRows(..., each [Column] >= RangeStart and [Column] < RangeEnd).Changes
src/power-bi/storage/Shared.Dataset/definition/tables/Costs.tmdl: Added aTable.SelectRowsstep as the final step of the Costs partition query that filters on[ChargePeriodStart] >= RangeStart and [ChargePeriodStart] < RangeEnd(null-guarded so normal, non-incremental refreshes still load all data by default). This coexists with the existing file-level pruning inftk_Storage— one skips whole files for performance, the other gives Power BI a row-level filter it can bind incremental refresh to.ChargePeriodStartis compared viaDateTime.From(...)since it's typeddatetimezonecoming out of the earlierFixesstep, whileRangeStart/RangeEndaredatetime.src/power-bi/queries/ftk_NormalizeSchema.pq: ChangedChargePeriodStart's cast fromtype datetotype datetime, resolving the// TODO: Should these be datetimezone?comment. FOCUSChargePeriodStartvalues are date-only in practice, sodatetime(notdatetimezone) matches theRangeStart/RangeEndparameter type without adding unneeded timezone semantics.Prices,ReservationDetails,ReservationTransactions,ReservationRecommendations) for the same type mismatch — none have it (their date columns are already cast todatetime), andReservationRecommendationsis explicitly excluded from incremental-refresh support by design inftk_Storage. No changes were needed there.docs-mslearn/toolkit/power-bi/setup.md: Updated theRangeStart/RangeEndguidance to explain incremental refresh now works out of the box for the Costs table, with the steps to enable it (Report view → right-click Costs → Incremental refresh and archive settings → configure → publish). No Power Query edits needed first.docs-mslearn/toolkit/changelog.md: Added an "Unreleased" entry under Power BI reports.Closes #2314
Test plan
This is Power BI TMDL/M code that can't be exercised end-to-end without Power BI Desktop, so validation here is syntactic/structural rather than a live refresh:
Costs.tmdlpartition,ftk_NormalizeSchema.pq) is unchanged relative to the pre-existing baseline, i.e. the additions are self-balanced.RangeStart/RangeEndare referenced unqualified elsewhere in the same dataset scope from within a table's own partition query (e.g.Costs.tmdlalready doesPerformDeprecatedOptimizations = #"Deprecated: Perform Extra Query Optimizations"), confirming the same unqualified-reference pattern used here is valid in this project's TMDL structure.markdownlint-cli2against the two changed docs files — no issues.RangeStart/RangeEndand that a configured incremental refresh policy publishes and runs correctly. This requires Power BI Desktop/Service and real storage data, which isn't available in this environment.Confidence: Medium-high on syntax/structure and on the type-mismatch fix; the one open risk is whether comparing a
datetimezonecolumn (viaDateTime.From(...)) againstdatetimeparameters behaves exactly as expected in Power BI's incremental refresh partitioning — this should be spot-checked in Power BI Desktop before merging.Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com