Skip to content

Fix ingestion-side ListCost repair gaps for ESR/savings accuracy (defects 2-4 from #2214) #2286

Open

Description

🤖 [AI] Split out from #2214, which covers 4 distinct defects behind the ADX dashboard's negative/wrong Effective Savings Rate. Defect 1 (dashboard aggregation bypassing the row-level x_TotalSavings clamp) is being fixed in #2248, scoped to dashboard.json only. This issue tracks the remaining 3, which live in the ingestion/data-repair pipeline (IngestionSetup_v1_0.kql / IngestionSetup_v1_2.kql) rather than the dashboard — a different layer with a different risk profile (write-path changes to stored columns vs. read-only dashboard queries), so it's tracked and will ship as a separate PR from #2248.

Defect 2 — Cost fallback gated behind an unrelated price-sheet lookup key

IngestionSetup_v1_2.kql has a fallback chain (ListCost → ContractedCost → EffectiveCost, lines ~460-487) that would repair rows with ListCost == 0. It never runs for rows without a Microsoft meter/offer ID — e.g. third-party Marketplace/ISV purchases — because the whole block is gated behind tmp_MissingPrices, which requires x_SkuMeterId and x_SkuOfferId to both be set. On one production hub, only 14 of 884 unrepaired rows had both IDs set; 0 passed all five conditions in the gate.

Fix: separate the price-sheet lookup (needs meter/offer ID) from the cost fallback (doesn't need it). Apply the fallback to any row with ListCost == 0 and EffectiveCost != 0, independent of the lookup gate.

Caution: on some of these rows, ContractedCost is already below EffectiveCost in the source export, so list >= contracted >= effective doesn't hold going in. The fallback chain alone won't repair that — decide what invariant to enforce before implementing.

Defect 3 — v1.0 ingestion path writes no x_SourceValues

IngestionSetup_v1_0.kql still repairs ListCost (line ~425) but never records the original value — the x_old_ListCost capture is a commented-out TODO. The v1.2 path does this correctly (10 checkReal() calls, x_SourceValues assignment). On one hub, 2.39M v1.0 rows have a hub-synthesized ListCost with no way to distinguish it from an exported value.

Fix: finish the TODO in IngestionSetup_v1_0.kql to record pre-repair values via x_SourceValues, matching the v1.2 pattern — or explicitly document that x_SourceValues is v1.2-only if backporting isn't feasible.

Defect 4 — MissingListCost flag unusable post-repair

The flag is computed before the repair runs, so it reflects the source export, not the current state. Combined with defect 3, this means v1.0 data has no way to tell "still broken" from "already repaired" — 98.9% of flagged rows on one hub now have a non-zero ListCost after repair, but the flag doesn't reflect that.

Fix: either clear the flag when repair succeeds, or add a distinct "repaired" marker so the flag can answer "what's still wrong" rather than "what was wrong on ingestion."

Minor cleanup noted alongside (optional, same files)

  • Unreachable branch in both cost case statements in IngestionSetup_v1_2.kql (lines ~473, ~486) — two preceding conditions are exact complements, so the final fallback can never be selected.
  • Comment/condition mismatch at IngestionSetup_v1_2.kql:466 — comment says "billed cost" but condition tests EffectiveCost.

Scope note

This is a write-path change to the ingestion pipeline (stored x_SourceValues/flags), not a read-only dashboard fix — different risk profile from #2248, needs its own verification against real export data before merging.

Related: #2214, #2235, #2248

Activity

  1. flanakin commented on Aug 26, 2026

    @flanakin
    CollaboratorAuthor

    🤖 [AI] Opened #2290 (draft) covering defects 2 and 4. Defect 3 (x_SourceValues for v1.0) is out of scope permanently, not deferred — v1.0's schema is intentionally frozen so people can revert to legacy behavior, and it must never gain new columns. v1.2 already handles defect 3's provenance tracking automatically once defect 2's fix lands, no separate work needed there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions