fix(orionbelt): export metric expressions that compute what OrionBelt computes - #472
Open
ralfbecher wants to merge 1 commit into
Open
ralfbecher wants to merge 1 commit into
ralfbecher wants to merge 1 commit into
Conversation
… computes
Other Ossie consumers read a metric's SQL, not the ORIONBELT extension. That
SQL referenced the physical table instead of the dataset, dropped measure
filters and totals, left {[Name]} references to counts and metrics unresolved,
and exported period-over-period as prev.value.
Export (new _portable.py renderer):
- reference "<dataset>"."<field>", always double-quoted so reserved words parse
- filters as AGG(CASE WHEN ... END), totals as the grand-total window
SUM(SUM(x)) OVER () (exact SUM/COUNT ratio for AVG), defaultValue as
COALESCE, count_distinct as COUNT(DISTINCT ...)
- inline synthesized counts and metric-on-metric references
- cumulative and window metrics order and partition by each dimension at its
timeGrain, cast back to a temporal resultType, as the query groups it
- leave out period-over-period, grain/filterContext/anchor measures, nested
windows and reference cycles with a warning, kept whole in the model
extension (obml_unexported)
- tag aggregation: measure as DATABRICKS
- carry each column's OBML name and each measure/metric's OBML definition
Import:
- restore definitions, left-out entities and column names, so
OBML -> Ossie -> OBML returns the model it was given instead of
re-parsing filters and totals out of the SQL
New tests run every exported metric in DuckDB, grouped as OrionBelt groups it
(skipped when duckdb or sqlglot is not installed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Other Ossie consumers (the Databricks, Snowflake and dbt converters, BI tools) read a metric's SQL expression, not the
ORIONBELTextension. The OrionBelt converter exported SQL that did not compute what OrionBelt computes:SUM(fact_orders.amount): physical table, not the datasetSUM("Orders"."amount"):<dataset>.<field>, always double-quotedSUM(CASE WHEN <condition> THEN <arg> END)total: truedroppedSUM(SUM(x)) OVER ()(exactSUM/COUNTratio forAVG){[Order Count]}and metric-on-metric refs left unresolvedx / NULLIF(prev.value, 0) - 1(prevundefined)timeGrain, as the query groups itA measure or metric with no faithful single expression (period-over-period,
grain,filterContext,anchor, window over a window, metric reference cycles) is left out ofmetricswith a warning and kept in the model-level extension (obml_unexported).The rendering lives in a new
_portable.py. It mirrors the OrionBelt compiler and adds no dependencies.Round trip: every exported measure/metric carries its OBML definition (
obml_definition), and every field its OBML column name (obml_column_name). OBML to Ossie to OBML now restores the original model instead of re-parsing filters and totals out of the SQL, where they would be applied twice. Column keys come back under their OBML names, not the physical codes, so several existing test assertions changed accordingly.aggregation: measureexportsMEASURE("<name>")taggedDATABRICKSinstead ofANSI_SQL.These fixes shipped in OrionBelt's in-repo copy of this converter first (ralforion/orionbelt-semantic-layer#480). They were reviewed there and verified against DuckDB.
Note: #410 (datatype in both directions) also edits
obml_to_ossie.pyandossie_to_obml.py. Whichever lands second needs a small rebase. I'll do it.Related Issues
None filed. Found while testing the OrionBelt to Ossie to Databricks Metric View chain.
Checklist
Converters
converters/is updatedconverters/orionbelt/tests/: newtest_ossie_portable_expressions.py, which runs every exported metric in DuckDB grouped as OrionBelt groups it (skipped whenduckdb/sqlglotare not installed)Documentation
converters/orionbelt/README.mdandossie_obml_mapping_analysis.mdupdatedTests
pytestinconverters/orionbelt: 186 passed, 3 skipped (189 passed withduckdbandsqlglot)ruff check/mypyreport nothing newCompliance