Repository navigation
[CDAP-21235] Exclude .raw stage metrics from pipeline UI metrics query - #1436
Conversation
There was a problem hiding this comment.
Code Review
This pull request excludes per-task-attempt '.raw' metrics from stage metrics and adds a type check to prevent a TypeError when processing 'recordsOut' metrics. The review feedback highlights a remaining issue where an aggregate metric processed after a port-specific metric could still overwrite the port-specific metrics object, and suggests adding a guard to the corresponding 'else' block to prevent this.
| if (metricName.indexOf(key + '.records.out.') !== -1) { | ||
| const port = split[split.length - 1]; | ||
| if (!metricObj[key].recordsOut) { | ||
| if (typeof metricObj[key].recordsOut !== 'object' || !metricObj[key].recordsOut) { |
There was a problem hiding this comment.
While guarding metricObj[key].recordsOut against non-object types prevents a TypeError when a port-specific metric is processed after an aggregate metric, the reverse order of processing still presents an issue. Specifically, if a port-specific metric (e.g., user.<stage>.records.out.port1) is processed first, recordsOut is initialized as an object. If the aggregate metric (user.<stage>.records.out) is processed afterwards, the else block (line 120) will unconditionally overwrite the port-specific metrics object with the aggregate number value. Consider guarding the else block as well to prevent overwriting the port-specific metrics object.
There was a problem hiding this comment.
Guarded the else branch as well so recordsOut is only set to the scalar value when it has not already been populated as a port map object
… overwriting port object
radhikav1
left a comment
There was a problem hiding this comment.
please merge all the commits before submitting.
[cherry-pick][6.10][CDAP-21235] Cherry-pick #1436 to release/6.10: Exclude .raw stage metrics from pipeline UI metrics query
Description
PR cdapio/cdap#16182 (
CDAP-21235) introduced.rawtask-level stage metrics (user.<stage>.records.{in,out,error,alert}.raw).When
PipelineMetricsStore/ActionCreator.jsqueries/v3/metrics/search?target=metric, it matches all metrics containinguser.<stage>, which includes*.raw. InparseMetrics(),metricName.indexOf(key + '.records.out.') !== -1treatsuser.<stage>.records.out.rawas a multi-port output metric (port = ""). Becauseuser.<stage>.records.outis parsed first as anumber, assigningmetricObj[key].recordsOut[""] = metricValuethrowsTypeError: Cannot create property '' on numberin strict mode, abortingparseMetrics()beforePipelineMetricsStore.dispatch(...)and causing all stageRECORDScounters in the Pipeline Studio UI to remain at0.Changes
.rawmetrics (!metric.endsWith('.raw')) ingetMetrics()before querying/v3/metrics/query.metricObj[key].recordsOutinparseMetrics()to ensure it is an object before assigning port properties.TESTING : Local Sandbox Verification (
DataFusionQuickstart)Verified on local
cdap-sandbox-6.12.0-SNAPSHOTbuilt fromcdap(developwith#16182.rawmetrics) +cdap-ui(CDAP-21235_exclude_raw_metrics_ui):/v3/metrics/search?target=metric&/v3/metrics/query): Both standard and.rawmetrics are discoverable and populated (records.in.raw = 10195,records.out.raw = 10195on source;records.in.raw = 10195,records.out.raw = 22on Wrangler transform;records.in.raw = 22,records.out.raw = 22on BigQuery sink).DataFusionQuickstart): DisplaysOut 10,195,Out 22, andIn 22across all three stages without error.