Skip to content

[CDAP-21235] Exclude .raw stage metrics from pipeline UI metrics query - #1436

Merged
sahusanket merged 2 commits into
developfrom
CDAP-21235_exclude_raw_metrics_ui
Oct 9, 2026
Merged

sahusanket merged 2 commits into
developfrom
CDAP-21235_exclude_raw_metrics_ui

Conversation

@sahusanket

@sahusanket sahusanket commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

PR cdapio/cdap#16182 (CDAP-21235) introduced .raw task-level stage metrics (user.<stage>.records.{in,out,error,alert}.raw).

When PipelineMetricsStore/ActionCreator.js queries /v3/metrics/search?target=metric, it matches all metrics containing user.<stage>, which includes *.raw. In parseMetrics(), metricName.indexOf(key + '.records.out.') !== -1 treats user.<stage>.records.out.raw as a multi-port output metric (port = ""). Because user.<stage>.records.out is parsed first as a number, assigning metricObj[key].recordsOut[""] = metricValue throws TypeError: Cannot create property '' on number in strict mode, aborting parseMetrics() before PipelineMetricsStore.dispatch(...) and causing all stage RECORDS counters in the Pipeline Studio UI to remain at 0.

Changes

  1. Filter out .raw metrics (!metric.endsWith('.raw')) in getMetrics() before querying /v3/metrics/query.
  2. Guard metricObj[key].recordsOut in parseMetrics() to ensure it is an object before assigning port properties.

TESTING : Local Sandbox Verification (DataFusionQuickstart)

Verified on local cdap-sandbox-6.12.0-SNAPSHOT built from cdap (develop with #16182 .raw metrics) + cdap-ui (CDAP-21235_exclude_raw_metrics_ui):

  • REST API (/v3/metrics/search?target=metric & /v3/metrics/query): Both standard and .raw metrics are discoverable and populated (records.in.raw = 10195, records.out.raw = 10195 on source; records.in.raw = 10195, records.out.raw = 22 on Wrangler transform; records.in.raw = 22, records.out.raw = 22 on BigQuery sink).
  • CDAP UI (DataFusionQuickstart): Displays Out 10,195, Out 22, and In 22 across all three stages without error.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request excludes per-task-attempt '.raw' metrics from stage metrics and adds a type check to prevent a TypeError when processing 'recordsOut' metrics. The review feedback highlights a remaining issue where an aggregate metric processed after a port-specific metric could still overwrite the port-specific metrics object, and suggests adding a guard to the corresponding 'else' block to prevent this.

if (metricName.indexOf(key + '.records.out.') !== -1) {
const port = split[split.length - 1];
if (!metricObj[key].recordsOut) {
if (typeof metricObj[key].recordsOut !== 'object' || !metricObj[key].recordsOut) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

While guarding metricObj[key].recordsOut against non-object types prevents a TypeError when a port-specific metric is processed after an aggregate metric, the reverse order of processing still presents an issue. Specifically, if a port-specific metric (e.g., user.<stage>.records.out.port1) is processed first, recordsOut is initialized as an object. If the aggregate metric (user.<stage>.records.out) is processed afterwards, the else block (line 120) will unconditionally overwrite the port-specific metrics object with the aggregate number value. Consider guarding the else block as well to prevent overwriting the port-specific metrics object.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guarded the else branch as well so recordsOut is only set to the scalar value when it has not already been populated as a port map object

@radhikav1 radhikav1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please merge all the commits before submitting.

@sahusanket
sahusanket merged commit 900128f into develop Oct 9, 2026
12 checks passed
@sahusanket
sahusanket deleted the CDAP-21235_exclude_raw_metrics_ui branch October 9, 2026 07:00
vsethi09 added a commit that referenced this pull request Oct 9, 2026
[cherry-pick][6.10][CDAP-21235] Cherry-pick #1436 to release/6.10: Exclude .raw stage metrics from pipeline UI metrics query
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants