Skip to content

sigproc: fix OOB block read in decon_2D that crashes restore_baseline (#491) - #493

Open
HaiwangYu wants to merge 1 commit into
WireCell:masterfrom
HaiwangYu:fix/issue-491-decon2d-oob-master
Open

sigproc: fix OOB block read in decon_2D that crashes restore_baseline (#491)#493
HaiwangYu wants to merge 1 commit into
WireCell:masterfrom
HaiwangYu:fix/issue-491-decon2d-oob-master

Conversation

@HaiwangYu

Copy link
Copy Markdown
Member

Fixes #491.

Symptom

The ICARUS 2D signal-processing chain crashes with

Waveform.cxx: Throw in function percentile
Dynamic exception type: boost::wrapexcept<WireCell::ValueError>
[WireCell::tag_errmsg*] = empty waveform

thrown from OmnibusSigProc::restore_baseline via
OmnibusSigProc::decon_2D_tighterROI, intermittently ("event 2" in CI, not seen
in production).

Root cause

Every 2D-decon helper extracts the physical wires with

m_r_data[plane] = tm_r_data.block(m_pad_nwires[plane], 0, m_nwires[plane], m_nticks);

which assumes tm_r_data still carries the FFT wire-padding (m_fft_nwires
rows). For the ICARUS "twofaced" layout the induction planes are split into ≥2
sub-planes, so decon_2D_init() calls unpad_data(), which shrinks the array
to m_nwires rows before the final m_c_data = fwd_r2c(m_r_data). Starting the
block at row m_pad_nwires then reads m_pad_nwires rows past the end.

Observed at runtime (plane 0, TPC WW):

tm_r_data = (2112, 4296)   block(start_row=10, nwires=2112, nticks=4096)   fft_nwires=2132

→ rows [10, 2122) from a 2112-row array → 10 rows out of bounds.

Production libraries are built with -DNDEBUG, so the Eigen bounds check is
compiled out and the read silently returns adjacent heap memory, which is
occasionally NaN/Inf depending on allocator/run/data. That NaN reaches
restore_baseline(): median() → NaN, the |x-baseline|<500 cut rejects every
sample, and median() is then called on an empty vector → "empty waveform".

Fix

  1. Root cause — derive the wire-axis padding offset from the actual row
    count at all decon_2D extraction sites (7 tm_r_data slices + the rawdecon
    finalization slice):

    const int roi_row_pad = (tm_r_data.rows() - m_nwires[plane]) / 2;
    m_r_data[plane] = tm_r_data.block(roi_row_pad, 0, m_nwires[plane], m_nticks);

    This equals m_pad_nwires when the FFT padding is present (unchanged
    behavior) and 0 once unpad_data() has removed it (in-bounds, correct
    wires).

  2. Defense-in-depth — harden restore_baseline() against non-finite
    samples: drop NaN/Inf (zeroing in place) before the baseline estimate and
    guard the second median() against an empty selection. No behavior change
    for finite data.

Validation

Rebuilt libWireCellSigProc.so from the v0.36.1 source (the version the
failing icaruscode v10_20_09p01 loads) with Eigen assertions ON:

  • Unpatched: aborts at the out-of-bounds .block() in
    decon_2D_tighterROI during the WW-TPC deconvolution on the CI input file.
  • Patched: the same job runs both events through 2D signal processing with
    no assertion and no "empty waveform" (exit status 0), and the
    restore_baseline non-finite guard never fires — i.e. removing the overrun
    removes the NaN at its source.

The changed restore_baseline / decon_2D_tighterROI bodies are byte-identical
between this master branch and the validated v0.36.1 tree.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Md5zsMdMEEgatTrumsBoV

…WireCell#491)

The ICARUS 2D signal processing crashed with a boost::wrapexcept<ValueError>
"empty waveform" thrown from Waveform::percentile via
OmnibusSigProc::restore_baseline (issue WireCell#491), intermittently ("event 2" in
CI, not seen in production).

Root cause: every decon_2D_* helper extracted the physical wires with

    tm_r_data.block(m_pad_nwires[plane], 0, m_nwires[plane], m_nticks)

assuming tm_r_data still carries the FFT wire-padding (m_fft_nwires rows).
For the ICARUS "twofaced" layout the induction planes are split into >=2
sub-planes, so decon_2D_init()->unpad_data() has already shrunk the array to
m_nwires rows.  Starting the block at row m_pad_nwires then reads m_pad_nwires
rows past the end (e.g. rows [10,2122) of a 2112-row array).  Production
libraries are built with -DNDEBUG, so the Eigen bounds check is compiled out
and the read silently returns adjacent heap memory, which is occasionally
NaN/Inf depending on allocator/run/data.  That NaN reaches restore_baseline:
median() -> NaN, the |x-baseline|<500 cut rejects every sample, and median()
is then called on an empty vector -> "empty waveform".

Fixes:
- Derive the wire-axis padding offset from the actual row count
  ((tm_r_data.rows() - m_nwires)/2) at all 7 decon_2D_* extraction sites.  This
  equals m_pad_nwires when the FFT padding is present (unchanged behavior) and
  0 once unpad_data() has removed it (in-bounds, correct wires).
- Harden restore_baseline() against non-finite samples as defense-in-depth:
  drop NaN/Inf (zeroing in place) before the baseline estimate and guard the
  second median() against an empty selection.  No behavior change for finite
  data.

Verified by rebuilding libWireCellSigProc.so from the v0.36.1 source with Eigen
assertions ON: the unpatched build aborts at the out-of-bounds block during the
WW-TPC deconvolution on the CI input file, and the patched build runs both
events through 2D signal processing with no assertion and no "empty waveform".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Md5zsMdMEEgatTrumsBoV
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ICARUS CI test failures - "empty waveform"

1 participant