Skip to content

fix: validate IRAOP evidence and support affected named search calls - #86

Open
JeffPengCoder wants to merge 1 commit into
NVIDIA:mainfrom
JeffPengCoder:fix/iraop-research-evidence-upstream
Open

JeffPengCoder wants to merge 1 commit into
NVIDIA:mainfrom
JeffPengCoder:fix/iraop-research-evidence-upstream

Conversation

@JeffPengCoder

@JeffPengCoder JeffPengCoder commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Description

IRAOP can return successful MCP transport with planning text or unsupported claims but no document sources. iraop_query currently announces completion and forwards that answer to the main model. Require a nonempty answer and at least one usable filename/page reference; otherwise return structured status: "incomplete", emit an error event and withhold the unverified answer from model-visible content. Keep the complete original MCP response in native transcript details, and clone its content array before adding the existing evidence-scope note.

Affected inference configurations also return named search arguments as ordinary content, so the research agent never executes the search. Add a default-off workaround in the existing LLM compatibility proxy: select the initial read-only opensearch_search for the supported query-only schema, validate the complete actual JSON arguments and restore their tool-call envelope. The original agent executes the real search. Preserve already structured calls, unrelated tools and post-tool synthesis; reject malformed, incomplete or unexpected calls. No fabricated retrieval results or answers.

Include proxy source and flags in the rollout checksum, handle ConfigMap symlink entrypoints, and provide optional temporary full-body tracing with export/disable instructions. Tracing is off by default. Synchronize the installation guide, builder checkbox/Helm flags and generated agent documentation. This branch is based on current upstream main 7cbca70 and retains its plugin SDK entry point and other recent changes.

Validation:

  • Full CI command: 275 tests, 273 passed, 2 optional skips. Includes 13 actual HTTP/MCP/SSE query-evidence cases, 30 proxy HTTP/SSE/CLI-startup cases, Helm rendering and installation-builder selection/rendering.
  • Standalone inference: 4 tests passed. Both Helm charts lint cleanly. Sphinx builds with warnings as errors, generated-document freshness and staged whitespace checks pass.
  • Actual rendered installation builder: four browser cases pass (default, Research selected, workaround selected, Research removed). The opt-in requires Research and does not enable body tracing.
  • Core runtime fix was backported to an existing persistent Spark deployment. Native HTTP/MCP and actual View-mode UI queries retrieved the control PDF's correct code and filename/page references; questions did not contain the code. A missing collection returns an explicit backend/tool error, and a question whose answer is absent declined to invent a GPU serial number. Full original requests/results and failed attempts are retained in the separate operator evidence PR.
  • All 12 application Secrets and the shared model process were retained. Temporary tracing was exported and disabled. This controlled corpus validation does not certify business-document relevance or all answer/citation claims. Research completion remains distinct from fault repair, and the older installed UI image has a separate progress-event filtering limitation.

Related: #31 adjusts reasoning-response handling in the same proxy. This change targets research evidence validation and named search arguments; the new behavior remains opt-in. There are no ignore, CI policy or cluster write-permission changes.

Checklist

Additional runtime validation and limits (2026-10-31)

A new isolated Pod failure triggers a real CrashLoopBackOff PrometheusRule, Alertmanager and MOSAIC polling. The actual View investigation and subsequent same-session review preserve29 messages,18 tool results and3 IRAOP calls, each returning2 sources. The complete reviewed alert/report is accepted by the Feishu bot (business code0); group-window confirmation for this new run is pending. All test objects are cleaned up. Spark uses the existing deployed backport corresponding to fork PR2, not a deployment of this rebased upstream head.

This test also exposes a separate content-quality limit: the first2 sourced research answers misattribute user-provided incident details to the control PDF and reuse the wrong injection example. They required explicit operator correction and a document-only follow-up. This PR's tool-protocol and source-presence guard does not guarantee sentence-level grounding, comprehensive GPU health or automatic unattended report publication. OpenClaw persists5 large details as truncation metadata; the available complete content and exact limitation are retained.

Full procedure, all selected raw evidence, failed attempts, original answers and review diff. No additional runtime code changes were made for this follow-up.

GPU validation and unattended-analysis limit (2026-10-31)

A separate GPU experiment exercised a new private-socket DCGM cache: fields230/311/313 read95/1/1, while production DCGM stayed0/0/0 and all8 target L20 GPUs retained their NVML ECC/remap/retired-page state. A clearly labelled snapshot replay was scraped through a new ServiceMonitor, a real45s Prometheus rule, Alertmanager and the MOSAIC UI. This is neither CPU hardware corruption nor a physical GPU defect/RMA test; the specialized Hardware Agent was disabled.

The first real-UI autonomous report made11 read-only tool calls. A Codex-guided evidence follow-up added14 calls; an explicit factual-correction prompt added4, for29 calls and41 messages total. Two actual IRAOP queries retrieved page1/2 sources from a new2-page/5-chunk GPU PDF, including the page2 verification code; all1920 embedding coordinates are retained. Follow-up instructions and corrections are not unattended baseline capability.

Unattended operational analysis did not pass. Original Qwen output misread the DCGM module denylist and timestamp offset; the guided Chinese report also used wrong metric labels, confused10 samples with10seconds and recommended clearing kernel evidence with dmesg -C. No destructive command was executed. Original outputs, the explicit Codex correction, Qwen revision and complete diff are retained separately. A corrected revision does not turn the original result into a pass, and Codex checking is not human approval.

The Feishu test group received an explicitly labelled failed-acceptance record with the deidentified, fact-uncorrected Chinese report:16359bytes before signing, no truncation, successful business response after one rejected rate-limit attempt, and user confirmation of this run ID. The original sensitive payload was blocked by automatic approval and never sent. Six exact-UID test objects were removed; source alerts are0/0;41 messages remain readable. Helm15/7Pods and the shared model PID/start time are unchanged, and the root control connection is closed.

This evidence confirms transport/retrieval behavior while exposing semantic/safety gaps. The existing source-presence/tool-protocol fix does not implement fact checking, report safety gating or reliable unattended GPU root-cause analysis. Those remain explicit follow-up work; no runtime code was added for this GPU test.

Separate GPU procedure, CPU-vs-GPU comparison and complete original evidence.

…lity

Signed-off-by: Jeff Peng <jepeng@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant