Skip to content

fix: validate research evidence and support affected named tool calls - #2

Open
JeffPengCoder wants to merge 2 commits into
mainfrom
fix/iraop-query-evidence
Open

JeffPengCoder wants to merge 2 commits into
mainfrom
fix/iraop-query-evidence

Conversation

@JeffPengCoder

@JeffPengCoder JeffPengCoder commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Upstream submission: NVIDIA PR #86 is based on upstream main 7cbca70, with the same core fix plus synchronized installation-builder controls. This fork PR retains the deployed Spark code snapshot.

Description

IRAOP can return HTTP/MCP success with planning text or fabricated facts and no document sources, and affected inference configurations can emit named search arguments as ordinary content without executing a search. Require a nonempty answer plus a usable filename/page reference; otherwise return structured status: "incomplete", emit an error event and withhold the unverified answer from the main model while retaining the complete original MCP result in native transcript details. Clone the content array before adding the existing scope note so the raw result is unchanged. OpenClaw transport isError is independent of research completeness.

Add an opt-in workaround to the existing LLM compatibility proxy. For a supported initial research request, explicitly select the read-only opensearch_search, disable first-generation thinking and wrap only complete, schema-valid query JSON into the named tool envelope. The original agent still executes the actual search. Preserve already structured calls, explicit other tool selections, post-tool synthesis and unrelated requests. Reject malformed/truncated/unexpected calls rather than inventing results. Research compatibility and temporary body tracing are disabled by default.

Move the proxy into a testable module, include its source/flags in the rollout checksum, handle ConfigMap symlink entrypoints, document enable/export/disable steps and refresh the generated agent documentation bundle. No ignore, CI policy, shared model startup or cluster write-permission changes.

Validation:

  • Full CI command: 261 tests, 259 passed, 2 optional skips. Includes 30 real HTTP/SSE/CLI-startup proxy cases and the existing 13 query-evidence integration cases. Helm/render, source policy, Sphinx with warnings as errors and documentation-bundle freshness passed; remote Tests/build are green for commit c3a2048.
  • Real Spark retained its authorized private child chart, Teams customizations and all 12 application Secret fingerprints. The ConfigMap startup failure, exact interim correction and subsequent persistent chart deployment are recorded. Final Helm revision 15 has research compatibility enabled and temporary auditing disabled; the proxy has no audit volume. The shared model PID/StartedAt stayed unchanged.
  • Actual deployed HTTP and MCP queries and three View-mode MOSAIC UI queries returned the control PDF's correct page-2 code and sources. The last UI query, after tracing was disabled, carried the existing synthetic alert context. Questions never included the code. Eight complete private traces establish unchanged messages and actual role=tool document content before answer generation; full native transcript details were retained.
  • These controlled retrieval results do not certify business-corpus relevance, all citation claims or general agent quality. Live EventSource/fetch capture still recorded no selected telemetry events; actual subagent-event delivery is not independently confirmed. Event emission is covered by real HTTP integration tests. No new alert injection, production repair or Feishu delivery is claimed.
  • Full dated commands, all eight agent comparisons, six named-wire replays, endpoint/UI outputs, failed attempts and provenance are in handoff PR #1.

Checklist

Follow-up live validation on the retained Spark deployment:

  • A question whose answer is absent explicitly declined to invent a GPU serial number; two document references did not make the missing serial a known fact. A nonexistent collection returned HTTP 502 with the actual missing-index error; the real UI/MCP call surfaced isError: true and no fabricated code.
  • Browser streams are NDJSON. After capturing the correct protocol, the installed a011dcc8 UI still did not forward IRAOP subagent progress: its tool-name filter only accepts the Observability/Grafana/DCGM prefixes. Machine tokens match, and the event endpoint rejects invalid input with 400. The normal tool result and sources remain visible. This is a separately identified limitation of that installed UI image; closed UI source is retained only internally.
  • The separate operator Feishu custom-bot workflow has now actually sent the reviewed full alert/analysis report to MOSAIC Alert Test. The API returned both code fields as 0, and the user confirmed receipt of the matching run ID. This is explicit report delivery, not automatic publishing or the native Feishu channel.

Additional runtime validation and limits (2026-10-31)

A new isolated Pod failure triggers a real CrashLoopBackOff PrometheusRule, Alertmanager and MOSAIC polling. The actual View investigation and subsequent same-session review preserve29 messages,18 tool results and3 IRAOP calls, each returning2 sources. The complete reviewed alert/report is accepted by the Feishu bot (business code0); group-window confirmation for this new run is pending. All test objects are cleaned up. Spark uses the existing deployed backport corresponding to fork PR2, not a deployment of this rebased upstream head.

This test also exposes a separate content-quality limit: the first2 sourced research answers misattribute user-provided incident details to the control PDF and reuse the wrong injection example. They required explicit operator correction and a document-only follow-up. This PR's tool-protocol and source-presence guard does not guarantee sentence-level grounding, comprehensive GPU health or automatic unattended report publication. OpenClaw persists5 large details as truncation metadata; the available complete content and exact limitation are retained.

Full procedure, all selected raw evidence, failed attempts, original answers and review diff. No additional runtime code changes were made for this follow-up.

GPU validation and unattended-analysis limit (2026-10-31)

A separate GPU experiment exercised a new private-socket DCGM cache: fields230/311/313 read95/1/1, while production DCGM stayed0/0/0 and all8 target L20 GPUs retained their NVML ECC/remap/retired-page state. A clearly labelled snapshot replay was scraped through a new ServiceMonitor, a real45s Prometheus rule, Alertmanager and the MOSAIC UI. This is neither CPU hardware corruption nor a physical GPU defect/RMA test; the specialized Hardware Agent was disabled.

The first real-UI autonomous report made11 read-only tool calls. A Codex-guided evidence follow-up added14 calls; an explicit factual-correction prompt added4, for29 calls and41 messages total. Two actual IRAOP queries retrieved page1/2 sources from a new2-page/5-chunk GPU PDF, including the page2 verification code; all1920 embedding coordinates are retained. Follow-up instructions and corrections are not unattended baseline capability.

Unattended operational analysis did not pass. Original Qwen output misread the DCGM module denylist and timestamp offset; the guided Chinese report also used wrong metric labels, confused10 samples with10seconds and recommended clearing kernel evidence with dmesg -C. No destructive command was executed. Original outputs, the explicit Codex correction, Qwen revision and complete diff are retained separately. A corrected revision does not turn the original result into a pass, and Codex checking is not human approval.

The Feishu test group received an explicitly labelled failed-acceptance record with the deidentified, fact-uncorrected Chinese report:16359bytes before signing, no truncation, successful business response after one rejected rate-limit attempt, and user confirmation of this run ID. The original sensitive payload was blocked by automatic approval and never sent. Six exact-UID test objects were removed; source alerts are0/0;41 messages remain readable. Helm15/7Pods and the shared model PID/start time are unchanged, and the root control connection is closed.

This evidence confirms transport/retrieval behavior while exposing semantic/safety gaps. The existing source-presence/tool-protocol fix does not implement fact checking, report safety gating or reliable unattended GPU root-cause analysis. Those remain explicit follow-up work; no runtime code was added for this GPU test.

Separate GPU procedure, CPU-vs-GPU comparison and complete original evidence.

Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder JeffPengCoder changed the title fix: mark source-free IRAOP research incomplete fix: validate research evidence and support affected named tool calls Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant