Repository navigation
fix: validate IRAOP evidence and support affected named search calls - #86
Open
JeffPengCoder wants to merge 1 commit into
Open
JeffPengCoder wants to merge 1 commit into
JeffPengCoder wants to merge 1 commit into
Conversation
…lity Signed-off-by: Jeff Peng <jepeng@nvidia.com>
JeffPengCoder
requested review from
RohanAdwankar and
kgengedullay
as code owners
October 31, 2026 07:37
This was referenced Oct 31, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
IRAOP can return successful MCP transport with planning text or unsupported claims but no document sources.
iraop_querycurrently announces completion and forwards that answer to the main model. Require a nonempty answer and at least one usable filename/page reference; otherwise return structuredstatus: "incomplete", emit an error event and withhold the unverified answer from model-visible content. Keep the complete original MCP response in native transcriptdetails, and clone its content array before adding the existing evidence-scope note.Affected inference configurations also return named search arguments as ordinary content, so the research agent never executes the search. Add a default-off workaround in the existing LLM compatibility proxy: select the initial read-only
opensearch_searchfor the supported query-only schema, validate the complete actual JSON arguments and restore their tool-call envelope. The original agent executes the real search. Preserve already structured calls, unrelated tools and post-tool synthesis; reject malformed, incomplete or unexpected calls. No fabricated retrieval results or answers.Include proxy source and flags in the rollout checksum, handle ConfigMap symlink entrypoints, and provide optional temporary full-body tracing with export/disable instructions. Tracing is off by default. Synchronize the installation guide, builder checkbox/Helm flags and generated agent documentation. This branch is based on current upstream main
7cbca70and retains its plugin SDK entry point and other recent changes.Validation:
Related: #31 adjusts reasoning-response handling in the same proxy. This change targets research evidence validation and named search arguments; the new behavior remains opt-in. There are no ignore, CI policy or cluster write-permission changes.
Checklist
Additional runtime validation and limits (2026-10-31)
A new isolated Pod failure triggers a real CrashLoopBackOff PrometheusRule, Alertmanager and MOSAIC polling. The actual View investigation and subsequent same-session review preserve29 messages,18 tool results and3 IRAOP calls, each returning2 sources. The complete reviewed alert/report is accepted by the Feishu bot (business code0); group-window confirmation for this new run is pending. All test objects are cleaned up. Spark uses the existing deployed backport corresponding to fork PR2, not a deployment of this rebased upstream head.
This test also exposes a separate content-quality limit: the first2 sourced research answers misattribute user-provided incident details to the control PDF and reuse the wrong injection example. They required explicit operator correction and a document-only follow-up. This PR's tool-protocol and source-presence guard does not guarantee sentence-level grounding, comprehensive GPU health or automatic unattended report publication. OpenClaw persists5 large details as truncation metadata; the available complete content and exact limitation are retained.
Full procedure, all selected raw evidence, failed attempts, original answers and review diff. No additional runtime code changes were made for this follow-up.
GPU validation and unattended-analysis limit (2026-10-31)
A separate GPU experiment exercised a new private-socket DCGM cache: fields230/311/313 read95/1/1, while production DCGM stayed0/0/0 and all8 target L20 GPUs retained their NVML ECC/remap/retired-page state. A clearly labelled snapshot replay was scraped through a new ServiceMonitor, a real45s Prometheus rule, Alertmanager and the MOSAIC UI. This is neither CPU hardware corruption nor a physical GPU defect/RMA test; the specialized Hardware Agent was disabled.
The first real-UI autonomous report made11 read-only tool calls. A Codex-guided evidence follow-up added14 calls; an explicit factual-correction prompt added4, for29 calls and41 messages total. Two actual IRAOP queries retrieved page1/2 sources from a new2-page/5-chunk GPU PDF, including the page2 verification code; all1920 embedding coordinates are retained. Follow-up instructions and corrections are not unattended baseline capability.
Unattended operational analysis did not pass. Original Qwen output misread the DCGM module denylist and timestamp offset; the guided Chinese report also used wrong metric labels, confused10 samples with10seconds and recommended clearing kernel evidence with
dmesg -C. No destructive command was executed. Original outputs, the explicit Codex correction, Qwen revision and complete diff are retained separately. A corrected revision does not turn the original result into a pass, and Codex checking is not human approval.The Feishu test group received an explicitly labelled failed-acceptance record with the deidentified, fact-uncorrected Chinese report:16359bytes before signing, no truncation, successful business response after one rejected rate-limit attempt, and user confirmation of this run ID. The original sensitive payload was blocked by automatic approval and never sent. Six exact-UID test objects were removed; source alerts are0/0;41 messages remain readable. Helm15/7Pods and the shared model PID/start time are unchanged, and the root control connection is closed.
This evidence confirms transport/retrieval behavior while exposing semantic/safety gaps. The existing source-presence/tool-protocol fix does not implement fact checking, report safety gating or reliable unattended GPU root-cause analysis. Those remain explicit follow-up work; no runtime code was added for this GPU test.
Separate GPU procedure, CPU-vs-GPU comparison and complete original evidence.