Repository navigation
fix: validate research evidence and support affected named tool calls - #2
Open
JeffPengCoder wants to merge 2 commits into
Open
JeffPengCoder wants to merge 2 commits into
JeffPengCoder wants to merge 2 commits into
Conversation
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream submission: NVIDIA PR #86 is based on upstream main
7cbca70, with the same core fix plus synchronized installation-builder controls. This fork PR retains the deployed Spark code snapshot.Description
IRAOP can return HTTP/MCP success with planning text or fabricated facts and no document sources, and affected inference configurations can emit named search arguments as ordinary content without executing a search. Require a nonempty answer plus a usable filename/page reference; otherwise return structured
status: "incomplete", emit an error event and withhold the unverified answer from the main model while retaining the complete original MCP result in native transcriptdetails. Clone the content array before adding the existing scope note so the raw result is unchanged. OpenClaw transportisErroris independent of research completeness.Add an opt-in workaround to the existing LLM compatibility proxy. For a supported initial research request, explicitly select the read-only
opensearch_search, disable first-generation thinking and wrap only complete, schema-valid query JSON into the named tool envelope. The original agent still executes the actual search. Preserve already structured calls, explicit other tool selections, post-tool synthesis and unrelated requests. Reject malformed/truncated/unexpected calls rather than inventing results. Research compatibility and temporary body tracing are disabled by default.Move the proxy into a testable module, include its source/flags in the rollout checksum, handle ConfigMap symlink entrypoints, document enable/export/disable steps and refresh the generated agent documentation bundle. No ignore, CI policy, shared model startup or cluster write-permission changes.
Validation:
c3a2048.Checklist
Follow-up live validation on the retained Spark deployment:
isError: trueand no fabricated code.a011dcc8UI still did not forward IRAOP subagent progress: its tool-name filter only accepts the Observability/Grafana/DCGM prefixes. Machine tokens match, and the event endpoint rejects invalid input with 400. The normal tool result and sources remain visible. This is a separately identified limitation of that installed UI image; closed UI source is retained only internally.MOSAIC Alert Test. The API returned both code fields as 0, and the user confirmed receipt of the matching run ID. This is explicit report delivery, not automatic publishing or the native Feishu channel.Additional runtime validation and limits (2026-10-31)
A new isolated Pod failure triggers a real CrashLoopBackOff PrometheusRule, Alertmanager and MOSAIC polling. The actual View investigation and subsequent same-session review preserve29 messages,18 tool results and3 IRAOP calls, each returning2 sources. The complete reviewed alert/report is accepted by the Feishu bot (business code0); group-window confirmation for this new run is pending. All test objects are cleaned up. Spark uses the existing deployed backport corresponding to fork PR2, not a deployment of this rebased upstream head.
This test also exposes a separate content-quality limit: the first2 sourced research answers misattribute user-provided incident details to the control PDF and reuse the wrong injection example. They required explicit operator correction and a document-only follow-up. This PR's tool-protocol and source-presence guard does not guarantee sentence-level grounding, comprehensive GPU health or automatic unattended report publication. OpenClaw persists5 large details as truncation metadata; the available complete content and exact limitation are retained.
Full procedure, all selected raw evidence, failed attempts, original answers and review diff. No additional runtime code changes were made for this follow-up.
GPU validation and unattended-analysis limit (2026-10-31)
A separate GPU experiment exercised a new private-socket DCGM cache: fields230/311/313 read95/1/1, while production DCGM stayed0/0/0 and all8 target L20 GPUs retained their NVML ECC/remap/retired-page state. A clearly labelled snapshot replay was scraped through a new ServiceMonitor, a real45s Prometheus rule, Alertmanager and the MOSAIC UI. This is neither CPU hardware corruption nor a physical GPU defect/RMA test; the specialized Hardware Agent was disabled.
The first real-UI autonomous report made11 read-only tool calls. A Codex-guided evidence follow-up added14 calls; an explicit factual-correction prompt added4, for29 calls and41 messages total. Two actual IRAOP queries retrieved page1/2 sources from a new2-page/5-chunk GPU PDF, including the page2 verification code; all1920 embedding coordinates are retained. Follow-up instructions and corrections are not unattended baseline capability.
Unattended operational analysis did not pass. Original Qwen output misread the DCGM module denylist and timestamp offset; the guided Chinese report also used wrong metric labels, confused10 samples with10seconds and recommended clearing kernel evidence with
dmesg -C. No destructive command was executed. Original outputs, the explicit Codex correction, Qwen revision and complete diff are retained separately. A corrected revision does not turn the original result into a pass, and Codex checking is not human approval.The Feishu test group received an explicitly labelled failed-acceptance record with the deidentified, fact-uncorrected Chinese report:16359bytes before signing, no truncation, successful business response after one rejected rate-limit attempt, and user confirmation of this run ID. The original sensitive payload was blocked by automatic approval and never sent. Six exact-UID test objects were removed; source alerts are0/0;41 messages remain readable. Helm15/7Pods and the shared model PID/start time are unchanged, and the root control connection is closed.
This evidence confirms transport/retrieval behavior while exposing semantic/safety gaps. The existing source-presence/tool-protocol fix does not implement fact checking, report safety gating or reliable unattended GPU root-cause analysis. Those remain explicit follow-up work; no runtime code was added for this GPU test.
Separate GPU procedure, CPU-vs-GPU comparison and complete original evidence.