Skip to content

feat: add Helm-managed native alert automation plugin - #89

Open
JeffPengCoder wants to merge 7 commits into
NVIDIA:mainfrom
JeffPengCoder:feat/background-alert-investigation
Open

JeffPengCoder wants to merge 7 commits into
NVIDIA:mainfrom
JeffPengCoder:feat/background-alert-investigation

Conversation

@JeffPengCoder

@JeffPengCoder JeffPengCoder commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Runtime follow-up fixes are reviewed separately in AFOA fork #3, stacked on this PR’s feature branch. That PR contains bb40700 (service-owned Webhook dispatch and Webhook-only UI recovery) and 85503b2 (durable recipient context across Agent hook registries). This PR’s current head does not yet contain those commits; the live validation below used the base plus that fix stack.

Latest validation (2026-11-03): Spark switched to Webhook-only. A real new GPU-cache alert autonomously ran one native Agent investigation, 11 tool results (1 error) and 3 Feishu DM messages; the source last-poll marker did not change. A real ordinary human follow-up without the answer returned the new report’s correct Run ID and alert name after the context fix, with three full-report context witnesses. Recovery cleared the imported UI alert. The Agent did not select Research this round; earlier Research results below remain dated evidence. Original failed attempts, stale-report replies, complete model output, every tool result and all cleanup checks are retained. Latest fix-stack focused tests: 19/19; full CI: 247 tests, 245 passed, 2 conditional skips, 0 failures.

MOSAIC can investigate Alertmanager alerts and publish a complete report while the browser is closed. An optional native OpenClaw service plugin now owns this lifecycle inside the existing Gateway Pod, so a Helm installation no longer needs a separate deployment-host Python/systemd worker. The selected Agent chooses its evidence sources, tools, order and optional Research queries.

The plugin is channel independent: Agent ID, native outbound channel, account, target and receiving session are configurable. Feishu is the channel used for the live deployment test.

  • Poll Alertmanager, receive an authenticated Webhook, or enable both. Configure label admission, whether to include existing alerts, polling interval, queue capacity and investigation timeout. Default Helm installation leaves this module disabled.
  • Persist incidents, every state transition, full requests/results, tool errors and original reports in SQLite and files on the existing Gateway PVC. Fingerprint plus start time identifies an incident. Interrupted investigations and uncertain/partial external deliveries stop for reconciliation instead of blindly repeating effects.
  • Submit one native api.runtime.subagent investigation goal under the existing View guard; follow actual run completion and retain complete native history. The service does not prescribe collection containers or call Research on the Agent's behalf.
  • Send through the configured native channel adapter. Preserve exact report text in lossless UTF-8 chunks, every chunk hash, every platform receipt/message ID, and complete report integrity metadata. Keep fixed configured recipients outside model control.
  • Bind the complete report to the configured receiving session in durable state. A before_prompt_build hook supplies that report on each real recipient turn, including child reply threads and after Gateway reconstruction. This requires no synthetic Agent ACK, transcript-only mirror, host CLI publisher, or additional publisher-device write grant.
  • Integrate optional module configuration, validation, seed installation/removal, installation-builder dependencies, operating documentation and the generated Agent documentation bundle. The previous standalone-worker guide/code remain available for existing deployments and migration; this feature's normal installation is the Helm-managed plugin.

Validation:

  • Focused native plugin suite: 17 passed. Covers generic configuration, legacy SQLite compatibility, durable deduplication/capacity, one native goal, full history, completion checks, context across multiple turns/restart/child threads, exact Unicode chunking, partial/uncertain sends, service stop, HTTP redirect isolation, and real local authenticated Webhook/service lifecycle with a test SDK runtime.
  • Full repository CI-command run before the final child-thread case: 244 tests, 242 passed, 2 optional skips. Staged source/SPDX/documentation-bundle/Helm-builder checks: 108 tests, 107 passed, 1 optional skip. Helm lint and strict Sphinx build passed. Upstream Tests and build checks passed on 4da5924811ea15a777a5d88974afacd97e6371f1.
  • Spark cutover: reviewed all 41 rendered Kubernetes resources; changed the seed, Gateway Deployment, and the UI's existing seed checksum dependency. Preserved all non-Helm Secret fingerprints, Agent/channel/model/tool configuration and other Pod templates. Migrated 4 legacy incident rows and 61 evidence files without resending the historical report. Stopped the old host worker.
  • Fresh run plugin-gpu-20261102t142359z: actual Prometheus evaluation (for=45s) and Alertmanager intake triggered one native Agent goal, 13 tool results, and 1 recorded argument-validation error. The Agent independently discovered knowledge collections, called IRAOP, inspected Kubernetes resources and queried metrics. The service did not collect evidence, ask Research, rewrite conclusions or send reports manually on the Agent's behalf.
  • Preserved the complete 4,656-byte model report and 6,595-byte publication, with no factual corrections or truncation. The native adapter returned 3 platform message IDs; readback confirmed all three messages in the intended private conversation. Feishu's interactive-card readback returns a client-upgrade placeholder rather than the rendered body, so complete client display and this new run's real human follow-up remain pending recipient acceptance; no synthetic question is counted as a pass.
  • Dedicated Gateway reconstruction preserved the incident, all 8 accumulated event rows, all 3 message IDs, report hashes and receiving-session context binding. The native status RPC is running; 7/7 Pods are ready. Application Secret fingerprints and the shared model PID/start time stayed unchanged.
  • Cleaned up this run's six isolated test objects and verified all pre-existing rule UIDs/specification hashes and all eight physical GPUs' ECC/remap counters unchanged. Production DCGM remained 0/0/0. The fixture cleanup timer's missing-import failure and the subsequent manual-cleanup connection timeout are preserved in the experiment record; neither is represented as a successful timer execution.

The GPU input was a private DCGM cache snapshot, exposed by a CPU-only replay Pod; it did not damage hardware or write production DCGM state. The deployment still admits the explicit test-label scope. This validates native background triggering, autonomous investigation and delivery; real physical fault diagnosis, broader production intake, remediation, and other native channels require their own acceptance. Earlier standalone-worker experiments and their real human DM follow-up remain historical evidence, separate from this new plugin run.

Closes #88.

…delivery

Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder JeffPengCoder changed the title feat: add persistent read-only alert investigation and native Feishu delivery feat: add Helm-managed native alert automation plugin Nov 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add persistent background alert investigation and native-channel report delivery

1 participant