Skip to content

docs: add Spark alert operations and verified background GPU-to-native-Feishu workflow - #1

Open
JeffPengCoder wants to merge 18 commits into
mainfrom
docs/spark-alert-feishu-runbook
Open

JeffPengCoder wants to merge 18 commits into
mainfrom
docs/spark-alert-feishu-runbook

Conversation

@JeffPengCoder

@JeffPengCoder JeffPengCoder commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

This provides the complete operator handoff for the Spark deployment: access and SSH forwarding, alert polling and injection, actual diagnostic evidence, IRAOP ingestion/retrieval, isolated View/Edit/Auto validation, both Feishu delivery modes, and separately dated CPU/GPU experiments.

The 2026-11-02 update records a new browser-independent background GPU telemetry test through the native Feishu application bot. The persistent host service automatically imported a real Alertmanager alert, collected five actual diagnostic tool results, called IRAOP with two document/page sources, generated the complete report in the same View session, and delivered 11,966 bytes through Jeff's AI Assistant. Platform readback succeeded and the user confirmed the complete matching Run ID and incident. The functional service implementation is separate in upstream #89; this PR contains documentation, reproduction examples and evidence snapshots.

Updated deliverables:

  • docs/spark_background_gpu_alert_native_feishu_20261102.zh-CN.md: complete new run, operational commands, responsibility/credential handoff, exact fixture UIDs, all hardware-control fields, all three attempts, every state transition, actual tools/research, the complete uncorrected model report, delivery and cleanup.
  • docs/images/mosaic-gpu-feishu-background-20261102.svg and .png: current persistent-worker/IRAOP/native-channel architecture, with the historical Webhook route explicitly retained.
  • docs/artifacts/spark-gpu-validation-20261102-external.pdf: 39 pages, including the new run and the full earlier GPU experiment; 60 tables and 541 data rows.
  • docs/artifacts/mosaic-feishu-webhook-native-20261102-external.pdf: complete 20-page dual-mode Feishu handoff; 16 tables and 149 data rows, including application creation, permissions, secrets/allowlists, actual deployment and the new acceptance cases.
  • docs/artifacts/mosaic-gpu-feishu-background-20261102.pdf: standalone architecture.
  • Existing alert, CPU, GPU and Feishu runbooks now distinguish October 31's Webhook experiments, November 1's native report replay, and November 2's new automated run. The documentation index and packaged agent documentation are synchronized.

Validation and evidence:

  • Worker restart while the same alert remained firing preserved the incident/message IDs, every event row and every evidence hash; there was no second investigation or send.
  • Recreated the OpenClaw gateway Pod; real DM and group mention received model-generated replies afterward. Nine cases ran against the installed official Feishu policy exports, with complete gate/allowlist results. A real second identity was unavailable; it remains untested. The no-mention observation and missing ordinary group-read scope are stated explicitly.
  • Both initial failures are retained: premature headless completion and a valid IRAOP source_file field rejected by the first client parser. All exact source versions, requests/responses, original reports, platform messages, full observation samples and cleanup commands are available through the evidence manifest.
  • Six test objects were deleted with UID-preconditioned Kubernetes API requests. Source alerts and worker-owned UI cards cleared naturally. Existing monitoring rules, all application Secret fingerprints, Helm revision 18, seven ready Pods and the shared model process were preserved; the root connection was closed.
  • All 436 historical artifacts, including the old PDF, retained their hashes. All 64 pre-existing tables and 93 command/code blocks in the four updated documents are preserved. Reports and error outputs are losslessly attached; addresses/accounts/device and application identities use consistent substitutions, and credential values are excluded.
  • Every page of both PDFs was rendered and visually inspected. Table parsing, source preservation, relative links, evidence hashes, private-value scans, glyphs and page bounds passed. Current documentation Node checks: 213 passed, two existing optional skips; strict Sphinx, documentation package check and git diff --check passed. Upstream feat: add Helm-managed native alert automation plugin NVIDIA/AI-Factory-Operations-Agent#89's separate Tests/build checks also passed.

The live background service currently selects only the two explicit validation labels, not every production alert. The GPU input is an isolated DCGM cache simulation with unchanged real hardware counters. Workflow/delivery acceptance is distinct from physical GPU diagnosis or production repair.

Automatic Feishu DM context and real follow-up verification

The new complete dated guide and architecture cover the proactive-report context gap: a transcript-only delivery mirror did not enter the model’s replay context. The persistent worker now transfers the unchanged full report into the configured receiving Agent/session before native publication. The runtime implementation is maintained in NVIDIA AFOA PR #89, code commit 63aa0d8, with the successful real follow-up recorded by e821ef5.

  • Complete Chinese DM context guide and updated architecture.
  • Fresh run handoff-gpu-20261102t085400z: five real diagnostic results, actual IRAOP with both page sources, original 12,186-byte report and complete 14,675-byte publication, real receiving input and native p2p receipt. No operator investigation, Research, runtime context insertion, report send or factual rewrite.
  • The actual recipient sent an ordinary, unquoted DM without copying the report or Run ID. Qwen summarized the report’s isolated injection and hardware comparison, and the native reply was recorded in the same p2p chat. Full question, answer and platform/native IDs are retained.
  • All 372 selected evidence files and 150 lossless gzip files verified; six Markdown tables, complete original model-report fence and every new relative link verified. Strict Sphinx and the documentation bundle checks passed; architecture rendered and visually checked. Original failed incidents, every ledger row and complete raw commands/results remain.
  • Worker restart preserved every event/evidence hash and message ID; exact-UID cleanup removed six fixtures, natural source/UI cleanup completed, and the eight physical GPUs, existing monitoring, seven Pods, Helm revision18, Secret fingerprints and shared model process stayed unchanged.

CBU China documentation is mirrored under aiops/mosaic/ through GitLab follow-up MR !2; original MR !1 was already merged. Runtime code is reviewed in AFOA, while the internal mirror carries documentation and versioned validation evidence. Production-label expansion, physical GPU diagnosis accuracy and a second real unauthorized identity are separate scopes.

JeffPengCoder and others added 3 commits October 30, 2026 14:27
…lans

Signed-off-by: Jeff Peng <239749643+JeffPengCoder@users.noreply.github.com>
Signed-off-by: Jeff Peng <239749643+JeffPengCoder@users.noreply.github.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder JeffPengCoder changed the title docs: add Spark alert handover and Feishu/research/repair runbooks docs: add Spark operations handover with live validation evidence Oct 31, 2026
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder JeffPengCoder changed the title docs: add Spark operations handover with live validation evidence docs: add verified Spark alert, research and repair handoff Oct 31, 2026
…etup

Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder JeffPengCoder changed the title docs: add verified Spark alert, research and repair handoff docs: add complete Spark alert, research, repair and Feishu handoff Oct 31, 2026
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder JeffPengCoder changed the title docs: add complete Spark alert, research, repair and Feishu handoff docs: add complete Spark handoff and GPU validation limits Oct 31, 2026
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
…idation

Signed-off-by: Jeff Peng <jepeng@nvidia.com>
@JeffPengCoder JeffPengCoder changed the title docs: add complete Spark handoff and GPU validation limits docs: add Spark alert operations and verified background GPU-to-native-Feishu workflow Nov 2, 2026
… evidence

Signed-off-by: Jeff Peng <jepeng@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant