diff --git a/docs/ingestion/index.md b/docs/ingestion/index.md index f5f2d6e..138c6b0 100644 --- a/docs/ingestion/index.md +++ b/docs/ingestion/index.md @@ -36,7 +36,9 @@ semantic-convention-compliant spans can be ingested, whatever produces them: package decorates agent, tool and workflow functions and emits spans to an OTLP/HTTP collector. - **[Insight module for agentic systems](https://github.com/outshift-open/insight-module-for-agentic-systems/)** - — telemetry from this module is supported out of the box. + — an OpenTelemetry plugin for OpenClaw-style systems, and the alternative to the + Observe SDK when the MAS is not an application you instrument with decorators. + Its telemetry is supported out of the box. - **Other emitters** — vendor SDKs, framework auto-instrumentation and custom emitters are all admissible, as are the boundary-level traces propagated in black-box mode. @@ -142,3 +144,4 @@ analysis composable. - [Insight module for agentic systems](https://github.com/outshift-open/insight-module-for-agentic-systems/) - [`ioc-project/scope`](https://github.com/ioc-project/scope) — SCOPE repo, under Apache 2.0 license - [Instrumenting a MAS: tutorial](../tutorials/instrument-your-app.md) +- [Instrumenting an OpenClaw-style MAS: tutorial](tutorials/instrument-openclaw.md) diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 3cb20c0..dcd69a3 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -13,6 +13,14 @@ traces and analysis ingestion. [Instrument an agent manually or with a coding harness](instrument-your-app.md){ .md-button .md-button--primary } +## Instrument an OpenClaw-style MAS + +The tutorial configures the Insight Module plugin for OpenClaw-style gateways, +enables provider-SDK auto-instrumentation and verifies the exported traces. It +also shows how to adapt the approach to other agent runtimes. + +[Instrument an OpenClaw-style MAS](instrument-openclaw.md){ .md-button .md-button--primary } + ## Deploy SCOPE The tutorial deploys the full stack, from the collector to the dashboard, with @@ -36,6 +44,7 @@ application, and follows it to the knowledge graph in Neo4j and to the dashboard | Walkthrough | Status | |---|---| | [Instrument your app](instrument-your-app.md) | Available | +| [Instrument an OpenClaw-style MAS](instrument-openclaw.md) | Available | | [Deploying SCOPE](deploy-scope.md) | Available | | [Your first trace](first-trace.md) | Available | | Exploring a session in the dashboard | In preparation | diff --git a/docs/tutorials/instrument-openclaw.md b/docs/tutorials/instrument-openclaw.md new file mode 100644 index 0000000..58bd2b8 --- /dev/null +++ b/docs/tutorials/instrument-openclaw.md @@ -0,0 +1,197 @@ +# SCOPE — Tutorial: Instrument an OpenClaw-style MAS + +

Instrumenting agent gateways and runtimes you configure rather than edit

+ +## Before starting + +Some multi-agent systems are not Python applications with agent functions you can +decorate. An OpenClaw-style system is a **gateway** that runs agents, tools and +model calls for you, and is extended through plugins. For these, use the +[Insight Module for Agentic Systems](https://github.com/outshift-open/insight-module-for-agentic-systems), +an OpenTelemetry plugin for OpenClaw. It is the alternative to the +[Observe SDK](instrument-your-app.md). + +As with the Observe SDK, SCOPE is **not** installed in the gateway. The plugin +emits OpenTelemetry to an OTLP/HTTP collector, and the +[ingestion pipeline](../ingestion.md#ingestion) reads the stored spans. + +```text +OpenClaw gateway + Insight Module plugin + -> OTLP/HTTP collector (port 4318) + -> telemetry store (ClickHouse in the reference deployment) + -> SCOPE ingestion and analysis +``` + +## What the module captures + +- Connected request, agent, tool and outbound-message traces. +- LLM usage, model, token and cost enrichment. +- Gateway diagnostics: queue, webhook, session and tool-loop signals. +- Optional auto-instrumentation of the Anthropic, OpenAI, Bedrock and Vertex AI + provider SDKs. + +A typical trace has this shape: + +```text +openclaw.request (root span) +├── openclaw.agent.turn +│ ├── llm. +│ ├── tool.Read / tool.exec / tool.Write / tool.web_search +└── openclaw.message.sent +``` + +## 1. Install and register the plugin + +The plugin is available on [ClawHub](https://clawhub.ai/outshift-open/plugins/insight-module). +Install it with the OpenClaw CLI (OpenClaw 2026.9.3 or later): + +```bash +openclaw plugins install @outshift-open/insight-module +``` + +OpenClaw resolves this scoped package from ClawHub. Configure it in +`~/.openclaw/openclaw.json`: + +```json +{ + "plugins": { + "entries": { + "insight-module": { + "enabled": true, + "hooks": { "allowConversationAccess": true }, + "config": { + "endpoint": "http://host.docker.internal:4318", + "serviceName": "openclaw-gateway", + "protocol": "http", + "traces": true, + "metrics": true, + "captureContent": true, + "emitIoaObserveAttributes": true, + "customAttributes": { + "workspace-id": "", + "mas-id": "" + } + } + } + } + } +} +``` + +| Key | Purpose | +|---|---| +| `endpoint` | OTLP/HTTP receiver. `host.docker.internal:4318` on macOS/Windows, `172.17.0.1:4318` on Linux Docker | +| `serviceName` | Service name on every span; use it to filter in the store | +| `traces` / `metrics` | Enable each signal | +| `captureContent` | Capture payloads on agent, request, LLM, tool and message spans | +| `emitIoaObserveAttributes` | Emit `ioa_observe.*` attributes (fork/join topology, handoff sequence, entity payloads). `false` keeps only OTel GenAI fields | +| `customAttributes` | Tags added to the root `openclaw.request` span only, such as `workspace-id` and `mas-id` | +| `hooks.allowConversationAccess` | Must be `true` for conversation-level capture | + +!!! warning "Content capture" + `captureContent` records prompts, responses and tool arguments and results. + Disable it, or check the store's access and retention policy, before pointing + a production gateway at a shared collector. + +Restart the gateway: + +```bash +openclaw gateway restart +``` + +## 2. Start a collector and store + +The module's repository ships a ClickHouse and OpenTelemetry Collector stack: + +```bash +cd observability-plugin/deploy +docker-compose up -d +``` + +Any OTLP/HTTP collector that forwards to a store SCOPE can read also works. See +[Instrumentation and ingestion](../ingestion/index.md) + +## 3. Optional: auto-instrument provider SDKs + +To get LLM spans from the provider SDKs, preload the instrumentation **before** +the gateway starts, using an absolute path to the installed plugin's +`instrumentation/preload.mjs` file. For a ClawHub installation, the package is +under OpenClaw's managed plugin project in `~/.openclaw/npm/projects/`: + +```bash +export NODE_OPTIONS="--import /absolute/path/to/managed-plugin-project/node_modules/@outshift-open/insight-module/instrumentation/preload.mjs" +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 +export OTEL_SERVICE_NAME=openclaw-gateway +openclaw gateway start +``` + +The log should show +`[insight-module] ✅ GenAI instrumentation active via NODE_OPTIONS preload`. For a +systemd-managed gateway, put the same variables in a user override at +`~/.config/systemd/user/openclaw-gateway.service.d/override.conf` and run +`systemctl --user daemon-reload`. + +## 4. Verify the traces + +Run one request through the gateway, then query the store. In the reference +ClickHouse schema: + +```sql +SELECT Timestamp, TraceId, SpanId, ParentSpanId, SpanName, StatusCode +FROM otel_traces +WHERE ServiceName = 'openclaw-gateway' + AND Timestamp > now() - INTERVAL 1 HOUR +ORDER BY Timestamp; +``` + +Confirm that: + +- One `openclaw.request` root span exists per request, with `workspace-id` and + `mas-id` on it. +- `openclaw.agent.turn`, `tool.*` and LLM spans share the root's `TraceId` and have + correct parents. +- Tool failures show an error status on the failing span. +- Content attributes (`gen_ai.input.messages`, `gen_ai.tool.call.arguments`) are + present or absent as `captureContent` dictates. + +Then locate the session in SCOPE's API or dashboard. As with the Observe SDK, +spans in the store do not by themselves start analysis; the deployment's +[ingestion mode](../ingestion/index.md#ingestion) and session-completion notification +must be configured. + +!!! note "Not yet verified here" + Which span attributes carry the session identity that SCOPE correlates on, and + how the module signals session completion, are not documented in the module's + README. Check `session.start`, `session.end` and `openclaw.runtime.session.*` + spans in your store against your ingestion configuration before relying on + completion-driven analysis. + +## Other OpenClaw-like systems + +For a gateway or runtime that is not OpenClaw, the module itself does not apply, +but the contract does. SCOPE needs OpenTelemetry spans that show: + +1. **One trace per request**, with a root span and a stable session identity. +2. **Agent turns** as spans, parented under the request. +3. **Tool and LLM calls** as child spans, with model, token and tool-argument + attributes following the OTel GenAI conventions. +4. **Handoffs between agents** with context propagated across the boundary. + +Options, from least to most effort: + +| Situation | Approach | +|---|---| +| The runtime has a plugin or hook API | Write a plugin like the Insight Module, which is the pattern to copy | +| The runtime is Python and you can edit it | Use the [Observe SDK](instrument-your-app.md) | +| You can only wrap the process | Use OTel auto-instrumentation for the provider SDKs; expect black-box fidelity | + +Fields that are not convention-compliant are inferred on a best-effort basis, so +analysis coverage is wider where instrumentation is richer. See +[Fidelity tracks instrumentation richness](../ingestion/index.md#instrumentation). + +## Related + +- [Insight Module for Agentic Systems](https://github.com/outshift-open/insight-module-for-agentic-systems) — source, plugin README and an SRE-triage sample MAS +- [Instrument your app (Observe SDK)](instrument-your-app.md) +- [Instrumentation and ingestion](../ingestion.md) +- [Tutorials and Demos](index.md) \ No newline at end of file diff --git a/docs/tutorials/instrument-your-app.md b/docs/tutorials/instrument-your-app.md index 630400b..0c459f8 100644 --- a/docs/tutorials/instrument-your-app.md +++ b/docs/tutorials/instrument-your-app.md @@ -9,39 +9,53 @@ SCOPE analyses OpenTelemetry traces from a multi-agent system. It is the need to be installed alongside them. The SDK sends telemetry to an OTLP/HTTP collector, which stores the spans for SCOPE to process. -## Instrumenting a multi-agent system +## Choose an instrumentation -The reference instrumentation is the [AGNTCY Observe SDK](https://github.com/agntcy/observe). -It supports Python agent applications, including LangGraph, LlamaIndex and custom -orchestrators, with instrumentation for A2A, SLIM and MCP interactions. +Two SDKs are supported. Both emit OpenTelemetry to an OTLP/HTTP collector, so the +rest of the path is identical. -### Observe SDK and the Insight Module - -Instrumentation and analysis are separate responsibilities. The **Observe SDK** -runs inside the agent application and emits OpenTelemetry data. The **Insight -Module** is on the analysis side: exporting spans is the input to analysis, not -proof that an execution has been ingested or an insight produced. This site's -[SCOPE overview](../index.md#how-scope-works) describes the session reconstruction, -knowledge graph and analysis pipeline, with findings exposed by its API and UI. -Do not install the analysis platform in each agent to enable instrumentation. +| | **AGNTCY Observe SDK** | **Insight Module for Agentic Systems** | +|---|---|---| +| Best for | Python agent apps you own and can edit (LangGraph, LlamaIndex, custom orchestrators) | OpenClaw-style gateways and agent runtimes, where you configure a plugin rather than edit agent code | +| Language | Python | Node.js / TypeScript (OpenClaw plugin) | +| How it hooks in | Decorators (`@graph`, `@agent`, `@task`, `@tool`) and `session_start()` | OpenClaw plugin config, plus an optional `NODE_OPTIONS` preload for provider SDKs | +| Protocol coverage | A2A, SLIM and MCP instrumentors | Request, agent-turn, tool, LLM and outbound-message spans; Anthropic, OpenAI, Bedrock and Vertex AI SDKs | +| Tutorial | This page | [Instrument an OpenClaw-style MAS](instrument-openclaw.md) | + +The Observe SDK is the reference instrumentation. The +[Insight Module](https://github.com/outshift-open/insight-module-for-agentic-systems) +is the alternative when the MAS is an OpenClaw-style system. The module can also +emit `ioa_observe.*` attributes, so its spans carry the same topology fields as +Observe SDK spans. + +Instrumentation and analysis are separate responsibilities. Both SDKs run inside +(or beside) the agent application and emit OpenTelemetry data. Exporting spans is +the input to analysis, not proof that an execution has been ingested or an insight +produced. This site's [SCOPE overview](../index.md#how-scope-works) describes the +session reconstruction, knowledge graph and analysis pipeline. Do not install the +analysis platform in each agent to enable instrumentation. The reference data path is: ```text -Agent application + Observe SDK +Agent application + Observe SDK (or Insight Module) -> OTLP/HTTP collector -> telemetry store (ClickHouse in the reference deployment) -> SCOPE ingestion and analysis -> knowledge graph, insights, API and dashboard ``` -Use the [SDK setup guide](https://github.com/agntcy/observe/blob/main/GETTING-STARTED.md) -for application instrumentation and the [SCOPE repository](https://github.com/ioc-project/scope) -for the analysis deployment. Other OpenTelemetry emitters are admissible through -the [ingestion pipeline](../ingestion/index.md#ingestion); the SDK is the reference, -not a prerequisite for adopting SCOPE. +For the Observe SDK, use the +[SDK setup guide](https://github.com/agntcy/observe/blob/main/GETTING-STARTED.md). +For the analysis deployment, use the +[SCOPE repository](https://github.com/ioc-project/ioc-scope). Other OpenTelemetry +emitters are admissible through the [ingestion pipeline](../ingestion.md#ingestion); +neither SDK is a prerequisite for adopting IoC. + +The sections below cover the **Observe SDK**. For OpenClaw, go to +[Instrument an OpenClaw-style MAS](instrument-openclaw.md). -## Manual instrumentation +## Manual instrumentation (Observe SDK) ### 1. Prepare the collector and application @@ -277,6 +291,8 @@ Evaluate both unchanged application behaviour and the expected span relationship ## Related +- [Instrument an OpenClaw-style MAS](instrument-openclaw.md) — the Insight Module alternative - [AGNTCY Observe SDK](https://github.com/agntcy/observe) - SDK source, installation and local collector stack +- [Instrumentation and ingestion](../ingestion/index.md) — overview of SCOPE's ingestion pipeline and modes - [SDK integration guide](https://github.com/agntcy/observe/blob/main/GETTING-STARTED.md) - framework and transport examples - [Tutorials and Demos](index.md) — the other SCOPE walkthroughs diff --git a/mkdocs.yml b/mkdocs.yml index 3d53525..5928d19 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -138,5 +138,6 @@ nav: - Getting Started: - tutorials/index.md - Instrument your app: tutorials/instrument-your-app.md + - Instrument OpenClaw-style MAS: tutorials/instrument-openclaw.md - Deploy SCOPE: tutorials/deploy-scope.md - Your first trace: tutorials/first-trace.md