Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/ingestion/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ semantic-convention-compliant spans can be ingested, whatever produces them:
package decorates agent, tool and workflow functions and emits spans to an
OTLP/HTTP collector.
- **[Insight module for agentic systems](https://github.com/outshift-open/insight-module-for-agentic-systems/)**
— telemetry from this module is supported out of the box.
— an OpenTelemetry plugin for OpenClaw-style systems, and the alternative to the
Observe SDK when the MAS is not an application you instrument with decorators.
Its telemetry is supported out of the box.
- **Other emitters** — vendor SDKs, framework auto-instrumentation and custom
emitters are all admissible, as are the boundary-level traces propagated in
black-box mode.
Expand Down Expand Up @@ -142,3 +144,4 @@ analysis composable.
- [Insight module for agentic systems](https://github.com/outshift-open/insight-module-for-agentic-systems/)
- [`ioc-project/scope`](https://github.com/ioc-project/scope) — SCOPE repo, under Apache 2.0 license
- [Instrumenting a MAS: tutorial](../tutorials/instrument-your-app.md)
- [Instrumenting an OpenClaw-style MAS: tutorial](tutorials/instrument-openclaw.md)
9 changes: 9 additions & 0 deletions docs/tutorials/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,14 @@ traces and analysis ingestion.
[Instrument an agent manually or with a coding harness](instrument-your-app.md){ .md-button .md-button--primary }
</div>

## Instrument an OpenClaw-style MAS

The tutorial configures the Insight Module plugin for OpenClaw-style gateways,
enables provider-SDK auto-instrumentation and verifies the exported traces. It
also shows how to adapt the approach to other agent runtimes.

[Instrument an OpenClaw-style MAS](instrument-openclaw.md){ .md-button .md-button--primary }

## Deploy SCOPE

The tutorial deploys the full stack, from the collector to the dashboard, with
Expand All @@ -36,6 +44,7 @@ application, and follows it to the knowledge graph in Neo4j and to the dashboard
| Walkthrough | Status |
|---|---|
| [Instrument your app](instrument-your-app.md) | Available |
| [Instrument an OpenClaw-style MAS](instrument-openclaw.md) | Available |
| [Deploying SCOPE](deploy-scope.md) | Available |
| [Your first trace](first-trace.md) | Available |
| Exploring a session in the dashboard | In preparation |
Expand Down
197 changes: 197 additions & 0 deletions docs/tutorials/instrument-openclaw.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
# SCOPE — Tutorial: Instrument an OpenClaw-style MAS

<p class="eyebrow">Instrumenting agent gateways and runtimes you configure rather than edit</p>

## Before starting

Some multi-agent systems are not Python applications with agent functions you can
decorate. An OpenClaw-style system is a **gateway** that runs agents, tools and
model calls for you, and is extended through plugins. For these, use the
[Insight Module for Agentic Systems](https://github.com/outshift-open/insight-module-for-agentic-systems),
an OpenTelemetry plugin for OpenClaw. It is the alternative to the
[Observe SDK](instrument-your-app.md).

As with the Observe SDK, SCOPE is **not** installed in the gateway. The plugin
emits OpenTelemetry to an OTLP/HTTP collector, and the
[ingestion pipeline](../ingestion.md#ingestion) reads the stored spans.

```text
OpenClaw gateway + Insight Module plugin
-> OTLP/HTTP collector (port 4318)
-> telemetry store (ClickHouse in the reference deployment)
-> SCOPE ingestion and analysis
```

## What the module captures

- Connected request, agent, tool and outbound-message traces.
- LLM usage, model, token and cost enrichment.
- Gateway diagnostics: queue, webhook, session and tool-loop signals.
- Optional auto-instrumentation of the Anthropic, OpenAI, Bedrock and Vertex AI
provider SDKs.

A typical trace has this shape:

```text
openclaw.request (root span)
├── openclaw.agent.turn
│ ├── llm.<model>
│ ├── tool.Read / tool.exec / tool.Write / tool.web_search
└── openclaw.message.sent
```

## 1. Install and register the plugin

The plugin is available on [ClawHub](https://clawhub.ai/outshift-open/plugins/insight-module).
Install it with the OpenClaw CLI (OpenClaw 2026.9.3 or later):

```bash
openclaw plugins install @outshift-open/insight-module
```

OpenClaw resolves this scoped package from ClawHub. Configure it in
`~/.openclaw/openclaw.json`:

```json
{
"plugins": {
"entries": {
"insight-module": {
"enabled": true,
"hooks": { "allowConversationAccess": true },
"config": {
"endpoint": "http://host.docker.internal:4318",
"serviceName": "openclaw-gateway",
"protocol": "http",
"traces": true,
"metrics": true,
"captureContent": true,
"emitIoaObserveAttributes": true,
"customAttributes": {
"workspace-id": "<workspace-uuid>",
"mas-id": "<mas-uuid>"
}
}
}
}
}
}
```

| Key | Purpose |
|---|---|
| `endpoint` | OTLP/HTTP receiver. `host.docker.internal:4318` on macOS/Windows, `172.17.0.1:4318` on Linux Docker |
| `serviceName` | Service name on every span; use it to filter in the store |
| `traces` / `metrics` | Enable each signal |
| `captureContent` | Capture payloads on agent, request, LLM, tool and message spans |
| `emitIoaObserveAttributes` | Emit `ioa_observe.*` attributes (fork/join topology, handoff sequence, entity payloads). `false` keeps only OTel GenAI fields |
| `customAttributes` | Tags added to the root `openclaw.request` span only, such as `workspace-id` and `mas-id` |
| `hooks.allowConversationAccess` | Must be `true` for conversation-level capture |

!!! warning "Content capture"
`captureContent` records prompts, responses and tool arguments and results.
Disable it, or check the store's access and retention policy, before pointing
a production gateway at a shared collector.

Restart the gateway:

```bash
openclaw gateway restart
```

## 2. Start a collector and store

The module's repository ships a ClickHouse and OpenTelemetry Collector stack:

```bash
cd observability-plugin/deploy
docker-compose up -d
```

Any OTLP/HTTP collector that forwards to a store SCOPE can read also works. See
[Instrumentation and ingestion](../ingestion/index.md)

## 3. Optional: auto-instrument provider SDKs

To get LLM spans from the provider SDKs, preload the instrumentation **before**
the gateway starts, using an absolute path to the installed plugin's
`instrumentation/preload.mjs` file. For a ClawHub installation, the package is
under OpenClaw's managed plugin project in `~/.openclaw/npm/projects/`:

```bash
export NODE_OPTIONS="--import /absolute/path/to/managed-plugin-project/node_modules/@outshift-open/insight-module/instrumentation/preload.mjs"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=openclaw-gateway
openclaw gateway start
```

The log should show
`[insight-module] ✅ GenAI instrumentation active via NODE_OPTIONS preload`. For a
systemd-managed gateway, put the same variables in a user override at
`~/.config/systemd/user/openclaw-gateway.service.d/override.conf` and run
`systemctl --user daemon-reload`.

## 4. Verify the traces

Run one request through the gateway, then query the store. In the reference
ClickHouse schema:

```sql
SELECT Timestamp, TraceId, SpanId, ParentSpanId, SpanName, StatusCode
FROM otel_traces
WHERE ServiceName = 'openclaw-gateway'
AND Timestamp > now() - INTERVAL 1 HOUR
ORDER BY Timestamp;
```

Confirm that:

- One `openclaw.request` root span exists per request, with `workspace-id` and
`mas-id` on it.
- `openclaw.agent.turn`, `tool.*` and LLM spans share the root's `TraceId` and have
correct parents.
- Tool failures show an error status on the failing span.
- Content attributes (`gen_ai.input.messages`, `gen_ai.tool.call.arguments`) are
present or absent as `captureContent` dictates.

Then locate the session in SCOPE's API or dashboard. As with the Observe SDK,
spans in the store do not by themselves start analysis; the deployment's
[ingestion mode](../ingestion/index.md#ingestion) and session-completion notification
must be configured.

!!! note "Not yet verified here"
Which span attributes carry the session identity that SCOPE correlates on, and
how the module signals session completion, are not documented in the module's
README. Check `session.start`, `session.end` and `openclaw.runtime.session.*`
spans in your store against your ingestion configuration before relying on
completion-driven analysis.

## Other OpenClaw-like systems

For a gateway or runtime that is not OpenClaw, the module itself does not apply,
but the contract does. SCOPE needs OpenTelemetry spans that show:

1. **One trace per request**, with a root span and a stable session identity.
2. **Agent turns** as spans, parented under the request.
3. **Tool and LLM calls** as child spans, with model, token and tool-argument
attributes following the OTel GenAI conventions.
4. **Handoffs between agents** with context propagated across the boundary.

Options, from least to most effort:

| Situation | Approach |
|---|---|
| The runtime has a plugin or hook API | Write a plugin like the Insight Module, which is the pattern to copy |
| The runtime is Python and you can edit it | Use the [Observe SDK](instrument-your-app.md) |
| You can only wrap the process | Use OTel auto-instrumentation for the provider SDKs; expect black-box fidelity |

Fields that are not convention-compliant are inferred on a best-effort basis, so
analysis coverage is wider where instrumentation is richer. See
[Fidelity tracks instrumentation richness](../ingestion/index.md#instrumentation).

## Related

- [Insight Module for Agentic Systems](https://github.com/outshift-open/insight-module-for-agentic-systems) — source, plugin README and an SRE-triage sample MAS
- [Instrument your app (Observe SDK)](instrument-your-app.md)
- [Instrumentation and ingestion](../ingestion.md)
- [Tutorials and Demos](index.md)
56 changes: 36 additions & 20 deletions docs/tutorials/instrument-your-app.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,39 +9,53 @@ SCOPE analyses OpenTelemetry traces from a multi-agent system. It is the
need to be installed alongside them. The SDK sends telemetry to an OTLP/HTTP
collector, which stores the spans for SCOPE to process.

## Instrumenting a multi-agent system
## Choose an instrumentation

The reference instrumentation is the [AGNTCY Observe SDK](https://github.com/agntcy/observe).
It supports Python agent applications, including LangGraph, LlamaIndex and custom
orchestrators, with instrumentation for A2A, SLIM and MCP interactions.
Two SDKs are supported. Both emit OpenTelemetry to an OTLP/HTTP collector, so the
rest of the path is identical.

### Observe SDK and the Insight Module

Instrumentation and analysis are separate responsibilities. The **Observe SDK**
runs inside the agent application and emits OpenTelemetry data. The **Insight
Module** is on the analysis side: exporting spans is the input to analysis, not
proof that an execution has been ingested or an insight produced. This site's
[SCOPE overview](../index.md#how-scope-works) describes the session reconstruction,
knowledge graph and analysis pipeline, with findings exposed by its API and UI.
Do not install the analysis platform in each agent to enable instrumentation.
| | **AGNTCY Observe SDK** | **Insight Module for Agentic Systems** |
|---|---|---|
| Best for | Python agent apps you own and can edit (LangGraph, LlamaIndex, custom orchestrators) | OpenClaw-style gateways and agent runtimes, where you configure a plugin rather than edit agent code |
| Language | Python | Node.js / TypeScript (OpenClaw plugin) |
| How it hooks in | Decorators (`@graph`, `@agent`, `@task`, `@tool`) and `session_start()` | OpenClaw plugin config, plus an optional `NODE_OPTIONS` preload for provider SDKs |
| Protocol coverage | A2A, SLIM and MCP instrumentors | Request, agent-turn, tool, LLM and outbound-message spans; Anthropic, OpenAI, Bedrock and Vertex AI SDKs |
| Tutorial | This page | [Instrument an OpenClaw-style MAS](instrument-openclaw.md) |

The Observe SDK is the reference instrumentation. The
[Insight Module](https://github.com/outshift-open/insight-module-for-agentic-systems)
is the alternative when the MAS is an OpenClaw-style system. The module can also
emit `ioa_observe.*` attributes, so its spans carry the same topology fields as
Observe SDK spans.

Instrumentation and analysis are separate responsibilities. Both SDKs run inside
(or beside) the agent application and emit OpenTelemetry data. Exporting spans is
the input to analysis, not proof that an execution has been ingested or an insight
produced. This site's [SCOPE overview](../index.md#how-scope-works) describes the
session reconstruction, knowledge graph and analysis pipeline. Do not install the
analysis platform in each agent to enable instrumentation.

The reference data path is:

```text
Agent application + Observe SDK
Agent application + Observe SDK (or Insight Module)
-> OTLP/HTTP collector
-> telemetry store (ClickHouse in the reference deployment)
-> SCOPE ingestion and analysis
-> knowledge graph, insights, API and dashboard
```

Use the [SDK setup guide](https://github.com/agntcy/observe/blob/main/GETTING-STARTED.md)
for application instrumentation and the [SCOPE repository](https://github.com/ioc-project/scope)
for the analysis deployment. Other OpenTelemetry emitters are admissible through
the [ingestion pipeline](../ingestion/index.md#ingestion); the SDK is the reference,
not a prerequisite for adopting SCOPE.
For the Observe SDK, use the
[SDK setup guide](https://github.com/agntcy/observe/blob/main/GETTING-STARTED.md).
For the analysis deployment, use the
[SCOPE repository](https://github.com/ioc-project/ioc-scope). Other OpenTelemetry
emitters are admissible through the [ingestion pipeline](../ingestion.md#ingestion);
neither SDK is a prerequisite for adopting IoC.

The sections below cover the **Observe SDK**. For OpenClaw, go to
[Instrument an OpenClaw-style MAS](instrument-openclaw.md).

## Manual instrumentation
## Manual instrumentation (Observe SDK)

### 1. Prepare the collector and application

Expand Down Expand Up @@ -277,6 +291,8 @@ Evaluate both unchanged application behaviour and the expected span relationship

## Related

- [Instrument an OpenClaw-style MAS](instrument-openclaw.md) — the Insight Module alternative
- [AGNTCY Observe SDK](https://github.com/agntcy/observe) - SDK source, installation and local collector stack
- [Instrumentation and ingestion](../ingestion/index.md) — overview of SCOPE's ingestion pipeline and modes
- [SDK integration guide](https://github.com/agntcy/observe/blob/main/GETTING-STARTED.md) - framework and transport examples
- [Tutorials and Demos](index.md) — the other SCOPE walkthroughs
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -138,5 +138,6 @@ nav:
- Getting Started:
- tutorials/index.md
- Instrument your app: tutorials/instrument-your-app.md
- Instrument OpenClaw-style MAS: tutorials/instrument-openclaw.md
- Deploy SCOPE: tutorials/deploy-scope.md
- Your first trace: tutorials/first-trace.md
Loading