Performance fixes + updated docs

README.md

@@ -1,39 +1,52 @@
 # System for Cognitive Observability Provenance and Explainability (SCOPE)
 
-SCOPE is a modular platform for observing, analyzing, and explaining the executions of Multi-Agent Systems (MAS) from OpenTelemetry traces. It ingests raw OTel spans, maps them onto a shared MAS ontology, and enriches each session with analysis such as semantic grouping, anomaly detection, consistency checking, and stateful trajectory evaluation — surfaced through a web dashboard.
+SCOPE observes, analyzes and explains the executions of Multi-Agent Systems (MAS) from their OpenTelemetry traces. It maps raw spans onto a shared MAS ontology, keeps every execution as a knowledge graph, and enriches each session with analyses such as semantic grouping, anomaly detection, consistency checking, cognitive-failure detection and stateful trajectory evaluation. A web dashboard and an API serve the result.
 
-## How it works
+![A session in the SCOPE dashboard: its spans and the execution timeline of the agents and tools](docs/assets/dashboard.png)
 
-1. A MAS sends OTel spans to an OTel collector, which stores them (ClickHouse by default, swappable for another backend).
-2. Once a trace is complete, the SCOPE ingestion pipeline maps the raw spans onto the [MAS ontology](ontology), producing a structured execution model.
-3. The session representation is enriched by the analysis workers (semantic grouping, anomaly detection, consistency checking, stateful evaluation, metrics).
-4. The enriched knowledge graph is served through the [API](backend/api) and visualized in the [UI](ui), giving explainability and insight into the MAS's inner workings.
+## Quick start
 
-## Repository structure
+Run the whole stack on your machine with Docker Compose, send it the trace of a small two-agent example, and look at it in the dashboard. It takes about 15 minutes, most of it the first image build.
 
-| Path | Description |
-| --- | --- |
-| [`backend/`](backend) | Python libraries and workers implementing ingestion, normalization, analysis, and the data API. See [backend/README.md](backend/README.md). |
-| [`backend/api`](backend/api) | Unified DB access layer and REST API over the raw OTel spans and the knowledge graph. |
-| [`backend/norm`](backend/norm) | Normalizes raw OTel traces into structured MAS execution models. |
-| [`backend/mce`](backend/mce) | Metric computation engine for sessions. |
-| [`backend/stateful-evals`](backend/stateful-evals) | Cost-effective evaluation of a session's trajectory. |
-| [`backend/workers`](backend/workers) | Standalone workers built on top of the backend libraries. |
-| [`ui/`](ui) | React/TypeScript web dashboard for browsing applications, sessions, execution graphs, and live topology. See [ui/README.md](docs/platform/ui/index.md). |
-| [`ontology/`](ontology) | The MAS telemetry ontology (Turtle/OWL) and the `scope-ontology` Python package. See [ontology/README.md](ontology/README.md). |
-| [`docs/`](docs) | Documentation site sources (MkDocs), built with `mkdocs.yml`. |
-| [`deploy/`](deploy) | Deployment: Helm charts for Kubernetes (`deploy/k8s`) and a Docker Compose stack (`deploy/docker-compose`) running the API, UI, workers and supporting infra. |
+You need Docker (Compose v2.17 or later, with at least **12 GB** of memory given to it) and [`uv`](https://docs.astral.sh/uv/) to run the example. An OpenAI-compatible LLM endpoint is only needed for the analyses that call a model: without it, traces are still collected and stored in the knowledge graph.
+
+### 1. Start SCOPE
+
+```bash
+git clone https://github.com/ioc-project/scope.git
+cd scope/deploy/docker-compose
+cp .env.template .env        # set LLM_MODEL_NAME, EMBEDDING_MODEL_NAME, LLM_GATEWAY_* and the change-me passwords
+docker compose up -d --build
+```
+
+### 2. Send a trace**
+
+```bash
+cd ../../examples/currency-demo
+OTLP_HTTP_ENDPOINT=http://localhost:4318 uv run main.py 100
+```
+
+```text
+100.00 USD = 85.90 EUR (example rate)
+```
 
-## Getting started
+### 3. Look at it
 
-Each component has its own setup instructions:
+| What | Where |
+|---|---|
+| Dashboard and API: *Applications* → `currency-demo` → *Inspect* | <http://localhost:8080> |
+| The knowledge graph, in the Neo4j browser (`neo4j` and your `NEO4J_PASSWORD`) | <http://localhost:7474> |
 
-- Backend (API, workers, local Docker Compose stack): [backend/README.md](backend/README.md)
-- UI (dashboard): [ui/README.md](docs/platform/ui/index.md)
-- Ontology (Python package, docs, versioning): [ontology/README.md](ontology/README.md)
-- Kubernetes deployment: [deploy/k8s/charts/scope/README.md](deploy/k8s/charts/scope/README.md)
+The session shows up a few seconds after the example exits. The [first-trace tutorial](docs/tutorials/first-trace.md) walks through each step with screenshots, and explains how to query the knowledge graph.
 
-For a fully local stack, see the [Docker Compose deployment](deploy/docker-compose/README.md), which runs the same components as the Helm chart: OTel collector, ClickHouse, RabbitMQ, Neo4j, the ingestion pipeline, the API and the UI.
+**Tear down** with `docker compose down` (add `-v` to also delete the data). To run SCOPE on Kubernetes instead, use the [Helm chart](deploy/k8s/charts/scope/README.md). To trace your own application, see [instrument your app](docs/tutorials/instrument-your-app.md); to see how SCOPE evaluates sessions against their peers, follow the [semantic analysis tutorial](docs/tutorials/semantic-analysis.md).
+
+## How it works
+
+1. A MAS sends OTel spans to an OTel collector, which stores them (ClickHouse by default, swappable for another backend).
+2. Once a trace is complete, the SCOPE ingestion pipeline maps the raw spans onto the [MAS ontology](ontology), producing a structured execution model: the topology of the system, and for each session its calls, states and transitions.
+3. The session is enriched by the analysis workers (semantic grouping, anomaly detection, consistency checking, cognitive failures, stateful evaluation, metrics).
+4. The enriched knowledge graph is served through the [API](backend/api) and visualized in the [UI](ui), giving explainability and insight into the MAS's inner workings.
 
 ## Documentation
 
@@ -43,13 +56,28 @@ The documentation site is built from [`docs/`](docs/index.md) with `mkdocs.yml`
 - **Ingestion**: [the telemetry path](docs/ingestion/index.md), [normalization](docs/ingestion/normalization/index.md) and [the ontology](https://ioc-project.github.io/scope/ontologies.html).
 - **Analysis**: [the analysis engines](docs/analysis/index.md), [pipelines](docs/analysis/pipelines.md), [workers](docs/analysis/workers/index.md) and [components](docs/analysis/components/index.md).
 - **Platform**: [the platform](docs/platform/index.md), [API](docs/platform/api/index.md) and [UI](docs/platform/ui/index.md).
-- **Tutorials**: [your first trace](docs/tutorials/first-trace.md), [instrument your app](docs/tutorials/instrument-your-app.md) and [deploy SCOPE](docs/tutorials/deploy-scope.md).
+- **Tutorials**: [your first trace](docs/tutorials/first-trace.md), [semantic analysis](docs/tutorials/semantic-analysis.md), [instrument your app](docs/tutorials/instrument-your-app.md) and [deploy SCOPE](docs/tutorials/deploy-scope.md).
 - Preview locally: `pip install -r docs/requirements.txt && mkdocs serve`.
 
 The ontology reference documentation (Widoco and WebVOWL) is generated separately:
 
 - **Ontology**: [ontology/index.md](ontology/index.md), also published at [ioc-project.github.io/scope/ontologies.html](https://ioc-project.github.io/scope/ontologies.html)
 
+## Repository structure
+
+| Path | Description |
+| --- | --- |
+| [`backend/`](backend) | Python libraries and workers implementing ingestion, normalization, analysis, and the data API. See [backend/README.md](backend/README.md). |
+| [`backend/api`](backend/api) | Unified DB access layer and REST API over the raw OTel spans and the knowledge graph. |
+| [`backend/norm`](backend/norm) | Normalizes raw OTel traces into structured MAS execution models. |
+| [`backend/mce`](backend/mce) | Metric computation engine for sessions. |
+| [`backend/stateful-evals`](backend/stateful-evals) | Cost-effective evaluation of a session's trajectory. |
+| [`backend/workers`](backend/workers) | Standalone workers built on top of the backend libraries. |
+| [`ui/`](ui) | React/TypeScript web dashboard for browsing applications, sessions, execution graphs, and live topology. See [ui/README.md](docs/platform/ui/index.md). |
+| [`ontology/`](ontology) | The MAS telemetry ontology (Turtle/OWL) and the `scope-ontology` Python package. See [ontology/README.md](ontology/README.md). |
+| [`docs/`](docs) | Documentation site sources (MkDocs), built with `mkdocs.yml`. |
+| [`deploy/`](deploy) | Deployment: Helm charts for Kubernetes (`deploy/k8s`) and a Docker Compose stack (`deploy/docker-compose`) running the API, UI, workers and supporting infra. |
+
 ## Contributing
 
 Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines, [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for community standards, and [SECURITY.md](SECURITY.md) for reporting vulnerabilities. Maintainers are listed in [MAINTAINERS.md](MAINTAINERS.md).

backend/CHANGELOG.md

@@ -9,6 +9,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **The `mce` worker computed one session at a time.** `MCEWrapper.process_mce_session` was `async` but called the blocking
+  evaluation directly, which held the event loop: whatever `MAX_INFLIGHT_MESSAGES` said, sessions went through one after the
+  other, about 8 seconds each with the default metrics. It now runs the evaluation in a thread, and the engine that the sessions
+  share is built once. A burst of 33 sessions had its metrics in 68 seconds instead of 260.
+- **Workers were capped at 16 sessions at a time, below their `MAX_INFLIGHT_MESSAGES`.** The handlers hand their blocking work to
+  `asyncio.to_thread`, whose pool holds `min(32, cpus + 4)` threads (16 on a 12-CPU host), so a worker allowed 32, 64 or 128
+  messages in flight never ran more than 16. `BaseWorker` now sizes the pool to `max_inflight_messages`.
+- **`Connection pool is full, discarding connection: clickhouse` in the logs.** The ClickHouse client of the workers shared
+  clickhouse-connect's default pool of 8 HTTP connections, and the `norm` worker, which reads the spans of many sessions
+  at once, opened more and threw the extra ones away after each request. `ClickHouseConnector` now has its own pool,
+  of 32 connections by default (`CLICKHOUSE_POOL_SIZE` sets it).
+- **Neo4j writes no longer lose a session to a deadlock.** When several workers wrote the same shared nodes at once (a
+  burst of sessions writing the same agents, tools and model), Neo4j rolled one of the transactions back with a
+  transient error (`DeadlockDetected`), and the `norm` worker dropped the whole session. `Neo4JConnector` now tries a query
+  again, up to five times with a short random wait, when Neo4j reports a transient error. The same burst on a new
+  database could also fail on the index that two sessions create at the same time
+  (`EquivalentSchemaRuleAlreadyExists`): that error is now what it says, the index exists.
 - **Normalization: identical boundary States are one State.** When a call started with what the previous step
   produced (a tool called with its agent's input, a tool returning what its agent returns, a handoff that passes
   its output on unchanged), `norm` kept two State nodes with the same content and no edge between them, so the

backend/api/scope/client/dal.py

@@ -527,6 +527,10 @@ def _neo4j_props(values: Dict[str, Any]) -> Dict[str, Any]:
     return props
 
 
+# Neo4j's answer when an index that is being created already exists.
+_INDEX_ALREADY_EXISTS = "Neo.ClientError.Schema.EquivalentSchemaRuleAlreadyExists"
+
+
 def ingest_normalized_kg(
     db: Connector,
     nodes: List[Dict[str, Any]],
@@ -587,9 +591,12 @@ def ingest_normalized_kg(
                     f"FOR (n:{node_type}) ON (n.id)"
                 )
             except Exception as exc:
-                raise DatabaseError(
-                    f"Failed to ensure index for '{node_type}': {exc}"
-                ) from exc
+                # Two sessions can create the same index at once on a new database: the
+                # loser is told that it already exists, which is what it wanted.
+                if getattr(exc, "code", "") != _INDEX_ALREADY_EXISTS:
+                    raise DatabaseError(
+                        f"Failed to ensure index for '{node_type}': {exc}"
+                    ) from exc
             _indexed_node_types.add(node_type)
 
         if node_type in _NAME_KEYED_NODE_TYPES:

backend/api/scope/connectors/clickhouse.py

@@ -11,6 +11,9 @@
 
 logger = logging.getLogger(__name__)
 
+# HTTP connections kept open to ClickHouse, for the requests that run at the same time.
+DEFAULT_POOL_SIZE = 32
+
 
 class ClickHouseConnector(Connector):
     """Manages a ClickHouse connection via clickhouse-connect."""
@@ -22,20 +25,27 @@ def __init__(
         username: str = "default",
         password: str = "",
         database: str = "default",
+        pool_size: int = DEFAULT_POOL_SIZE,
     ):
         self._host = host
         self._port = port
         self._username = username
         self._password = password
         self._database = database
+        self._pool_size = pool_size
         self._client: Any = None  # clickhouse_connect.Client
 
     # ── lifecycle ─────────────────────────────────────────────────────────
 
     def connect(self) -> None:
         import clickhouse_connect
+        from clickhouse_connect.driver import httputil
 
+        # The client shares one HTTP connection pool across all the threads that use it, and
+        # clickhouse-connect's default pool keeps 8 connections: a worker that reads spans for
+        # many sessions at once opens more, and the extra ones are discarded after each request.
         self._client = clickhouse_connect.get_client(
+            pool_mgr=httputil.get_pool_manager(maxsize=self._pool_size),
             host=self._host,
             port=self._port,
             username=self._username,

backend/api/scope/connectors/neo4j.py

@@ -5,18 +5,32 @@
 from __future__ import annotations
 
 import logging
+import random
 import re
+import time as _time
 from datetime import date, datetime, time, timedelta
-from typing import Any, Dict, Optional
+from typing import Any, Callable, Dict, Optional, TypeVar
 from urllib.parse import urlparse
 
 import neo4j.time
 from neo4j import GraphDatabase
+from neo4j.exceptions import TransientError
 
 from scope.connectors.base import Connector
 
 logger = logging.getLogger(__name__)
 
+T = TypeVar("T")
+
+# Neo4j rolls a transaction back and says "retry me" (TransientError) when two writers
+# collide, typically a deadlock on the nodes they share. Several workers write the same
+# structural nodes at once, so a query is tried again a few times before it fails.
+TRANSIENT_ATTEMPTS = 5
+TRANSIENT_BACKOFF_SECONDS = 0.2
+# The first retries are routine under concurrency (two sessions collide about once each), so they are
+# logged at INFO; from this attempt on, something is wrong enough to be worth a WARNING.
+TRANSIENT_WARN_FROM_ATTEMPT = 3
+
 
 SEMANTICGROUP_NODE_CHILDREN_NODES = "childrenNodes"
 SEMANTICGROUP_NODE_GROUP_NAME = "groupName"
@@ -137,16 +151,50 @@ def is_connected(self) -> bool:
 
     # ── query execution ───────────────────────────────────────────────────
 
+    @staticmethod
+    def _retry_transient(run: Callable[[], T]) -> T:
+        """Call ``run``, again after a short random wait when Neo4j reports a transient error.
+
+        A transient error means the transaction was rolled back, so running it again is safe.
+        """
+        for attempt in range(1, TRANSIENT_ATTEMPTS + 1):
+            try:
+                return run()
+            except TransientError as exc:
+                if attempt == TRANSIENT_ATTEMPTS:
+                    raise
+                delay = (
+                    TRANSIENT_BACKOFF_SECONDS
+                    * 2 ** (attempt - 1)
+                    * (0.5 + random.random())
+                )
+                logger.log(
+                    logging.WARNING
+                    if attempt >= TRANSIENT_WARN_FROM_ATTEMPT
+                    else logging.INFO,
+                    "Neo4j transient error (%s), trying again in %.2fs (attempt %d of %d)",
+                    getattr(exc, "code", "unknown"),
+                    delay,
+                    attempt,
+                    TRANSIENT_ATTEMPTS,
+                )
+                _time.sleep(delay)
+        raise AssertionError("unreachable")  # pragma: no cover
+
     def execute(self, query: str, params: Optional[Dict[str, Any]] = None) -> Any:
         self.ensure_connected()
         if logger.isEnabledFor(logging.DEBUG):
             logger.debug(
                 "Executing query:\n***********************\n%s",
                 _render_query_with_params(query, params),
             )
-        with self._client.session(database=self._database) as session:
-            result = session.run(query, parameters=params or {})
-            return [_convert_neo4j_types(record.data()) for record in result]
+
+        def run() -> Any:
+            with self._client.session(database=self._database) as session:
+                result = session.run(query, parameters=params or {})
+                return [_convert_neo4j_types(record.data()) for record in result]
+
+        return self._retry_transient(run)
 
     def execute_command(
         self, command: str, params: Optional[Dict[str, Any]] = None
@@ -158,8 +206,12 @@ def execute_command(
                 "Executing command:\n***********************\n%s",
                 _render_query_with_params(command, params),
             )
-        with self._client.session(database=self._database) as session:
-            session.run(command, parameters=params or {}).consume()
+
+        def run() -> None:
+            with self._client.session(database=self._database) as session:
+                session.run(command, parameters=params or {}).consume()
+
+        self._retry_transient(run)
 
     @property
     def driver(self):

backend/api/scope/dependencies.py

@@ -20,7 +20,10 @@
     redis_lib = None
 
 from scope.connectors.base import Connector
-from scope.connectors.clickhouse import ClickHouseConnector
+from scope.connectors.clickhouse import (
+    DEFAULT_POOL_SIZE as DEFAULT_CLICKHOUSE_POOL_SIZE,
+    ClickHouseConnector,
+)
 from scope.connectors.neo4j import Neo4JConnector
 from scope.connectors.sqlalchemy import SQLAlchemyConnector
 from scope.core.config import settings
@@ -73,6 +76,9 @@ def _get_db_singleton() -> ClickHouseConnector:
             username=(os.getenv("CLICKHOUSE_USERNAME") or settings.CLICKHOUSE_USERNAME),
             password=os.getenv("CLICKHOUSE_PASSWORD", settings.CLICKHOUSE_PASSWORD),
             database=os.getenv("CLICKHOUSE_DATABASE", settings.CLICKHOUSE_DATABASE),
+            pool_size=int(
+                os.getenv("CLICKHOUSE_POOL_SIZE", str(DEFAULT_CLICKHOUSE_POOL_SIZE))
+            ),
         )
         _db_connector.connect()
         logger.info("ClickHouse singleton connector initialised")