Feat/lower deployment footprint

.github/CODEOWNERS

@@ -0,0 +1 @@
+* @scope-maintainers

.github/workflows/backend-main.yaml

@@ -41,6 +41,10 @@ jobs:
           - intelligence-worker
           - stateful-eval-worker
           - cognitive-failure-worker
+          - cognition-protocols-worker
+          - ingestion-worker
+          - semantic-analysis-worker
+          - eval-worker
     permissions:
       contents: read  # required for actions/checkout
       packages: write # required to push the image to GHCR

.github/workflows/backend-verify.yaml

@@ -30,6 +30,10 @@ jobs:
           - intelligence-worker
           - stateful-eval-worker
           - cognitive-failure-worker
+          - cognition-protocols-worker
+          - ingestion-worker
+          - semantic-analysis-worker
+          - eval-worker
     permissions:
       contents: read  # required for actions/checkout
       packages: read

README.md

@@ -8,7 +8,7 @@ SCOPE observes, analyzes and explains the executions of Multi-Agent Systems (MAS
 
 Run the whole stack on your machine with Docker Compose, send it the trace of a small two-agent example, and look at it in the dashboard. It takes about 15 minutes, most of it the first image build.
 
-You need Docker (Compose v2.17 or later, with at least **12 GB** of memory given to it) and [`uv`](https://docs.astral.sh/uv/) to run the example. An OpenAI-compatible LLM endpoint is only needed for the analyses that call a model: without it, traces are still collected and stored in the knowledge graph.
+You need Docker (Compose v2.17 or later, with at least **6 GB** of memory given to it) and [`uv`](https://docs.astral.sh/uv/) to run the example. An OpenAI-compatible LLM endpoint is only needed for the analyses that call a model: without it, traces are still collected and stored in the knowledge graph.
 
 ### 1. Start SCOPE
 
@@ -19,7 +19,7 @@ cp .env.template .env        # set LLM_MODEL_NAME, EMBEDDING_MODEL_NAME, LLM_GAT
 docker compose up -d --build
 ```
 
-### 2. Send a trace**
+### 2. Send a trace
 
 ```bash
 cd ../../examples/currency-demo

backend/CHANGELOG.md

@@ -63,6 +63,60 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Workers can run in one process, and Compose runs three services instead of eight.** `worker_base.runner` starts the
+  workers of a group in threads of one process, each with its own event loop, queues and options, so the interpreter and
+  the libraries they share are loaded once. The new images are `ingestion-worker` (norm, embedding),
+  `semantic-analysis-worker` (grouping, hierarchical-grouping, analysis, intelligence) and `eval-worker` (mce,
+  stateful-eval); the analysis worker, which forks a pool of CPU-bound processes, runs as a child process of its group.
+  `--workers` (or `INGESTION_WORKERS`, `SEMANTIC_ANALYSIS_WORKERS`, `EVAL_WORKERS`) picks some of them, and the settings
+  that all the workers read from one variable can be set per worker with its prefix (`NORM_MAX_INFLIGHT_MESSAGES`,
+  `HIERARCHICAL_GROUPING_PERIODIC_TRIGGER_INTERVAL_SECONDS`; `scoped_getenv` in `worker_base.utils`), and
+  `<WORKER>_ARGS` adds command line options. The single-worker images and CLIs are unchanged (and still needed for Argo
+  run-once mode); the Helm chart still deploys the single workers. On a 33-session replay the three services took 1.6 GB
+  against 2.0 GB for the eight containers, and the pipeline finished in 85 s against 101 s.
+- **The MCE can ignore the metric plugins of some packages (`MCE_IGNORED_PLUGIN_PACKAGES`).** It loads every package that
+  registers a metric in the `metrics_computation_engine.plugins` entry-point group and skips the ones that are not MCE
+  metrics, but loading is importing: in an image that also has the stateful evaluation (`eval-worker`), the MCE imported
+  `mdt-metrics` and `litellm` with it, which took its process from 102 MB to 388 MB after discovery. The `eval-worker` image
+  sets it to `mdt-metrics`; the metric catalog is the same (45 metrics).
+- **The stateful evaluation calls the judge model with the OpenAI SDK, not LiteLLM.** `metrics-computation-engine`'s
+  `LLMClient` is a thin wrapper around `litellm.completion`, and importing it (which `BaseMetric` and `Jury` do) loaded
+  LiteLLM, about 300 MB per process. `stateful_evals_be.llm_client.LLMClient` has the same constructor and `query`
+  signature and calls the `openai` SDK (Azure OpenAI when `api_version` is set); it replaces the package's module when
+  `stateful_evals_be` is imported. A parameter the endpoint refuses (`reasoning_effort`, `temperature`, `response_format`)
+  is dropped and the call retried, as LiteLLM's `drop_params` did. LiteLLM is still installed, since
+  `metrics-computation-engine` depends on it, but no longer imported.
+- **Insights from the cognitive failure detections.** Two intelligence-catalog templates read the
+  `CognitiveFailureReport` nodes: one insight per failure and session (priority from the detector's confidence), and one
+  per failure and semantic group when it is recurring (at least 2 sessions and 30% of the group, medium or high
+  confidence).
+- **Fixed: the intelligence worker failed with `Insight failed ontology validation: concerns: Less than 1 values`.**
+  `ingest_insights` validated each insight without its `concerns` edge to the node it is attached to, which the
+  ontology requires, so no insight could be written. It now passes that edge.
+- **`cognition-protocols-worker` depends on `ioc-scope-mce-core`, not the `ioc-scope-mce` meta package.** It only runs the
+  ACCORD and CONCORD plugin metrics, but the meta package installs every provider (native, sdk, deepeval, opik, ragas,
+  legacy), with `litellm`, `pyarrow`, `scipy`, `scikit-learn` and `pandas` behind them. The image's virtual environment goes
+  from 788 MB to 233 MB; the same 49 protocol metrics are discovered.
+- **`clickhouse-sqlalchemy` is an optional extra of `ioc-scope-api`.** Nothing imports it (span reads use
+  `clickhouse-connect`), but it pulled `asynch` (66 MB) into every worker image. The API image still installs it
+  (`clickhouse-sqlalchemy` extra), for a `SQLAlchemyConnector` pointed at a ClickHouse URL.
+- **`umap-learn` and `numba` are only installed in the API image.** They serve the UMAP projection of
+  `GET /sessions/{id}/latent-space`, but `ioc-scope-api` is also the knowledge-graph client of every worker, so all the worker
+  images carried them, with llvmlite, scikit-learn and scipy behind them. They are now the `latent-space` extra of
+  `ioc-scope-api`, which the API Dockerfile installs (`uv sync --extra latent-space`). The norm worker image goes from
+  970 MB to 494 MB, and no worker image has numba, llvmlite or umap any more (three Dockerfiles no longer need their
+  `--no-install-package numba` workaround). Without the extra, the latent-space endpoint answers 501 and names it.
+- **The embedding worker image no longer installs torch.** `dem.embedding` imported torch, transformers,
+  sentence-transformers, gensim and google-genai as soon as it was imported, so the worker paid about 360 MB of memory for
+  libraries that `OpenAIEmbedder` (the embedder of the Compose file and the Helm chart) never uses. Each embedder now imports
+  its library when it is built. `dem[embedding]` is the API embedders only (openai, google-genai); the local models
+  (`SentenceTransformerEmbedder`, `Qwen3Embedder`, `GensimEmbedder`) moved to the new `dem[embedding-local]` extra, and the
+  embedding worker has a matching `local` extra, which `Dockerfile.gpu` installs. The default image goes from 1.54 GB to
+  0.72 GB and the process from about 470 MB to about 105 MB. A local embedder without its libraries fails with a message
+  that names the extra. To use one in the default image, build a variant with `--extra local`.
+- **The cognition-protocols image builds without an SSH agent.** The ACCORD and CONCORD metric packages are read from the
+  public `ioc-project/cognition-protocols` repository over HTTPS (`ssh: [default]` is gone from the Compose file and the bake
+  target), and the image is part of the CI matrix like the others.
 - **Breaking: metrics are stored as the ontology's `Metric` + `MetricResult`.** A `MetricResult`
   (`result`, `reasoning`, `timestamp`, `evaluation*` cost fields) is linked to its `Metric` definition
   (one per metric name and provider, with `metricProvider`, `unit`, `valueRange`) by `ofMetricType`, and

backend/api/Dockerfile

@@ -25,14 +25,14 @@ COPY --from=ontology . /ontology
 
 # Install dependencies without installing source code yet.
 RUN --mount=type=cache,target=/root/.cache/uv \
-  cd /src && uv sync --frozen --no-dev --no-install-project --no-install-workspace --no-sources --project /src/api
+  cd /src && uv sync --frozen --no-dev --no-install-project --no-install-workspace --no-sources --extra latent-space --extra clickhouse-sqlalchemy --project /src/api
 
 # Copy the API source code.
 COPY api /src/api
 
 # Install the ioc-scope-api package
 RUN --mount=type=cache,target=/root/.cache/uv \
-  cd /src && uv sync --frozen --no-dev --no-editable --project /src/api
+  cd /src && uv sync --frozen --no-dev --no-editable --extra latent-space --extra clickhouse-sqlalchemy --project /src/api
 
 FROM python:3.13-slim-trixie AS runner
 

backend/api/pyproject.toml

@@ -10,23 +10,30 @@ readme = "README.md"
 requires-python = ">=3.13"
 dependencies = [
     "clickhouse-connect>=0.11.0",
-    "clickhouse-sqlalchemy>=0.3.2",
     "fastapi>=0.129.0",
     "neo4j>=6.1.0",
     "packaging>=26.0",
-    "numba>=0.60.0",
     "numpy>=2.3.5",
     "pydantic-settings>=2.13.1",
     "python-dotenv>=1.2.1",
     "requests>=2.32.5",
     "redis>=5.0.0",
     "sqlalchemy>=2.0.46",
-    "umap-learn>=0.5.7",
     "uvicorn>=0.41.0",
     "ruff<=0.15.22",
     "ioc-scope-ontology[models,verify]>=1.1.0",
 ]
 
+[project.optional-dependencies]
+# The SQLAlchemy dialect for ClickHouse (and its asynch driver), for a SQLAlchemyConnector pointed at a ClickHouse URL.
+# Nothing imports it by default: the span reads use clickhouse-connect. Only the API image installs it.
+clickhouse-sqlalchemy = ["clickhouse-sqlalchemy>=0.3.2"]
+# The UMAP projection behind GET /sessions/{id}/latent-space. It pulls in numba and llvmlite, so only the API image
+# installs it: the workers depend on this package for the knowledge-graph client and do not need a projection.
+latent-space = [
+    "umap-learn>=0.5.7",
+    "numba>=0.60.0",
+]
 
 
 [build-system]
@@ -45,7 +52,9 @@ dev = [
     "httpx>=0.28.1",
     "ioc-scope-mce-provider-native",
     "ioc-scope-mce-provider-sdk",
+    "numba>=0.60.0",
     "pytest>=9.0.2",
+    "umap-learn>=0.5.7",
 ]
 
 [tool.uv]

backend/api/scope/client/dal.py

@@ -13,6 +13,7 @@
 from scope_ontology import OntologyValidationError, verify_kg_object
 from scope_ontology.models.edges import (
     aboutMetric,
+    concerns,
     containsSession,
     hasContext,
     hasMedioidSession,
@@ -1835,7 +1836,14 @@ def ingest_insights(
     current_ids_by_target_template: Dict[Tuple[str, str, str], set] = {}
 
     for insight in insights:
-        verification = verify_kg_object(insight)
+        # ``concerns`` is mandatory on an Insight. It is not stored (the target's ``hasInsight`` edge below is what
+        # the graph keeps), so it is asserted here for the shape to see the target the insight is attached to.
+        concerns_edges = (
+            [concerns(source_id=insight.id, target_id=insight.targetNodeId)]
+            if insight.targetNodeId
+            else []
+        )
+        verification = verify_kg_object(insight, edges=concerns_edges)
         if not verification.conforms:
             raise OntologyValidationError(verification)
 

backend/api/scope/client/sessions.py

@@ -1476,7 +1476,14 @@ def _get_session_latent_space(
     """Fetch state embeddings and project them to 2D using UMAP."""
     import numpy as np
     from fastapi import HTTPException
-    from umap import UMAP
+
+    try:
+        from umap import UMAP
+    except ImportError as exc:  # the `latent-space` extra is only installed in the API image
+        raise HTTPException(
+            status_code=501,
+            detail="The latent space needs umap-learn: install ioc-scope-api[latent-space].",
+        ) from exc
 
     query, params = ui_queries.latent_space_query(session_id=session_id)
     try:

backend/api/tests/test_ingest_insights.py

@@ -0,0 +1,58 @@
+#  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+#  SPDX-License-Identifier: Apache-2.0
+"""``ingest_insights`` validates an Insight against its shape, which requires the ``concerns`` edge to its target."""
+
+import datetime
+from unittest.mock import MagicMock
+
+import pytest
+from scope_ontology import OntologyValidationError
+from scope_ontology.models.nodes.insight import Insight
+
+from scope.client import dal
+
+
+def _insight(**overrides) -> Insight:
+    fields = {
+        "id": "insight-1",
+        "dataType": "text",
+        "templateId": "high_cost_agent",
+        "description": "Agent X spent 3x the median cost.",
+        "scope": "Agent",
+        "labels": "[]",
+        "targetNodeId": "agent-x",
+        "createdAt": datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc),
+    }
+    fields.update(overrides)
+    return Insight(**fields)
+
+
+@pytest.fixture()
+def graph(monkeypatch):
+    nodes, rels = [], []
+    monkeypatch.setattr(
+        dal, "_ingest_node", lambda db, label, props: nodes.append(props)
+    )
+    monkeypatch.setattr(dal, "_create_rel", lambda *args: rels.append(args))
+    return nodes, rels
+
+
+def test_an_insight_with_a_target_is_written_and_attached(graph):
+    nodes, rels = graph
+    db = MagicMock()
+
+    dal.ingest_insights(db, [_insight()])
+
+    assert [n["id"] for n in nodes] == ["insight-1"]
+    assert rels[0][1:4] == ("Agent", ["agentName"], ["agent-x"])
+    assert rels[0][-1] == "hasInsight"
+    db.execute.assert_called_once()  # clean-up of the outdated insights of this template
+
+
+def test_an_insight_without_a_target_is_rejected(graph):
+    nodes, _ = graph
+
+    with pytest.raises(OntologyValidationError):
+        dal.ingest_insights(MagicMock(), [_insight(targetNodeId="")])
+
+    assert nodes == []

backend/api/tests/test_latent_space_extra.py

@@ -0,0 +1,20 @@
+#  Copyright (c) 2026 Cisco Systems, Inc. and its affiliates
+#  SPDX-License-Identifier: Apache-2.0
+"""The UMAP projection is an optional extra of the API: without it the endpoint says so instead of failing."""
+
+from __future__ import annotations
+
+import sys
+
+import pytest
+from fastapi import HTTPException
+
+from scope.client import sessions
+
+
+def test_latent_space_without_umap_answers_501(monkeypatch):
+    monkeypatch.setitem(sys.modules, "umap", None)  # importing it raises ImportError, as when it is not installed
+    with pytest.raises(HTTPException) as raised:
+        sessions._get_session_latent_space(None, None, session_id="s1")
+    assert raised.value.status_code == 501
+    assert "ioc-scope-api[latent-space]" in raised.value.detail

backend/dem/pyproject.toml

@@ -39,13 +39,17 @@ grouping = [
 llm = [
     "openai>=2.8.1",
 ]
+# Embedders that call an API (OpenAI-compatible gateway, Google). This is what the embedding worker image installs.
 embedding = [
+    "openai>=2.8.1",
+    "google-genai>=1.51.0",
+]
+# Embedders that run a model in the process (sentence-transformers, Qwen3, gensim): they pull in torch.
+embedding-local = [
     "sentence-transformers>=5.1.2",
     "transformers>=4.57.1",
     "torch>=2.0.0",
     "gensim>=4.4.0",
-    "openai>=2.8.1",
-    "google-genai>=1.51.0",
 ]
 analysis = [
     "scikit-learn>=1.7.2,<1.9.0",
@@ -55,6 +59,7 @@ all = [
     "dem[grouping]",
     "dem[llm]",
     "dem[embedding]",
+    "dem[embedding-local]",
     "dem[analysis]",
 ]
 dev = [