feat(tracing): own the traces with OpenTelemetry instead of Langfuse

.env.eval.example

@@ -16,7 +16,5 @@ S3_BUCKET_NAME=uploads
 S3_REGION=us-east-1
 S3_FORCE_PATH_STYLE=true
 
-LANGFUSE_ENABLED=false
-
 # Evals call a real model — this is the only cost of a run.
 ANTHROPIC_API_KEY=

.env.example

@@ -82,12 +82,9 @@ S3_FORCE_PATH_STYLE=true  # Required for MinIO, false for AWS S3
 # S3_FORCE_PATH_STYLE=false
 
 
-# Langfuse Observability (optional)
-# Set LANGFUSE_ENABLED=true and provide your API keys to enable LLM tracing.
-# Cloud: https://cloud.langfuse.com
-# Self-hosted: point LANGFUSE_BASE_URL to your own instance (e.g. http://localhost:3030)
-# See docs/OBSERVABILITY.md for setup instructions.
-LANGFUSE_ENABLED=false
-LANGFUSE_SECRET_KEY="sk-lf-..."
-LANGFUSE_PUBLIC_KEY="pk-lf-..."
-LANGFUSE_BASE_URL="https://cloud.langfuse.com"
+# Tracing (optional, OpenTelemetry). Off unless an endpoint is set — nothing leaves the process
+# by default. `docker compose --profile tracing up -d` starts a local Phoenix on 6006.
+# See docs/OBSERVABILITY.md (other backends, auth headers, what a trace contains).
+# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:6006
+# Prompts, messages, tool arguments and results are NOT sent unless this is "true".
+# OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false

AGENTS.md

@@ -42,12 +42,13 @@ pnpm db:studio                # Database UI
 `compose.yaml` defines **two stacks that share no data**. The separation is physical — separate
 services, separate named volumes — not a naming convention.
 
-|          | Default (`up -d`)            | `--profile full`                  |
-| -------- | ---------------------------- | --------------------------------- |
-| Postgres | `db` → `mydb_dev`, port 5544 | `db-prod` → `mydb`, port **5546** |
-| MinIO    | `minio`, 9100/9101           | `minio-prod`, **9102/9103**       |
-| Volumes  | `db_data`, `minio_data`      | `prod_db_data`, `prod_minio_data` |
-| Contents | seed/scratch data            | **real financial records**        |
+|          | Default (`up -d`)                               | `--profile full`                                       |
+| -------- | ----------------------------------------------- | ------------------------------------------------------ |
+| Postgres | `db` → `mydb_dev`, port 5544                    | `db-prod` → `mydb`, port **5546**                      |
+| MinIO    | `minio`, 9100/9101                              | `minio-prod`, **9102/9103**                            |
+| Phoenix  | `phoenix`, 6006 (only with `--profile tracing`) | `phoenix-prod`, **6016**                               |
+| Volumes  | `db_data`, `minio_data`, `phoenix_data`         | `prod_db_data`, `prod_minio_data`, `prod_phoenix_data` |
+| Contents | seed/scratch data                               | **real financial records and their traces**            |
 
 **`docker compose down -v` deletes named volumes.** Because the volume sets are disjoint, a `-v`
 teardown of the dev stack cannot reach production. Removing production data requires naming that
@@ -578,14 +579,27 @@ how this repo's code gets written. Tool conventions here follow
 npx skills add agentailor/skills --skill tool-design
 ```
 
-## Langfuse Observability
-
-LLM tracing is implemented via two mechanisms — see [docs/OBSERVABILITY.md](docs/OBSERVABILITY.md) for full setup instructions.
-
-- **`instrumentation.ts`** (project root): Next.js calls `register()` once before any route; initializes the OTel SDK with `LangfuseSpanProcessor` when `LANGFUSE_ENABLED=true`
-- **`src/services/agentService.ts`**: Conditionally attaches `CallbackHandler` to each `agent.stream()` call for LangGraph-specific traces (LLM calls, tool invocations, token counts)
-- Toggle tracing: set `LANGFUSE_ENABLED=true/false` in `.env` — no code changes needed
-- Works with Langfuse Cloud or a self-hosted Docker instance
+## Tracing (OpenTelemetry)
+
+Agent runs are traced as OTel GenAI spans and exported over OTLP to a backend the owner runs —
+Phoenix by default. Full detail in [docs/OBSERVABILITY.md](docs/OBSERVABILITY.md).
+
+- **Off unless `OTEL_EXPORTER_OTLP_ENDPOINT` is set** (`src/lib/tracing/config.ts`). There is no
+  default destination in code; the production compose stack points at its own `phoenix-prod`.
+- **Content capture is a separate opt-in** (`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`).
+  Off, no prompt, message, tool argument, tool result or error message leaves the process — only
+  names, counts, timings and token usage.
+- **`GenAiTracer`** (`src/lib/tracing/genAiTracer.ts`) is a LangChain callback attached per
+  request in `agentService.ts`. It spans the run (`invoke_agent`), each model call (`chat`) and
+  each tool call (`execute_tool`) — never graph nodes or middleware — and records the approval
+  gate as an event (tool names only) and the resume's approve/reject counts.
+- **`semconv.ts` is the only file naming `gen_ai.*` / `cameron.*` attributes.** The conventions are
+  still Development stability; isolate any rename there.
+- **`src/instrumentation.ts`** starts the Node SDK with the OTLP/proto exporter. It must live in
+  `src/`: with an `src/app` layout the production build only looks there (a root-level file still
+  runs under `pnpm dev`, which hides the gap). Next.js adds its own route and `fetch` spans to the
+  same trace.
+- No vendor SDK: Langfuse, Jaeger or anything OTLP is an endpoint change, not a code change.
 
 ## OpenAPI / API Docs
 

README.md

@@ -184,7 +184,7 @@ The technical detail lives in [`docs/`](docs/) rather than here:
 | [TESTING.md](docs/TESTING.md)             | Why the unit suite stays free and offline, and what belongs in evals instead                                                                            |
 | [API.md](docs/API.md)                     | The generated OpenAPI spec, served at `/api/openapi` and browsable at `/api-docs`                                                                       |
 | [OAUTH.md](docs/OAUTH.md)                 | OAuth for MCP servers that need it                                                                                                                      |
-| [OBSERVABILITY.md](docs/OBSERVABILITY.md) | Langfuse tracing setup                                                                                                                                  |
+| [OBSERVABILITY.md](docs/OBSERVABILITY.md) | OpenTelemetry tracing to a self-hosted backend (Phoenix by default), what a trace contains, and swapping backends                                       |
 | [eval/README.md](eval/README.md)          | The paid, non-deterministic eval suite — never in CI                                                                                                    |
 
 [AGENTS.md](AGENTS.md) carries the conventions a coding agent needs to work in this repo.

compose.yaml

@@ -2,13 +2,15 @@
 #
 #   docker compose up -d                 DEV dependencies: Postgres `mydb_dev` (5544) + MinIO
 #                                        (9100/9101), app on the host. Disposable by design.
-#   docker compose --profile full up     PRODUCTION: the whole app on its OWN Postgres (5546) and
-#                                        its OWN MinIO (9102/9103). This is the database holding
-#                                        real financial records.
+#   docker compose --profile tracing up -d
+#                                        ...plus the DEV trace backend, Phoenix (6006).
+#   docker compose --profile full up     PRODUCTION: the whole app on its OWN Postgres (5546), its
+#                                        OWN MinIO (9102/9103) and its OWN Phoenix (6016). This is
+#                                        the stack holding real financial records — and their traces.
 #
-# The split is physical, not by convention: `db-prod` and `minio-prod` use their own named volumes,
-# so `docker compose down -v` (which deletes volumes) on the default profile CANNOT reach
-# production data. Removing production data takes naming that stack explicitly.
+# The split is physical, not by convention: `db-prod`, `minio-prod` and `phoenix-prod` use their
+# own named volumes, so `docker compose down -v` (which deletes volumes) on the default profile
+# CANNOT reach production data. Removing production data takes naming that stack explicitly.
 #
 #   docker compose down                             stop dev, keep data
 #   docker compose down -v                          stop dev, DESTROY dev data (safe: dev only)
@@ -74,6 +76,21 @@ services:
       exit 0;
       "
 
+  # Dev trace backend, `--profile tracing` only. Point the host app at it with
+  # OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:6006 (see docs/OBSERVABILITY.md).
+  phoenix:
+    profiles: [tracing]
+    image: arizephoenix/phoenix:version-20.23.0-nonroot
+    container_name: cameron-ai-phoenix
+    environment:
+      PHOENIX_TELEMETRY_ENABLED: "false"
+    ports:
+      # Loopback only: Phoenix has no login, and traces describe your finances.
+      - "127.0.0.1:${PHOENIX_PORT:-6006}:6006"
+    volumes:
+      # The nonroot image can only write under its own home, so the volume mounts there.
+      - phoenix_data:/home/nonroot
+
   # ── `--profile full` only: PRODUCTION. Real financial data. ────────────────────────────────────
 
   # Separate service and separate volume from `db` above — that separation is the whole point, so
@@ -135,6 +152,19 @@ services:
       exit 0;
       "
 
+  # Production traces: the same split as the database, so a dev teardown cannot reach them.
+  phoenix-prod:
+    profiles: [full]
+    image: arizephoenix/phoenix:version-20.23.0-nonroot
+    container_name: cameron-ai-phoenix-prod
+    environment:
+      PHOENIX_TELEMETRY_ENABLED: "false"
+    ports:
+      - "127.0.0.1:${PHOENIX_PROD_PORT:-6016}:6006"
+    volumes:
+      - prod_phoenix_data:/home/nonroot
+    restart: unless-stopped
+
   # Applies pending migrations, then exits. Its own service (and its own build stage) because
   # drizzle-kit is a devDependency that the runtime image deliberately does not carry.
   migrate:
@@ -182,17 +212,22 @@ services:
       ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
       OPENAI_API_KEY: ${OPENAI_API_KEY:-}
       GOOGLE_API_KEY: ${GOOGLE_API_KEY:-}
+      OPENAI_COMPATIBLE_API_KEY: ${OPENAI_COMPATIBLE_API_KEY:-}
 
-      LANGFUSE_ENABLED: ${LANGFUSE_ENABLED:-false}
-      LANGFUSE_SECRET_KEY: ${LANGFUSE_SECRET_KEY:-}
-      LANGFUSE_PUBLIC_KEY: ${LANGFUSE_PUBLIC_KEY:-}
-      LANGFUSE_BASE_URL: ${LANGFUSE_BASE_URL:-https://cloud.langfuse.com}
+      # Traces go to this stack's own Phoenix, metadata only. PROD_-prefixed so a host-dev value in
+      # .env (e.g. localhost:6006) never redirects production traces. See docs/OBSERVABILITY.md.
+      OTEL_EXPORTER_OTLP_ENDPOINT: ${PROD_OTEL_EXPORTER_OTLP_ENDPOINT:-http://phoenix-prod:6006}
+      OTEL_EXPORTER_OTLP_HEADERS: ${PROD_OTEL_EXPORTER_OTLP_HEADERS:-}
+      OTEL_SDK_DISABLED: ${PROD_OTEL_SDK_DISABLED:-false}
+      OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT: ${PROD_OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT:-false}
     restart: unless-stopped
 
 volumes:
   # Dev — disposable.
   db_data:
   minio_data:
-  # Production — real financial records. Only a `--profile full down -v` removes these.
+  phoenix_data:
+  # Production — real financial records and their traces. Only a `--profile full down -v` removes these.
   prod_db_data:
   prod_minio_data:
+  prod_phoenix_data: