feat(agent): add opt-in agent tracing to Inference.net - #197
Open
github-actions[bot] wants to merge 17 commits into
Open
feat(agent): add opt-in agent tracing to Inference.net#197github-actions[bot] wants to merge 17 commits into
github-actions[bot] wants to merge 17 commits into
Conversation
Adds agent/instrumentation.ts so eve exports its AI SDK spans over OTLP. A turn becomes a trace: ai.eve.turn, one ai.streamText per step, the model call under it, and an ai.toolCall per tool, carrying token counts and, on gateway-served calls, cost. Built on `eve add instrumentation/raindrop`, then hardened for this repo: - The scaffold interpolates the write key with no guard, so an install without one sends "Bearer undefined" to the vendor on every turn. resolveTraceDestination treats blank and whitespace as unset. - Falls back to OTEL_EXPORTER_OTLP_ENDPOINT, so a local Jaeger or any other OTLP backend works with no code change. - registerOTel is wrapped. A tracing failure logs and the agent runs on. - The boot line names the destination, never the key. - instrumentation.ts loads @crm/env itself: eve runs it before any agent code, so agent.ts's own load has not happened yet. - The variable goes in the root .env. eve add writes a per-package .env.local, which AGENTS.md forbids. recordInputs and recordOutputs are true, by the owner's decision, so spans carry the system prompt, message history and model output. On this CRM that is customer email bodies, contact names and deal amounts. Redaction is not built. docs/agent.md records it as the one deliberate exception to "no customer text in a third-party query". Both flags are set explicitly: eve 0.29.4's types say they default to true and the published guide says false, so naming them stops an upgrade changing what leaves. Adding this file replaces eve's zero-config local trace store, so `eve traces ls` and the /traces TUI record nothing. eve's local runtime is internal and throws if a second OTel runtime registers, so it cannot be kept as a fallback.
Replaces the raw OTLP POST with raindrop-ai's supported external-OTel path. Raindrop's own skill prescribes it: when a project already has a tracing system with an exporter or plugin pattern, integrate as a processor rather than adding separate middleware. eve is that system. @raindrop-ai/ai-sdk is the wrong package here. It needs wrap(ai) at the call site, and no workspace imports the AI SDK — eve owns it. Wrapping it would need a fork of eve, which the skill's guardrails rule out. bun add raindrop-ai@otelv2 The otelv2 tag because @opentelemetry/sdk-trace-base resolves to 2.10.0 in this tree; plain raindrop-ai is built against OTEL v1. Attribution comes from eve's step.started hook, on the keys Raindrop actually reads: traceloop.association.properties.user_id session initiator principal traceloop.association.properties.convo_id eve's durable session id Set as span attributes and returned as runtimeContext, so child spans inherit them. Not done, deliberately: - trackSignal. The thumbs up/down exists and writes AgentConversationFeedback, but the write path is in Nest, and AGENTS.md forbids a vendor client there. Doing it properly means the API writing a row and the agent forwarding it. - redactPii stays false. The owner chose full content while agents are being built, so spans still carry customer email bodies, contact names and deal amounts. One line in tracing-config.ts flips it. - projectId omitted. No slug was given, so events go to the org's default Production project. The OTLP endpoint fallback is kept, so a local Jaeger still works when no write key is set.
Replaces the Raindrop span processor with @inference/tracing, which has a first-class eve integration. agent/instrumentation.ts now exports defineCatalystEveInstrumentation() rather than eve's defineInstrumentation: it installs the OTel provider from eve's startup hook and enriches the spans with OpenInference attributes and the $eve.* workflow tags. This is the first vendor in this sequence that supports the stack natively. Foglamp and Raindrop both instrument by wrapping the `ai` module, which no workspace here imports — eve owns it. Inference hooks eve itself, so a turn arrives shaped rather than raw: ai.eve.turn CHAIN, with $eve.parent / $eve.root / $eve.trigger invoke_agent AGENT, with agent.name and gen_ai.system=eve model calls LLM, with messages, finish reason and token usage tools TOOL, with name, call id, arguments and result The integration composes events["step.started"] rather than replacing it, so the attribution survives the swap. It types session.auth as unknown, so principalOf parses it with zod instead of casting, and prefers the session initiator — a subagent turn stays attributed to the rep who started the root session. Dropped raindrop-ai, @vercel/otel and @opentelemetry/api. The agent now carries one tracing dependency instead of three. Kept from the previous pass: a blank key reads as unset so an install without one runs untraced rather than exporting with no token, the boot line names the endpoint and never the token, and instrumentation.ts loads @crm/env itself because eve runs it before any agent code. recordInputs and recordOutputs stay true by the owner's decision, so spans still carry customer email bodies, contact names and deal amounts. Redaction is still not built.
`bun run db:test` failed with "Neither TEST_DATABASE_URL nor DATABASE_URL is set" even with both set in the root .env. The script read process.env directly and relied on Bun auto-loading a working-directory .env, but `bun run --filter=@crm/db` sets the cwd to packages/db, where there is no .env and never should be — AGENTS.md allows one .env, at the root. `@crm/env/load` walks up to the workspace root, which is what every other entry point in the package already does, prisma.config.ts included. Verified: `bun run db:test` from the repo root now creates and migrates crm_test with no manually exported variables.
…al user Two changes, both from reading the platform rather than guessing. ## inference_model_ prefix A model id stays a Vercel AI Gateway id by default. Prefixing it with `inference_model_` routes that one model to Inference.net instead, via an @ai-sdk/openai-compatible provider. eve accepts a LanguageModel instance as well as a gateway string and classifies it as `external` routing, so the gateway is bypassed rather than wrapped. zai/glm-5.2-fast -> Vercel AI Gateway inference_model_glm-5.2-fast -> Inference.net, as "glm-5.2-fast" The prefix is stripped before the id is sent, which is the point: Inference prices `glm-5.2-fast`, the gateway calls the same model `zai/glm-5.2-fast`, and that mismatch is why cost came back null on every trace. A missing INFERENCE_API_KEY logs one line and falls back to the default model rather than failing the session. ## user.id attribution Two bugs, one visible symptom — an empty user facet on every span. First, the value never reached the span. @inference/tracing's eve integration copies an allowlist out of runtime context — the ten $eve.* keys and nothing else — so our `user.id` arrived as `ai.settings.context.user.id` and never became the attribute the dashboard reads. It is now set with `trace.getActiveSpan()?.setAttribute`; the runtimeContext copy stays so child spans inherit it. Second, the value was wrong anyway. A probe of the live session showed `principalId: "eve:app", principalType: "runtime"` — the background runtime principal, not a person. Filling a people facet with a robot is worse than leaving it empty, so `principalOf` now returns a principal only when `principalType` is `user`, which is what the bridge mints for a rep (channels/eve.ts) and never what dispatch mints (lib/app-auth.ts).
eve fixes model routing at compile time from the authored model value, so a provider instance returned from the `session.started` resolver never reaches the wire — `routeOf` resolved to Inference and every call still went through the Vercel AI Gateway. Verified against live spans: the model stayed `zai/glm-5.2-fast` with provider `gateway` across five runs. That left the prefix looking functional while doing nothing, which is worse than not having it: setting `inference_model_*` in Settings would silently keep using the gateway. Making it work needs the model authored as a provider instance in defineAgent, which makes Inference the compile-time route for every run and collides with ModelCatalogService and the Settings model chooser. That is a separate decision, and the owner has chosen to stay on the gateway and read cost from Vercel instead. Reverts model.ts to the gateway-string form and drops model-config.ts, its spec, and @ai-sdk/openai-compatible. The user.id attribution fix from the same commit is kept — that one works and is verified.
Sets recordInputs and recordOutputs to false, so spans carry the shape of a turn — tree, timings, tokens, tool names, errors — and not what the model was told or said. Both were true by a deliberate earlier decision, to debug agents while they were being built. A canary test showed what that meant in practice: a planted customer email arrived verbatim in the vendor, on the TOOL span as a tool result and again on the AGENT span inside the prompt. Adds scripts/scan-trace-egress.ts, which scans a Catalyst trace export for personal data and reports the carrying spans. Against 44,957 exported spans it found 264 carriers: 510 email addresses, 22 UK mobiles, 22 postcodes, 22 national insurance numbers, 20 US phone numbers and 30 sensitive-category terms. It takes an export file or URL rather than fetching one, because there is no public REST endpoint for trace exports; queue one from the dashboard or over MCP. Outstanding, and the reason this is not yet a complete fix: 122 of those carriers were eve's own `step.execute` spans, which have no OpenInference kind. Whether these two flags suppress those is unverified. Re-run the scan over a window recorded after this change and compare before relying on it.
step//eve@0.29.4//turnStep matched the email pattern, so every export reported five people who do not exist. A scanner that cries wolf is worse than no scanner: the one real finding is lost in its own noise. The pattern now requires an alphabetic top-level domain, which 0.29.4 is not, and step. joins the skipped key prefixes.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 12 files
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
The anti-slop check failed on five findings in the tracing files. principalOf now takes eve's step-started session, typed from EveInstrumentationStepStartedEventInput["session"], and parses the unknown auth field with the existing sessionAuth schema. The unknown value no longer travels as a parameter. scan-trace-egress.ts now owns a traceSpan schema. It parses each JSONL line into span ids, a name and a dictionary of attribute text. A string attribute stays as it is and every other JSON value is stringified in the schema, so the loop reads plain strings and needs no typeof check and no unknown dictionary. instrumentation.ts passes the Inference token, endpoint, service name and function id as direct properties of a narrowed destination, and builds runtimeContext from a conditional expression. Both conditional empty-object spreads are gone. An absent field is undefined, which the tracing SDK skips exactly as it skipped a missing key.
docs/agent.md already described this as a deliberate exception to the egress rules, chosen so agents can be debugged while they are being built. The code set both flags to false, so the doc, the boot message and the code disagreed about whether customer text leaves. The doc is the intent. Every span now carries the system prompt, the message history and the model's reply, which on this CRM means email bodies, contact names, addresses and deal amounts reaching Inference.net. There is no redaction. Rule 1 in docs/agent.md now names its own exception, and skills/data-boundaries.md says the same thing to the agent, because a trace is not private.
A trace carries the system prompt, the message history and the model's reply, so on this CRM it carries customer email bodies, contact names, addresses and deal amounts. Recording stays the default, because that is what the traces are for while agents are being built, but an install that cannot put the tracing backend under the same access control as the CRM now has a switch. Only 0, false, no and off withhold it. Any other value records, so a typo cannot silence a trace somebody thought they were capturing. The boot line says which mode is live rather than claiming one.
docs/telemetry.md listed prompts, completions and reasoning traces under "what is never sent". That is true of PostHog and false of the install, because tracing sends exactly those to Inference.net. A reader took the claim for a property of the product. The bullet now says which destination it describes, and a section states the other one: what a span carries, that CRM_TELEMETRY_DISABLED does not govern it, how to withhold the content without losing the traces, and that an install sending content is holding CRM data in a second system that needs the same access control. instrumentation.spec.ts proves the withheld decision reaches the SDK rather than stopping at the resolver.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 11 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
A postcode, a national insurance number, a social security number and a bare date are all shorter than the 12-character content gate, so the audit reported a clean export while the patterns never ran. A failed download also read as an empty export and exited 0.
…refix The tracing sections still named recordInputs and recordOutputs as the flags to flip, which INFERENCE_RECORD_CONTENT replaced, so an operator could not find the switch. The inference_model_ section documented routeOf and INFERENCE_BASE_URL, both of which the revert removed.
Only the withheld path reached the SDK, so a hardcoded false passed.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 7 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
The usage text asks for a get_trace_export_download_url link, whose query string is the credential. A failed download printed the whole url to stderr. The origin and the path are enough to name the host that refused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opened automatically when
feat/agent-tracingwas pushed.The title is written from the diff and rewritten as you push, because this is squashed onto
mainand the title becomes the commit subject and the changelog line. Retitle it yourself and it is yours — the automation stops touching it.Summary by cubic
Adds opt-in agent tracing to Inference.net so every agent turn can be inspected in their dashboard. Tracing is off unless
INFERENCE_API_KEYis set; when on, spans carry the system prompt, message history, and model reply — customer email bodies, contact names, addresses, and deal amounts — with no redaction, andINFERENCE_RECORD_CONTENT="0"withholds content while keeping spans, timings, and tokens; the instrumentation replaces eve's local trace store, soeve traces lsand the/tracesTUI record nothing.Notes
INFERENCE_API_KEY,INFERENCE_OTLP_ENDPOINT, andINFERENCE_SERVICE_NAMEconfigure the destination; a blank key is treated as unset, andCRM_TELEMETRY_DISABLEDdoes not govern tracing.scripts/scan-trace-egress.tsto scan a Catalyst trace export for personal data; it catches short values like postcodes and national insurance numbers, and a failed download is an error rather than a clean export. The error names only the host, keeping the presigned download URL's credential out of stderr.bun run db:testso it loads the root.envvia@crm/env/load.Written for commit dbdb802. Summary will update on new commits.