Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Merged
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector
Aug 21, 2026
Merged

feat(dev): collect local OTEL traces in project dev#1980
tejaskash merged 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskash tejaskash commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review after #2043

Until #2043 merges, the diff shows both PRs' commits (GitHub's stack tooling wouldn't retarget the base); the work to review here is the single collector commit on top. It collapses to just that once #2043 merges.

What this does

agentcore project dev gives a local agent tracing with no extra config: an in-process OTLP/HTTP receiver starts on a loopback port, the agent's OpenTelemetry SDK is pointed at it via env vars, and traces persist through #2043's TraceStore to agentcore/.cli/traces/. Nothing leaves the machine; --no-traces or instrumentation.enableOtel: false opts out.

  • io/httpServer.ts — loopback-only node:http primitive (body caps, abort-signal close).
  • otel/collector.ts — POST /v1/traces + /v1/logs, protobuf or JSON. Pinned @opentelemetry/otlp-transformer@0.213.0, the last version shipping request decoders — do not bump.
  • Handler wiring — collector lifecycle on the command's abort signal; signal-specific OTEL env vars so stray shell values can't redirect traces; host.docker.internal rewrite for containers; sitecustomize-on-PYTHONPATH so uvicorn --reload workers stay instrumented (a wrapper would only instrument the reloader parent).

Verification

Per-layer tests (receiver over real ephemeral-port HTTP with protobuf + JSON fixtures; handler with fake runners/collector) plus end to end: built bundle → real project create → project dev → real Bedrock invocation → traces on disk. Protobuf decode verified under bun, the Node bundle, and a compiled binary.

@github-actions github-actions Bot added the size/xl PR size: XL label Aug 12, 2026
@github-actions github-actions Bot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696c Compare August 14, 2026 20:38
@github-actions github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1 Compare August 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.28315% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.17%. Comparing base (a41b4de) to head (10be0a6).

Files with missing lines Patch % Lines
src/io/httpServer.ts 97.05% 2 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##           refactor    #1980      +/-   ##
============================================
+ Coverage     97.15%   97.17%   +0.02%     
============================================
  Files           387      389       +2     
  Lines         23124    23386     +262     
============================================
+ Hits          22465    22726     +261     
- Misses          659      660       +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9 Compare August 17, 2026 17:53
@tejaskash tejaskash mentioned this pull request Aug 17, 2026
Closed
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from 84429af to d070faf Compare August 20, 2026 16:09
tejaskash added a commit that referenced this pull request Aug 20, 2026
@tejaskash
feat(dev): OTLP trace storage for local dev
6943243 
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from d070faf to dd15e9e Compare August 20, 2026 16:18
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from dd15e9e to 0c0682e Compare August 21, 2026 15:23
tejaskash added a commit that referenced this pull request Aug 21, 2026
@tejaskash
feat(dev): OTLP trace storage for local dev
dffe4ee 
Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from be5c3d5 to 4b80241 Compare August 21, 2026 15:44
Base automatically changed from feat/dev-otel-storage to refactor August 21, 2026 17:51
tejaskash added a commit that referenced this pull request Aug 21, 2026
@tejaskash
feat(dev): OTLP trace storage for local dev (#2043)
b03cecb 
* feat(dev): OTLP trace storage for local dev

Pure OTLP wire handling (per-trace batch partitioning, id normalization,
frontend shaping) and append-only per-trace JSONL storage. A batch routinely
carries spans from several traces, so persistence partitions by trace id —
writing whole batches under the first id corrupts trace identity. Consumed by
the OTLP collector in #1980, which stacks on this.

* fix(dev): match trace filters against every participating service

A distributed trace spans several local agents whose exports append to the
same trace file; filtering by any participant must find it, not only the
first service seen.

* fix(dev): address review on OTLP trace storage

- narrow OtlpAttributes to the key/value wire form (the flat-record variant
  had no producer); drop the dead passthrough branches
- flatten array attributes through extractAnyValue so ints stay numeric and
  nested kvlists survive (was stringifying and dropping them)
- count rendered (post-filter) spans for the list summary instead of raw
  records, so the count matches the waterfall the inspector shows
- surface non-ENOENT fs errors from reads instead of masking them as empty
- add newest-N limit to list() for the inspector's per-invocation poll

* fix(dev): flatten kvlist-valued OTLP attributes instead of dropping them

flattenAttributes hand-rolled a value branch per AnyValue kind and had no
kvlistValue case, so an attribute whose value is a kvlist (or anything the
chain didn't enumerate) silently vanished. Route every attribute value
through extractAnyValue, which already unwraps all variants including
kvlist — smaller and complete.

Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist
must not disappear).
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from 4b80241 to 733c274 Compare August 21, 2026 17:51
@tejaskash

tejaskash commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

#2039 was superseded by #2043, which just merged — this is unblocked and rebased on it.

@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch 2 times, most recently from fce12fd to 27fbae3 Compare August 21, 2026 17:59
Hweinstock
Hweinstock reviewed Aug 21, 2026
View reviewed changes
Comment thread src/core/dev/otel/collector.ts
Comment thread src/core/dev/otel/collector.ts Outdated
Comment thread src/io/httpServer.ts
Comment thread src/router/flags.tsx Outdated
Comment thread src/core/dev/otel/collector.ts
Comment thread src/core/dev/otel/collector.ts Outdated
Comment thread src/core/dev/otel/collector.ts
Comment thread src/core/dev/otel/collector.ts
Comment thread src/core/dev/codezip.ts
Comment thread src/core/dev/codezip.ts Outdated
tejaskash added a commit that referenced this pull request Aug 21, 2026
@tejaskash
docs(dev): clarify collector/httpServer/flag comments per review
507dfd6 
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
  — the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
  (ack + hand to onError) rather than the caller's report-once behavior; drop
  the volatile "matches the reference CLI" aside.
notgitika
notgitika reviewed Aug 21, 2026
View reviewed changes
Comment thread src/core/dev/otel/collector.ts Outdated
Comment thread src/core/dev/otel/collector.ts Outdated
Comment thread src/core/dev/codezip.ts Outdated
Comment thread src/core/dev/container.ts
Comment thread src/io/httpServer.ts
aidandaly24
aidandaly24 reviewed Aug 21, 2026
View reviewed changes
Comment thread src/core/dev/container.ts
Comment thread src/handlers/project/dev/index.ts
Comment thread src/handlers/project/dev/index.ts Outdated
Comment thread src/handlers/project/dev/index.ts
Comment thread src/core/dev/otel/collector.ts Outdated
Comment thread src/core/dev/codezip.ts Outdated
@tejaskash
feat(dev): collect local OTEL traces in project dev
70e299b 
An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer
decoders, or JSON) persists agent traces through the storage layer. project
dev starts it unless --no-traces or the runtime disables instrumentation,
points every spawned agent at it (signal-specific OTEL env), rewrites the
endpoint to host.docker.internal for containers — with an explicit
host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn
--reload workers instrumented via sitecustomize on PYTHONPATH. Oversized
collector requests get a 413 before the connection closes so exporters do
not retry them as transient failures.

Rebuilt from explicit paths: the previous tree-snapshot commit accidentally
reverted unrelated merged work (cdk target guard, config-bundle TUI, error
classification).
@tejaskash
fix(dev): ack OTLP exports and report trace-persistence failures
9b90471 
A batch that can't be persisted (disk full, permissions) was being
turned into a 500, which the OTEL SDK exporter retries forever while the
user sees nothing. Ack the export (200) so retries stop, and surface the
failure once via an onError sink threaded from the collector to the dev
handler, which owns the IO to warn the user.

Addresses Gitika's review on store.ts:45 (catch in one place; don't let
persistence faults read as a silent, retried loss).
@tejaskash
test(dev): cover spanCount noise-filtering; clarify hex-id helper name
b34bd04 
Follow-ups from #2043 review (Gitika, Harrison):
- Add a TraceStore.list spanCount test with transport-noise spans
  (1 agent + 4 http-send -> "1"), guarding the post-filter count.
- Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.
@tejaskash
docs(dev): clarify collector/httpServer/flag comments per review
8cabff7 
Harrison review on #1980:
- httpServer: generalize the answer-before-close comment (drop OTLP specificity)
  — the module is a shared io primitive (the Inspector server reuses it).
- flags: explain why a default-true boolean is exposed as --no-<name>.
- collector: reword the onError comment to state the collector's guarantee
  (ack + hand to onError) rather than the caller's report-once behavior; drop
  the volatile "matches the reference CLI" aside.
@tejaskash
fix(dev): harden OTEL collector, container reachability, and dev life…
10be0a6 
…cycle

Address reviewer findings on the collector and dev wiring:

- Validate top-level OTLP shape and return 400 instead of mislabeling a bad
  payload as a persistence error.
- Guard the shared HTTP server against a client that disconnects mid-response
  so it can no longer crash project dev; add an optional bind host.
- Bind the collector to 0.0.0.0 for container runtimes so a container can
  reach it over the host bridge on Linux.
- Run the container template under opentelemetry-instrument so it emits traces.
- Keep the collector alive through the child's shutdown grace so final spans
  are not lost.
- Force the OTEL settings that would otherwise let shell or .env.local values
  disable or break local collection.
- Make the uv sitecustomize discovery abortable and read its path from a
  marker rather than the last merged output line.
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from adf371b to 10be0a6 Compare August 21, 2026 18:55
aidandaly24
aidandaly24 approved these changes Aug 21, 2026
View reviewed changes

@aidandaly24 aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the fixes. This looks good to me!

@tejaskash
tejaskash merged commit f38b18d into refactor Aug 21, 2026
11 checks passed
@tejaskash
tejaskash deleted the feat/dev-otel-collector branch August 21, 2026 21:32
tejaskash added a commit that referenced this pull request Aug 21, 2026
@tejaskash
fix(dev): bind collector to 0.0.0.0 when any runtime is a container
7b4b60b 
The #1980 rebase carried a single-runtime host check (runtime.build) into
the multi-agent dev handler, where the variable is the runtimes array. Bind
all interfaces when any selected runtime runs in a container.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Hweinstock Hweinstock Hweinstock left review comments
@notgitika notgitika notgitika approved these changes
@aidandaly24 aidandaly24 aidandaly24 approved these changes
@jariy17 jariy17 Awaiting requested review from jariy17

Assignees

No one assigned

Labels

size/xl PR size: XL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@tejaskash @codecov-commenter @Hweinstock @notgitika @aidandaly24