From 16307b1cdd55898d1e8784eb72c29b1a11afe635 Mon Sep 17 00:00:00 2001 From: YehoshuaSagron Date: Fri, 14 Aug 2026 21:41:33 +0300 Subject: [PATCH 1/2] Feat: Add the lineage demo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Shows how to get per-request data lineage out of the AuthBridge sidecar: attach the envoy-sidecar to an agent or tool, switch on the lineage-telemetry plugin, and every HTTP exchange becomes two facts-only spans sent to any OTLP consumer. On a stock install that is the platform's own collector, whose default pipeline exports to debug — so the spans are readable from its log with nothing extra deployed. Phoenix is not installed by default; the README shows both routes. Five mechanism files plus a README: attach-lineage.sh the one generator — emits every YAML byte, as a full manifest, the plugin ConfigMap alone, or a sidecar patch Dockerfile.otel-shim the propagate-only shim layer build-otel-shim.sh builds the shim onto an app image, refusing only images where wrapping would stack a second instrumentor on a library the shim already covers sidecar-patch.sh the patch path, for a Deployment you do not own container-runtime.sh docker-vs-podman detection and kind loading The shim is the non-obvious half. The sidecar sees every hop but lives outside the app's execution context, so it cannot know which coroutine issued which outbound call; attributing an outbound call to its causing inbound needs traceparent carried through the app in-process. Without it the plugin falls back to "this agent's current inbound span", correct only at concurrency 1. The README is written around what actually goes wrong, all of it measured: the ownership fork (EMIT=manifest replaces a Deployment; an operator-owned workload must take the additive patch instead), the half-instrumented quadrant (an app with a client instrumentor but no server-side one can neither be shimmed nor propagate on its own — its outbound work scatters into one-interaction traces), that a refusal to bake is not a promise the app propagates, that baking is not purely additive on an image already carrying an SDK, and that a structural cleanliness check cannot detect total attribution failure. The generated workload deliberately carries no /type label: a current platform reserves that label for its operator through a ValidatingAdmissionPolicy and rejects manifests that set it by hand. Omitting it costs platform-inventory registration, which an AgentRuntime CR provides, and costs lineage nothing. Defaults target a stock platform: otel-collector.rossoctl-system, kind cluster rossoctl, and the published ghcr.io/rossoctl/cortex sidecar images — all overridable by environment variable. A published sidecar image carries the lineage-telemetry plugin only once that plugin has merged and a release is cut; until then, build the images from this repo and point SIDECAR_IMAGE / PROXY_INIT_IMAGE at them. Signed-off-by: YehoshuaSagron --- authbridge/demos/lineage/Dockerfile.otel-shim | 63 +++ authbridge/demos/lineage/README.md | 461 ++++++++++++++++++ authbridge/demos/lineage/attach-lineage.sh | 413 ++++++++++++++++ authbridge/demos/lineage/build-otel-shim.sh | 105 ++++ authbridge/demos/lineage/container-runtime.sh | 39 ++ authbridge/demos/lineage/sidecar-patch.sh | 91 ++++ 6 files changed, 1172 insertions(+) create mode 100644 authbridge/demos/lineage/Dockerfile.otel-shim create mode 100644 authbridge/demos/lineage/README.md create mode 100755 authbridge/demos/lineage/attach-lineage.sh create mode 100755 authbridge/demos/lineage/build-otel-shim.sh create mode 100644 authbridge/demos/lineage/container-runtime.sh create mode 100755 authbridge/demos/lineage/sidecar-patch.sh diff --git a/authbridge/demos/lineage/Dockerfile.otel-shim b/authbridge/demos/lineage/Dockerfile.otel-shim new file mode 100644 index 000000000..929b64c77 --- /dev/null +++ b/authbridge/demos/lineage/Dockerfile.otel-shim @@ -0,0 +1,63 @@ +# GENERALIZED deploy-time propagation shim for ANY uninstrumented Python app. +# +# We DO NOT touch the app's source. We layer OpenTelemetry auto-instrumentation +# ON TOP of the user's image so it PROPAGATES traceparent (extract on the inbound +# ASGI/Starlette/FastAPI request, inject on the outbound httpx/requests/aiohttp +# call) — nothing more. Spans are NOT exported (the Deployment passes +# --traces_exporter none), so the shim adds nothing to your telemetry backend; +# only the W3C traceparent header flows to the sidecar, which does the actual +# capture. +# +# The instrumentors target LIBRARIES, not the app. `opentelemetry-instrument` +# auto-activates whichever of these the app actually imports, so the SAME image +# recipe works across every app whose server is ASGI/Starlette/FastAPI and whose +# client is httpx / requests / aiohttp — the mainstream Python A2A/MCP stack. +# +# Build (parameterized — only BASE_IMAGE changes per app): +# podman build -f demos/lineage/Dockerfile.otel-shim \ +# --build-arg BASE_IMAGE=docker.io/library/:latest \ +# -t -otel:latest demos/lineage +# or use demos/lineage/build-otel-shim.sh which also kind-loads it. +# +ARG BASE_IMAGE +FROM ${BASE_IMAGE} + +# Where the app's virtualenv python lives. The common uv-built layout puts it at +# /app/.venv; override for anything else. +ARG VENV_PYTHON=/app/.venv/bin/python +# The non-root UID the app runs as. 1001 is the usual agent-image default, but +# platform images differ (999 is also common) — always confirm with `id` in the +# base image rather than assuming. +ARG APP_UID=1001 + +USER root +# Bring our own uv, pinned. Many app images ship uv on PATH, but an image that +# builds its venv in a throwaway builder stage has NEITHER uv NOR pip in the +# runtime layer (`uv venv` makes a pip-less venv). A copied static uv makes this +# shim work on both without assuming the base carries a Python installer. +# Overwriting an existing /usr/local/bin/uv is harmless. +COPY --from=ghcr.io/astral-sh/uv:0.9.24 /uv /usr/local/bin/uv +# One layer, all mainstream instrumentors. opentelemetry-distro pulls the SDK + +# the `opentelemetry-instrument` launcher; the rest are the server/client +# library instrumentors. starlette+asgi cover the a2a-sdk servers, fastapi+asgi +# the FastAPI tools, and httpx / requests / aiohttp-client the three client +# libraries those apps call out with. +# +# `-threading` is load-bearing for frameworks that run the LLM/tool call in a +# worker thread (ag2/autogen use `loop.run_in_executor`; others use +# ThreadPoolExecutor). OTEL context is contextvars-based and does NOT cross a +# thread boundary by default, so without this the inbound trace is lost before +# the outbound call and traceparent propagation silently breaks (the sidecar +# then collapses all outbound onto one inbound under concurrency — 1/N). The +# threading instrumentor copies the active context across `Thread.start` / +# `ThreadPoolExecutor.submit`, restoring propagation. +RUN uv pip install --python ${VENV_PYTHON} \ + opentelemetry-distro \ + opentelemetry-instrumentation-starlette \ + opentelemetry-instrumentation-asgi \ + opentelemetry-instrumentation-fastapi \ + opentelemetry-instrumentation-httpx \ + opentelemetry-instrumentation-requests \ + opentelemetry-instrumentation-aiohttp-client \ + opentelemetry-instrumentation-threading +USER ${APP_UID} diff --git a/authbridge/demos/lineage/README.md b/authbridge/demos/lineage/README.md new file mode 100644 index 000000000..3383845f7 --- /dev/null +++ b/authbridge/demos/lineage/README.md @@ -0,0 +1,461 @@ +# Lineage — per-request data lineage from the AuthBridge sidecar + +Attach the AuthBridge envoy-sidecar to an agent or tool, switch on the +`lineage-telemetry` plugin, and every HTTP exchange the workload takes part in +becomes **two OTLP spans**: one when the request is seen, one when the response +stream ends. They are facts only — who called whom, over which protocol, with +what outcome — and they go to **any OTLP consumer**: the platform's own +collector, Jaeger, Phoenix, or your own endpoint. + +> **Where you will actually see them.** A stock install runs the OTel collector +> but **not** Phoenix (`components.phoenix.enabled` defaults to `false`), and the +> collector's default pipeline exports to `debug` — so spans arriving there are +> printed to the collector's log and stored nowhere queryable: +> +> ```sh +> kubectl logs -n rossoctl-system deploy/otel-collector | grep lineage.exchange.id +> ``` +> +> That is enough to prove the plugin works, and it is what the worked example +> below relies on. For a UI, either install the platform with Phoenix enabled +> (`--set components.phoenix.enabled=true`, which adds an `otlp/phoenix` +> exporter) or point `OTEL_ENDPOINT` straight at a sink of your own. Nothing in +> this demo depends on which you choose. + +The demo has two halves, and the second is the interesting one: + +1. **The sidecar** captures. That part is just configuration — a plugin entry in + the pipeline (`attach-lineage.sh` writes it for you). +2. **A propagate-only OTel shim** makes the capture *correct under concurrency*. + Without it a busy agent's outbound calls get attributed to the wrong inbound + request. `Dockerfile.otel-shim` + `build-otel-shim.sh` layer it onto an app + image without touching the app's source. + +--- + +## What you get + +Per HTTP exchange, two spans. The request span is named +`{self_id} {protocol} {operation}` and the response span appends ` response` — +so an inbound A2A call on a workload called `weather-lineage` produces +`weather-lineage a2a message/send` and `weather-lineage a2a message/send +response`. The pair is joined by `lineage.exchange.id` (the request span's own +id) and told apart by `lineage.role`: + +| attribute | what it records | +|---|---| +| `lineage.exchange.id` | pairs the two spans of one exchange | +| `lineage.role` | `request` / `response` | +| `lineage.direction` | `inbound` / `outbound` | +| `lineage.self.id` | this workload's own stable id | +| `lineage.peer.host` | the other end of the hop | +| `lineage.protocol` | `a2a` / `mcp` / `inference` / `http` | +| `lineage.principal.sub`, `lineage.principal.client` | the caller's identity, when a validated token carried one | +| `lineage.outcome`, `lineage.denied_by` | how the exchange ended | +| `lineage.parent.source` | how this hop was linked to its parent | + +With `capture_io: true` the request span also carries `input.value` and the +response span `output.value` — the parsed message content, so a trace viewer +shows the actual A2A message, MCP tool arguments or LLM prompt inline. + +Nothing here interprets the traffic. The spans say what happened on the wire; +whatever consumes them decides what it means. + +--- + +## Why the shim is needed (and what breaks without it) + +The sidecar sits at the pod's network boundary. It sees every hop with parsed +bodies — but it lives **outside** the app's execution context and has no way to +know which internal coroutine issued which outbound call. To attribute an +outbound call to the inbound request that caused it, something must carry a +token **with the execution scope through the app**. That is precisely what the +W3C `traceparent` header is for, and only code running *inside* the request's +context can copy it from the inbound request onto the outbound ones. + +When `traceparent` is missing, the plugin falls back to "this agent's current +inbound span". That heuristic is correct only while the agent handles one +request at a time. Under concurrency it collapses: with 6 concurrent requests, +all 6 outbound calls pile onto whichever inbound updated the process-wide +pointer last — **1/6**, measured. So: + +> In-process `traceparent` propagation is mandatory, and the sidecar cannot do +> it from outside the app. + +The shim supplies it with stock OpenTelemetry auto-instrumentation and +**exports nothing**: + +- **server side** (`starlette` / `asgi` / `fastapi`) — extract the inbound + `traceparent`, make it the active context. +- **client side** (`httpx` / `requests` / `aiohttp-client`) — inject + `traceparent` on outbound calls. +- **`threading`** — carry the active context across `Thread.start` / + `ThreadPoolExecutor.submit`. **Load-bearing**: frameworks that run the LLM + call in a worker thread (anything using `loop.run_in_executor`) otherwise lose + the context at the thread boundary and silently fall back to 1/N. +- **`--traces_exporter none`, and no `OTEL_EXPORTER_OTLP_ENDPOINT`** — the shim's + instrumentation generates spans that go nowhere. Your telemetry backend sees + only what the sidecar emits. + +An app that configures its **own** exporter in code keeps exporting exactly as +before; the shim silences only its own auto-instrumentation. Nothing about the +app's telemetry changes. + +--- + +## Which path — decide this before you run anything + +**Two paths ship here, and choosing the wrong one destroys resources.** The +question is not what your app is; it is **who owns the Deployment**. + +| | you own the Deployment | an operator, controller or UI owns it | +|---|---|---| +| **use** | `attach-lineage.sh` (`EMIT=manifest`) | `sidecar-patch.sh` | +| **what it does** | emits a complete ConfigMap + Service + Deployment and you apply it | strategic-merge patch that **adds** the sidecar containers to what is already there | +| **if you get it wrong** | **it replaces the existing Deployment.** Applying a generated manifest over an operator-managed workload overwrites the owner's spec — this has taken production services down | the patch is additive and safe, but its owner keeps owning the object (see below) | + +`EMIT=manifest` is a *replacement*, by design: it is how you deploy an app under +lineage in one step. Point it at a name some other controller manages and you +have silently rewritten that controller's object. + +Two honest limits on the patch path: + +- **A patch is not durable.** The owner still owns the Deployment. Any + platform-side rewrite — an operator reconcile, a chart upgrade, a UI redeploy — + silently drops the sidecar. There is no error; lineage just stops. Re-run + `sidecar-patch.sh` after any platform-side change. +- **Uninstrumented *and* operator-owned is unsolved.** The shim needs an image + change, and you cannot change the image of a workload you do not control. You + can still attach the sidecar and get every hop captured, but pairing under + concurrency will be the 1/N heuristic. Sequential traffic is fine; concurrent + traffic will mis-attribute. + +One more route exists on an operator-managed platform, and it touches no +Deployment at all: when the platform injects its own AuthBridge sidecar (via an +`AgentRuntime` CR), lineage can be enabled for the whole namespace by adding the +three parsers + `lineage-telemetry` to both directions of the namespace's +`authbridge-runtime-config` ConfigMap. Leave `self_id` unset — each pod resolves +its own identity from the operator-mounted credential via `self_id_file`. The +propagation question is unchanged (the app still needs the shim or its own +instrumentation), and a platform upgrade re-renders the namespace ConfigMaps, +reverting the edit — re-apply it after any upgrade. + +--- + +## The envelope + +The shim is generic across the mainstream Python stack, not universal: + +- **Python**, with its virtualenv at `/app/.venv` (override with an argument). +- Server is **ASGI / Starlette / FastAPI**; HTTP client is **httpx**, + **requests** or **aiohttp**. +- The entry caller supplies the first `traceparent`. Nothing here seeds a root + for an untraced entry point. +- **Not covered:** work handed to a `multiprocessing` child or a `subprocess`, + and threads started at app boot rather than inside a request. + +Outside the envelope, use the sidecar alone: every hop is still captured. Whether +those hops are *correctly attributed* depends entirely on the app propagating +`traceparent` itself — and that is a stronger condition than it sounds. + +### Half-instrumented apps: the case that looks fine and is not + +Propagation needs **two** halves: something that extracts the inbound +`traceparent` into an active context (`starlette` / `asgi` / `fastapi`), and +something that injects it on the way out (`httpx` / `requests` / `aiohttp`). An +app carrying only the client half can inject, but has nothing to inject *from*. + +That app is in a corner this demo cannot get you out of: + +- **The shim refuses it**, correctly — its client instrumentor is one this shim + installs, so wrapping would stack a second one on the same library. +- **Sidecar-only does not save it** — with no server-side extraction, every + outbound call starts a *fresh* trace. + +Measured on exactly such an app: it served requests, called its LLM +successfully, and produced 26 outbound interactions — **every one of them alone +in its own trace, none sharing a trace with the inbound that caused it.** Not the +1/N collapse described above, which at least attaches outbound work to *some* +inbound. Here no correlation survives at all. + +**Check the shape, not just the count.** A concurrency check that only asks "is +each trace internally consistent?" will pass this app perfectly: a trace holding +one lonely inbound is clean by every structural measure. What exposes it is +expecting a *number of hops* per trace and finding one. If your app should make +three outbound calls per request, assert that three appear in the same trace — +otherwise a total attribution failure reads as a clean run. + +If you own the image, the fix is to add the missing server-side instrumentor (or +run the app under `opentelemetry-instrument`, which activates both halves). If +you do not, this is the same unsolved corner as the operator-owned quadrant. + +**Baking is not purely additive.** The shim installs its instrumentors with +`uv pip install`, which resolves the whole OpenTelemetry set upward — on an image +that already carries an SDK, the bake *upgrades* it (observed: +`opentelemetry-sdk 1.42.1 → 1.44.0`, semconv `0.63b1 → 0.65b0`). Harmless for an +app that does not pin, but an app pinning an older SDK can be affected by being +wrapped. Check the build output if your app is sensitive to those versions. + +`build-otel-shim.sh` refuses rather than guesses. It probes the image and stops +if there is no runnable Python at the venv path, or if +`opentelemetry.instrumentation` is already importable (an app that instruments +itself — wrapping it again would double-instrument). `FORCE_BAKE=1` overrides, +which makes the human judgment explicit and greppable. + +--- + +## Files + +| file | what it is | +|---|---| +| `attach-lineage.sh` | **The one generator.** Emits every YAML byte: `EMIT=manifest` (ConfigMap + Service + Deployment), `EMIT=cm` (the plugin ConfigMap alone), `EMIT=patch` (sidecar pieces for an existing Deployment). Env-driven; writes to stdout. | +| `Dockerfile.otel-shim` | The propagate-only shim layer. One recipe for every in-envelope app; only `BASE_IMAGE` changes. | +| `build-otel-shim.sh` | Builds the shim onto an app image, loads it into kind, and refuses images it cannot safely wrap. **The baked image is not self-activating**: it installs the instrumentors, and the `opentelemetry-instrument` launcher is supplied by the generated Deployment's `command:`. Deploy the `-otel` image with an unwrapped command and you get the worst outcome — every exchange alone in its own trace (measured: 268 spans / 134 traces from one turn). | +| `sidecar-patch.sh` | The patch path — attaches the sidecar to a Deployment you do not own. Generates its YAML from `attach-lineage.sh`. | +| `container-runtime.sh` | Sourced helper: picks docker vs podman and loads images into kind either way. | + +--- + +## Prerequisites + +- A Kubernetes cluster with the platform installed, and the platform-rendered + **`envoy-config` ConfigMap** present in your target namespace — the sidecar + mounts it. Both paths check for it. +- The sidecar images resolvable from the cluster. Defaults are the published + ones: + + ``` + SIDECAR_IMAGE=ghcr.io/rossoctl/cortex/authbridge-envoy:latest + PROXY_INIT_IMAGE=ghcr.io/rossoctl/cortex/proxy-init:latest + ``` + + > **Caveat.** A published image carries the `lineage-telemetry` plugin only + > from the release that first contains it. Until then, build the two images + > from this repo (`authbridge/cmd/authbridge-envoy/Dockerfile` and + > `authbridge/proxy-init/Dockerfile.init`), load them into your cluster, and + > point `SIDECAR_IMAGE` / `PROXY_INIT_IMAGE` at your local tags. + +- An **OTLP/gRPC endpoint**. Default is + `otel-collector.rossoctl-system.svc.cluster.local:4317`. Point `OTEL_ENDPOINT` + anywhere else — Phoenix, Jaeger, your own collector. Nothing downstream of it + is assumed. + +--- + +## Worked example — the weather agent + +Using this repo's own weather agent image, measured as of writing: + +```console +$ podman inspect --format '{{json .Config.Cmd}}' ghcr.io/rossoctl/examples/weather_service:latest +["uv","run","--no-sync","server"] +``` + +It is a Python app with its venv at `/app/.venv`, running as UID 1001 — inside +the envelope. It also ships the **httpx** instrumentor already, which is one of +the seven this shim installs, so wrapping it would stack a second instrumentor +on the same library. The interlock says so and stops: + +```console +$ ./build-otel-shim.sh ghcr.io/rossoctl/examples/weather_service:latest +REFUSING to bake ...: it already instruments httpx +``` + +Note what the check is asking: *would wrapping double-instrument?* Not "does this +image contain anything OpenTelemetry-shaped". The base +`opentelemetry-instrumentation` package arrives as a transitive dependency of +plenty of things and carries no library instrumentation at all — an app holding +only that still needs the shim, and refusing it would cost you propagation for +no reason. + +**Read that refusal precisely: it means "do not wrap this", not "this one is +fine".** The probe detects that `opentelemetry.instrumentation` is *importable*. +Whether the app actually *activates* it — and therefore whether it propagates +`traceparent` on its outbound calls — is not statically detectable, because +activation happens at runtime through `opentelemetry-instrument` or in the app's +own code. An image can carry the packages as dormant transitive dependencies and +propagate nothing. + +So the recipe for such an app is sidecar-only, with `NO_PROPAGATE=1` to run its +own command untouched — and then **verify pairing before trusting it**. Drive +several concurrent requests, each with a distinct `traceparent`, and check that +every outbound hop lands under the inbound that caused it. If they collapse onto +one inbound, the app is not propagating, and you are on the 1/N heuristic +described above with no way to fix it from outside the process. That is the same +unsolved corner as the operator-owned quadrant, reached from a different +direction. + +```sh +cd authbridge/demos/lineage + +NAME=weather-lineage \ +IMAGE=ghcr.io/rossoctl/examples/weather_service:latest \ +SELF_ID=weather-lineage \ +APP_PORT=8000 SVC_PORT=8080 \ +APP_ENTRYPOINT='uv run --no-sync server' \ +NO_PROPAGATE=1 \ +NAMESPACE=team1 \ +ENV_VARS='MCP_URL=http://weather-tool-advanced-mcp:8000/mcp LLM_API_BASE=http://host.docker.internal:11434/v1 LLM_API_KEY=ollama LLM_MODEL=llama3.2:3b-instruct-fp16' \ +OTEL_ENDPOINT=otel-collector.rossoctl-system.svc.cluster.local:4317 \ +./attach-lineage.sh | kubectl apply -f - +``` + +Read the manifest before you apply it — drop the `| kubectl apply -f -` and the +script just prints. Then drive one A2A request from **inside** the cluster (a +`kubectl port-forward` reaches the app on loopback and bypasses the sidecar's +inbound listener, so it captures nothing — and an agent-card fetch is bypassed +too, since `/.well-known/` is on the plugin's default `bypass_paths`): + +```sh +kubectl run -n team1 --rm -i drive --image=curlimages/curl --restart=Never -- \ + curl -s -X POST http://weather-lineage:8080/ \ + -H 'content-type: application/json' \ + -d '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{"message": + {"role":"user","messageId":"m1", + "parts":[{"kind":"text","text":"what is the weather in Haifa?"}]}}}' +``` + +Two spans for that exchange now exist, in one trace. On a stock install they +land in the collector's `debug` exporter, so: + +```sh +kubectl logs -n rossoctl-system deploy/otel-collector | grep -A2 lineage.exchange.id +``` + +This is the actual result of the run above: + +``` +weather-lineage a2a message/send role=request direction=inbound protocol=a2a +weather-lineage a2a message/send response role=response direction=inbound protocol=a2a outcome=ok +``` + +both carrying the same `lineage.exchange.id`. That pair is the whole claim: the +sidecar saw an exchange and recorded it as facts. + +**The agent's own outbound hops are a separate matter.** The `MCP_URL` above +points at the weather tool from this repo's +[weather-agent advanced demo](../weather-agent/demo-ui-advanced.md) — if you have +not deployed it, the agent answers with a tool-connection error and you see only +the inbound pair, because a connection that never completes is not an exchange +to record. Deploy that demo's tool first and the same request adds a hop pair for +the MCP call and another for the LLM, linked to the inbound one through the +`traceparent` the app propagated. + +For an app that does **not** instrument itself, the same flow gains one step — +bake the shim first, deploy the `-otel` image, and drop `NO_PROPAGATE=1`: + +```sh +./build-otel-shim.sh :latest # -> -otel:latest, kind-loaded +NAME=... IMAGE=docker.io/library/-otel:latest ... ./attach-lineage.sh | kubectl apply -f - +``` + +### The other path: adopt a Deployment you do not own + +```sh +DEPLOY=weather-service-advanced NAMESPACE=team1 \ +OUTBOUND_PORTS_EXCLUDE=8335 \ +./sidecar-patch.sh +``` + +This adds the sidecar and leaves the app container exactly as its owner wrote +it. `OUTBOUND_PORTS_EXCLUDE` keeps a port out of the iptables redirect — use it +for a port the app exports its *own* telemetry on, so that keeps flowing +untouched. Do **not** exclude LLM or tool ports; those are the hops lineage +exists to observe. + +Re-read the two limits above before you rely on this path. + +--- + +## Configuration + +`attach-lineage.sh` writes the plugin entry into the generated ConfigMap. Three +keys matter: + +```yaml +- name: lineage-telemetry + config: + otel_endpoint: "otel-collector.rossoctl-system.svc.cluster.local:4317" + capture_io: true + self_id: "weather-lineage" +``` + +| key | meaning | +|---|---| +| `otel_endpoint` | OTLP/gRPC endpoint. `http://` and `https://` prefixes are stripped. | +| `capture_io` | attach parsed request/response content as `input.value` / `output.value`. **Off by default** — enable only where the traffic carries no PII, or the backend enforces access control. | +| `self_id` | this workload's stable identity, emitted as `lineage.self.id`. Falls back to `self_id_file` (default `/shared/client-id.txt`, the operator-mounted credential). | + +`bypass_paths` and `bypass_hosts` keep infrastructure noise out of the graph; +their defaults already cover agent-card discovery, health probes and the common +telemetry backends. + +Script-level knobs (both paths): `NAME`, `IMAGE`, `SELF_ID`, `APP_PORT`, +`SVC_PORT`, `APP_ENTRYPOINT`, `ENV_VARS`, `NAMESPACE`, `OTEL_ENDPOINT`, +`OUTBOUND_PORTS_EXCLUDE`, `WORKLOAD_TYPE`, `LABEL_PREFIX`, `SIDECAR_IMAGE`, +`PROXY_INIT_IMAGE`, `NO_PROPAGATE`, `NO_EMIT`, `EMIT`. Each script's header +documents its own; `NO_PROPAGATE=1 NO_EMIT=1` together is lineage fully off for +one app, which makes a clean A/B baseline. + +`LABEL_PREFIX` (default `rossoctl.io`) sets the platform labels the generated +workload carries — `protocol./a2a`, and the `/inject: disabled` +opt-out that keeps the platform from injecting a second sidecar next to the one +this manifest already brings. Retarget it for a differently-branded platform. + +### Why the generated workload has no `/type` label + +Because the platform will not let you set one, and it is right not to. + +A `ValidatingAdmissionPolicy` named `agent-label-protection` reserves +`/type` for the operator, on `deployments` and `statefulsets`, with +`validationActions: [Deny]`. Apply a manifest carrying that label by hand and +admission rejects it outright: + +``` +deployments.apps "…" is forbidden: ValidatingAdmissionPolicy +'agent-label-protection' denied request: The rossoctl.io/type label … can only +be applied by the rossoctl-operator via an AgentRuntime CR. +``` + +So this demo does not set it. Nothing is lost for lineage: the label is how the +platform *registers and classifies* a workload, not how this manifest attaches +its sidecar — the sidecar is in the manifest already, as an explicit +`proxy-init` initContainer and `envoy-proxy` container. Service and Deployment +selectors key on `app.kubernetes.io/name`, which is unique per workload, so +routing is unaffected. + +What you give up is the workload appearing in the platform's agent inventory. +If you want that, the supported route is to create an **AgentRuntime CR** +targeting the workload and let the operator apply the label — which is exactly +what the policy's message tells you. + +`WORKLOAD_TYPE` exists for platforms that do *not* guard the label: set it to +`agent` or `tool` and the label is emitted. It is empty by default, deliberately, +because the default has to work on a stock install. + +--- + +## Limits + +- **The sidecar sees plaintext HTTP only.** An HTTPS destination is TLS + passthrough — no hop is recorded for it. +- **No producer-side payload size cap.** With `capture_io: true` a large message + is attached whole. +- **Trace-context propagation is the app's job** — the shim does it for the + in-envelope stack, and nothing else can do it from outside the process. +- **The patch path is not durable** and the uninstrumented + operator-owned + quadrant is unsolved. Both are spelled out above; they are the two things most + likely to surprise you in production. + +--- + +## Troubleshooting + +| symptom | cause | +|---|---| +| No spans at all | Wrong `OTEL_ENDPOINT`, or the sidecar image predates the plugin. Check the `envoy-proxy` container's logs. | +| Only inbound hops, never outbound | `proxy-init` did not install its iptables rules — check the init container's logs. | +| Every outbound piles onto one inbound (**1/N**) | `traceparent` is not propagating. If the app runs the LLM/tool call in a worker thread, the `threading` instrumentor is required — it is bundled in `Dockerfile.otel-shim`. If it still collapses, the caller is not sending a `traceparent` at all. | +| Nothing captured when testing | You drove the app through `kubectl port-forward`. Loopback bypasses the inbound listener; drive it from inside the cluster. | +| `kind load` fails under podman | `container-runtime.sh` detects the runtime and saves + loads an image archive for podman, because `kind load docker-image` misbehaves under podman v5. Set `CONTAINER_TOOL` to force a runtime, `KIND_CLUSTER_NAME` for the cluster name. | +| Pod stuck `ImagePullBackOff` on the sidecar | `SIDECAR_IMAGE` / `PROXY_INIT_IMAGE` point at tags the cluster cannot resolve. Load them locally, or use the published refs. | diff --git a/authbridge/demos/lineage/attach-lineage.sh b/authbridge/demos/lineage/attach-lineage.sh new file mode 100755 index 000000000..a1aaef501 --- /dev/null +++ b/authbridge/demos/lineage/attach-lineage.sh @@ -0,0 +1,413 @@ +#!/usr/bin/env bash +# GENERALIZED lineage-sidecar attachment. Emits (to stdout) a complete manifest — +# lineage ConfigMap + Service + Deployment — that runs ANY app image with: +# (a) the AuthBridge lineage sidecar (proxy-init initContainer + envoy-proxy sidecar, +# AuthBridge envoy-sidecar mode, capture_io:true, no auth/SPIRE), and +# (b) the propagate-only OTEL shim launcher wrapping the app command +# (--traces_exporter none: export nothing, only propagate traceparent). +# +# Per app, only a handful of variables change (image + self_id + the app's own +# entrypoint + LLM env). +# +# Usage (pipe to kubectl apply): +# NAME=a2a-currency-converter \ +# IMAGE=docker.io/library/a2a_currency_converter-otel:latest \ +# APP_PORT=8000 SVC_PORT=8080 \ +# APP_ENTRYPOINT='app --host 0.0.0.0 --port 8000' \ +# ENV_VARS='LLM_API_BASE=http://host.containers.internal:11434/v1 LLM_MODEL=qwen2.5:7b LLM_API_KEY=ollama' \ +# demos/lineage/attach-lineage.sh | kubectl apply -f - +# +# Variables: +# NAME (required) k8s resource name + app.kubernetes.io/name label. +# IMAGE (required for EMIT=manifest) the -otel wrapper image +# (build-otel-shim.sh output). +# SELF_ID lineage self_id (default: NAME). Only this varies in the config. +# APP_PORT app container port (default 8000). +# SVC_PORT service port (default 8080). +# APP_ENTRYPOINT the app's OWN command tokens (default 'server'). +# Check the app image's Dockerfile CMD, +# dropping any leading `uv run --no-sync`. +# ENV_VARS space-separated KEY=VALUE app env (LLM_* etc). Values must +# not contain spaces (fine for our URLs/models). May include +# OTEL_SERVICE_NAME to override the default (SELF_ID). +# OUTBOUND_PORTS_EXCLUDE iptables outbound excludes (default ''). Set to an +# app's OWN OTLP export port (e.g. 8335/4318) for an app that +# already exports spans, so that export keeps flowing +# untouched. Do NOT exclude LLM/tool ports — we want those seen. +# NAMESPACE (default team1, the platform's demo namespace). +# OTEL_ENDPOINT OTLP/gRPC endpoint the lineage plugin exports to (default +# otel-collector.rossoctl-system.svc.cluster.local:4317). +# Any OTLP consumer works — Phoenix and Jaeger included; +# nothing downstream of this endpoint is assumed. +# WORKLOAD_TYPE agent | tool. **Empty by default, and leaving it empty +# is the right choice on a current platform**, which +# forbids setting /type by hand: a +# ValidatingAdmissionPolicy reserves that label for the +# operator, so a manifest carrying it is REJECTED at +# admission. Omitting it costs only platform-UI +# registration (use an AgentRuntime CR for that); it costs +# nothing for lineage. Set it only on a platform you know +# does not guard the label. +# WORKLOAD_PROTOCOL protocol label value (default a2a). Set mcp for MCP +# servers — the label is a factual claim a platform UI +# reads — or empty to omit it. Presentational only. +# LABEL_PREFIX platform label domain (default rossoctl.io). Sets +# protocol./, the +# /inject: disabled opt-out, and /type +# when WORKLOAD_TYPE is set. Retarget it for a +# differently-branded platform. +# SIDECAR_IMAGE envoy+authbridge sidecar image (default +# ghcr.io/rossoctl/cortex/authbridge-envoy:latest). +# PROXY_INIT_IMAGE iptables init image (default +# ghcr.io/rossoctl/cortex/proxy-init:latest). +# Point both at locally-built images when you build from +# source — the published images carry the lineage plugin +# only from the release that first contains it. +# NO_PROPAGATE if set to 1, run the app command bare (no opentelemetry- +# instrument wrapper): trace context stops flowing THROUGH +# the app — and nothing more (the sidecar still emits). +# For a baseline/uninstrumented run. (Alias: NO_OTEL.) +# NO_EMIT if set to 1, omit the lineage-telemetry plugin from the +# generated ConfigMap: the sidecar emits zero spans — and +# nothing more (it still proxies; parsers stay — legal +# alone; the plugin declares RequiresAny{parsers}, not the +# reverse). NO_PROPAGATE=1 NO_EMIT=1 together = lineage +# fully off for this app. (Alias: NO_LINEAGE.) +# EMIT what to emit (this script is the ONE source of every +# lineage YAML byte): +# manifest (default) ConfigMap + Service + Deployment +# cm the per-app lineage ConfigMap alone +# patch a strategic-merge patch adding the sidecar +# pieces to an EXISTING Deployment (used by +# sidecar-patch.sh; app container untouched) +set -euo pipefail + +EMIT="${EMIT:-manifest}" +NAME="${NAME:?set NAME}" +case "$EMIT" in + manifest) IMAGE="${IMAGE:?set IMAGE}" ;; + cm|patch) IMAGE="${IMAGE:-}" ;; + *) echo "error: EMIT must be manifest|cm|patch (got '$EMIT')" >&2; exit 2 ;; +esac +SELF_ID="${SELF_ID:-$NAME}" +APP_PORT="${APP_PORT:-8000}" +SVC_PORT="${SVC_PORT:-8080}" +APP_ENTRYPOINT="${APP_ENTRYPOINT:-server}" +ENV_VARS="${ENV_VARS:-}" +OUTBOUND_PORTS_EXCLUDE="${OUTBOUND_PORTS_EXCLUDE:-}" +PVC_NAME="${PVC_NAME:-}" +PVC_MOUNT="${PVC_MOUNT:-}" +NAMESPACE="${NAMESPACE:-team1}" +OTEL_ENDPOINT="${OTEL_ENDPOINT:-otel-collector.rossoctl-system.svc.cluster.local:4317}" +LABEL_PREFIX="${LABEL_PREFIX:-rossoctl.io}" +WORKLOAD_TYPE="${WORKLOAD_TYPE:-}" +# Emitted only when the caller asks for it — see the header. Selectors key on +# app.kubernetes.io/name alone, which is already unique per workload, so the +# label is presentational and its absence changes nothing functional. +type_label="" +type_label_8="" +if [ -n "$WORKLOAD_TYPE" ]; then + type_label=" + ${LABEL_PREFIX}/type: ${WORKLOAD_TYPE}" + type_label_8=" + ${LABEL_PREFIX}/type: ${WORKLOAD_TYPE}" +fi +# Protocol label: a factual claim a platform UI reads, so it must be true — +# an MCP tool must not advertise protocol./a2a. Defaults to a2a for +# the common case; set WORKLOAD_PROTOCOL=mcp for MCP servers, or empty to +# omit the label entirely. Presentational only, like the type label. +WORKLOAD_PROTOCOL="${WORKLOAD_PROTOCOL:-a2a}" +proto_label="" +proto_label_8="" +if [ -n "$WORKLOAD_PROTOCOL" ]; then + proto_label=" + protocol.${LABEL_PREFIX}/${WORKLOAD_PROTOCOL}: \"\"" + proto_label_8=" + protocol.${LABEL_PREFIX}/${WORKLOAD_PROTOCOL}: \"\"" +fi +# Published images by default so the demo runs against a stock platform; point +# these at locally-built tags when you build the sidecar from source. +SIDECAR_IMAGE="${SIDECAR_IMAGE:-ghcr.io/rossoctl/cortex/authbridge-envoy:latest}" +PROXY_INIT_IMAGE="${PROXY_INIT_IMAGE:-ghcr.io/rossoctl/cortex/proxy-init:latest}" +# Each toggle kills exactly one layer and nothing more; the old names are +# accepted as aliases (new name wins if both are set). +NO_PROPAGATE="${NO_PROPAGATE:-${NO_OTEL:-0}}" +NO_EMIT="${NO_EMIT:-${NO_LINEAGE:-0}}" + +# ---- build the container command array (YAML flow sequence) ---- +otel_prefix='"uv","run","--no-sync","opentelemetry-instrument","--traces_exporter","none","--metrics_exporter","none","--logs_exporter","none"' +app_tokens="" +for tok in $APP_ENTRYPOINT; do app_tokens="${app_tokens}\"${tok}\","; done +app_tokens="${app_tokens%,}" +if [ "$NO_PROPAGATE" = "1" ]; then + CMD_ARRAY="[${app_tokens}]" +else + CMD_ARRAY="[${otel_prefix},${app_tokens}]" +fi + +# ---- build env block ---- +# Exporter suppression lives ONLY in the command wrapper's --*_exporter none +# flags (the launcher writes its flags over the environment, so an env copy is +# dead weight — and under NO_PROPAGATE=1 it could clobber an instrumented app's own +# exporter config). Env carries only what the wrapper actually reads and the +# flags don't cover: propagators + service name. +env_block=$(cat < mcp > inference). The parsers are NOT + # mutually exclusive — mcp-parser attaches to any JSON-RPC body, so on + # every a2a exchange both extensions are populated; precedence picks the + # label and the protocol-keyed payload read keeps the wrong parser's + # output from ever landing on a span. Non-matching traffic falls through + # untouched as plain http. + pipeline: + inbound: + plugins: + - name: a2a-parser + - name: mcp-parser + - name: inference-parser${lineage_plugin} + outbound: + plugins: + - name: a2a-parser + - name: mcp-parser + - name: inference-parser${lineage_plugin} +EOF +} + +emit_service() { + cat < [wrapper-tag] [venv-python] [app-uid] +# +# Examples: +# # app image already loaded as docker.io/library/a2a_currency_converter:latest +# ./build-otel-shim.sh a2a_currency_converter:latest +# # -> builds+loads docker.io/library/a2a_currency_converter-otel:latest +# +# Runtime-agnostic: container-runtime.sh picks docker or podman (override with +# CONTAINER_TOOL) and kind_load does the right load per runtime. BRACE +# "${x}:latest" — zsh applies a :l modifier to $x:latest (this script is bash +# so it's fine, but keep the habit). +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +. "${SCRIPT_DIR}/container-runtime.sh" + +BASE_IMAGE="${1:?usage: build-otel-shim.sh [wrapper-tag] [venv-python] [app-uid]}" +# derive default wrapper tag: strip registry/path + tag, append -otel +base_short="${BASE_IMAGE##*/}"; base_name="${base_short%%:*}" +WRAPPER_TAG="${2:-${base_name}-otel:latest}" +VENV_PYTHON="${3:-/app/.venv/bin/python}" +APP_UID="${4:-1001}" + +# Normalize the base image to the docker.io/library/ name kind resolves against, +# unless a registry was already given. +case "$BASE_IMAGE" in + */*) base_ref="$BASE_IMAGE" ;; + *) base_ref="docker.io/library/${BASE_IMAGE}" ;; +esac + +# ---- refuse-to-bake interlock ---- +# Baking the shim is safe ONLY for an in-envelope image that is not already +# auto-instrumented. That judgment must not depend on a human getting a config +# entry right, so probe the image itself — sufficient, because no entrypoint +# can activate packages the image lacks. +# +# What the probes can and cannot decide (measured across a mixed app set): +# - a runnable venv python IS the envelope test — a self-instrumenting app +# correctly lands in sidecar-only here. +# - the refusal test is "would wrapping DOUBLE-instrument?", so it asks +# exactly that: is any of the instrumentors this shim installs already +# present? Testing for the `opentelemetry.instrumentation` package instead +# is far too broad — that namespace exists whenever the *base* +# `opentelemetry-instrumentation` package is installed, which arrives as a +# transitive dependency of anything OTel-adjacent and brings no library +# instrumentation with it. An app carrying only the base package needs the +# shim and must not be refused. +# - bare SDK/exporter presence is NOT a refusal signal: a2a-sdk ships both +# as dormant transitive deps on every stock agent, and the wrap is proven +# safe on them (exporters stay off; the shim never sets an endpoint). +# An in-envelope app that ACTIVATES its SDK in code is not statically +# detectable; deciding that needs a runtime probe, which this does not do. +# FORCE_BAKE=1 overrides, turning the implicit human judgment into an +# explicit, greppable one. +FORCE_BAKE="${FORCE_BAKE:-0}" +if [ "$FORCE_BAKE" != "1" ]; then + if ! probe_out=$("$CONTAINER_TOOL" run --rm --entrypoint "$VENV_PYTHON" "$base_ref" -c 'import sys' 2>&1); then + echo "REFUSING to bake ${base_ref}: no runnable Python at ${VENV_PYTHON}." >&2 + echo " The image is outside the shim envelope (non-Python, or a different venv path)." >&2 + echo " -> pass the right venv-python arg, or attach the sidecar only:" >&2 + echo " DEPLOY= ./sidecar-patch.sh (still captures every HTTP hop," >&2 + echo " but pairing under concurrency needs the app to propagate on its own)" >&2 + echo " -> FORCE_BAKE=1 overrides. Probe said: ${probe_out}" >&2 + exit 3 + fi + # The seven this shim installs; presence of ANY means a wrap would stack a + # second instrumentor on the same library. + if already=$("$CONTAINER_TOOL" run --rm --entrypoint "$VENV_PYTHON" "$base_ref" -c ' +import importlib.util as u +mods = ["starlette", "asgi", "fastapi", "httpx", "requests", "aiohttp_client", "threading"] +found = [m for m in mods if u.find_spec("opentelemetry.instrumentation." + m)] +print(",".join(found)) +raise SystemExit(0 if found else 1)' 2>/dev/null); then + echo "REFUSING to bake ${base_ref}: it already instruments ${already}" >&2 + echo " Those are instrumentors this shim installs, so wrapping would stack a" >&2 + echo " second one on the same library (an -otel image, or an app that bundles" >&2 + echo " its own instrumentation)." >&2 + echo " -> if this is a stock app image, point me at the un-shimmed base." >&2 + echo " -> FORCE_BAKE=1 overrides if you know the wrap is safe." >&2 + exit 3 + fi +fi + +echo ">> building shim ${WRAPPER_TAG} FROM ${base_ref} (${CONTAINER_TOOL})" +"$CONTAINER_TOOL" build -f "${SCRIPT_DIR}/Dockerfile.otel-shim" \ + --build-arg "BASE_IMAGE=${base_ref}" \ + --build-arg "VENV_PYTHON=${VENV_PYTHON}" \ + --build-arg "APP_UID=${APP_UID}" \ + -t "${WRAPPER_TAG}" "${SCRIPT_DIR}" + +# alias under docker.io/library so containerd resolves the bare name in manifests +wrapper_name="${WRAPPER_TAG%%:*}"; wrapper_ver="${WRAPPER_TAG##*:}" +alias_ref="docker.io/library/${wrapper_name}:${wrapper_ver}" +"$CONTAINER_TOOL" tag "${WRAPPER_TAG}" "${alias_ref}" + +kind_load "${alias_ref}" +echo ">> loaded ${alias_ref} into kind cluster ${KIND_CLUSTER_NAME}" +echo ">> NOTE: the -otel image INSTALLS the instrumentors but does not activate them —" +echo ">> its CMD is the app's own. The opentelemetry-instrument launcher comes from" +echo ">> the generated Deployment (attach-lineage.sh writes command:). Deploy this" +echo ">> image with an unwrapped command and every exchange lands in its own trace." diff --git a/authbridge/demos/lineage/container-runtime.sh b/authbridge/demos/lineage/container-runtime.sh new file mode 100644 index 000000000..31abec509 --- /dev/null +++ b/authbridge/demos/lineage/container-runtime.sh @@ -0,0 +1,39 @@ +# shellcheck shell=bash +# container-runtime.sh — shared container-runtime detection + kind image loading. +# Sourced (not executed), so it carries no shebang: build-otel-shim.sh sources it, +# and so may any script that needs to put an image into the kind cluster. +# +# CONTAINER_TOOL: env override -> podman if on PATH -> docker. Podman is checked +# FIRST deliberately: on podman hosts a `docker` CLI is often a compat client to +# the podman socket (docker info reports the podman version), and picking it +# would route `kind load docker-image` at a podman-provider cluster — the exact +# breakage this helper exists to avoid. Docker-only hosts (e.g. WSL2) fall +# through to docker. +# +# kind_load : +# docker -> `kind load docker-image` (the direct path; works on Linux/WSL2). +# podman -> save + `kind load image-archive` under KIND_EXPERIMENTAL_PROVIDER=podman +# (the docker daemon is off and `kind load docker-image` misbehaves +# under podman v5 — see README.md "Troubleshooting"). +# +# KIND_CLUSTER_NAME: target kind cluster name (default rossoctl). + +CONTAINER_TOOL="${CONTAINER_TOOL:-}" +if [ -z "$CONTAINER_TOOL" ]; then + if command -v podman >/dev/null 2>&1; then CONTAINER_TOOL=podman + elif command -v docker >/dev/null 2>&1; then CONTAINER_TOOL=docker + else echo "error: neither docker nor podman on PATH (set CONTAINER_TOOL)" >&2; exit 1; fi +fi +KIND_CLUSTER_NAME="${KIND_CLUSTER_NAME:-rossoctl}" + +kind_load() { + local ref="$1" + if [ "$CONTAINER_TOOL" = "podman" ]; then + local base="${ref##*/}" + local tar="/tmp/${base//:/-}.tar" + podman save -o "$tar" "$ref" + KIND_EXPERIMENTAL_PROVIDER=podman kind load image-archive "$tar" --name "$KIND_CLUSTER_NAME" + else + kind load docker-image "$ref" --name "$KIND_CLUSTER_NAME" + fi +} diff --git a/authbridge/demos/lineage/sidecar-patch.sh b/authbridge/demos/lineage/sidecar-patch.sh new file mode 100755 index 000000000..62352e96d --- /dev/null +++ b/authbridge/demos/lineage/sidecar-patch.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# sidecar-patch.sh — attach the lineage sidecar to an EXISTING Deployment. +# +# attach-lineage.sh owns the deploy-it-yourself path: it EMITS a complete +# Deployment (app re-wrapped with the OTel shim + sidecar). This script is the +# complement for apps you do NOT deploy yourself — operator-managed or +# UI-imported workloads — where the Deployment already exists and must keep its +# owner's spec. It only ADDS the sidecar pieces via a strategic-merge +# patch (lists merge by name: the app container is untouched; interception is +# transparent iptables — no HTTP_PROXY, no code change, no image change). +# +# Every YAML byte comes from attach-lineage.sh, the ONE generator (EMIT=cm for +# the per-app plugin ConfigMap, EMIT=patch for the sidecar patch) — this script +# only applies them and waits. CAVEAT the owner keeps owning the object: a +# platform rewrite of the Deployment silently drops the patch (observed live +# when an operator reconciled a patched Deployment) — re-run this script after +# any platform-side change. +# +# NOTE: natively-instrumented apps need no shim — in-process context already +# propagates. Apps that are NOT instrumented still need the shim for correct +# pairing under concurrency; for those, use attach-lineage.sh (EMIT=manifest), +# which deploys the app re-wrapped with the shim. See README.md "Which path". +# +# Usage (env-driven, like attach-lineage.sh): +# DEPLOY=weather-service OUTBOUND_PORTS_EXCLUDE=8335 ./sidecar-patch.sh +# +# Env: +# DEPLOY target Deployment name (required) +# NAMESPACE default team1 +# SELF_ID lineage self_id (default: $DEPLOY) +# OTEL_ENDPOINT OTLP/gRPC endpoint the lineage plugin exports to +# (default otel-collector.rossoctl-system.svc.cluster.local:4317). +# SIDECAR_IMAGE sidecar image override, passed through to +# PROXY_INIT_IMAGE attach-lineage.sh (see its header for defaults). +# OUTBOUND_PORTS_EXCLUDE comma-separated ports proxy-init must NOT intercept. +# An app that exports its own OTLP telemetry should +# have that export port excluded, so its telemetry +# keeps flowing untouched. LLM / MCP ports are +# deliberately NOT excluded — those are the hops +# lineage exists to observe. +# +# Requires in the target namespace: the platform-rendered `envoy-config` +# ConfigMap, and the sidecar + proxy-init images resolvable from the cluster +# (see README.md "Prerequisites"). +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" + +DEPLOY="${DEPLOY:?usage: DEPLOY= [NAMESPACE=team1] [SELF_ID=] [OUTBOUND_PORTS_EXCLUDE=ports] sidecar-patch.sh}" +NAMESPACE="${NAMESPACE:-team1}" +SELF_ID="${SELF_ID:-$DEPLOY}" + +kubectl get deploy -n "$NAMESPACE" "$DEPLOY" >/dev/null +kubectl get cm -n "$NAMESPACE" envoy-config >/dev/null || { + echo "error: ConfigMap envoy-config missing in $NAMESPACE (rendered by the platform chart)" >&2 + exit 1 +} +# Refuse a target that already carries an injected authbridge sidecar: the +# strategic-merge patch would add a SECOND container binding 15123/15124/9090 +# and the pod would crash-loop with nothing pointing back here. Detected by +# the sidecar's ports rather than a container name, which the operator owns. +if kubectl get deploy -n "$NAMESPACE" "$DEPLOY" -o json | grep -q '"containerPort": *15124'; then + echo "error: $DEPLOY already has a sidecar bound on 15124 (operator-injected?) — adopt would add a second one" >&2 + exit 1 +fi + +# Adopt targets are natively-instrumented apps (no shim) — they only propagate +# trace context when their own OTel SDK is configured. A Deployment without +# OTEL_EXPORTER_OTLP_ENDPOINT (seen with UI-"Deploy From Image" imports, which +# drop the example manifests' env) records exchanges that scatter into orphan +# traces: sidecar attaches fine, forests never link. Warn at attach time. +if ! kubectl get deploy -n "$NAMESPACE" "$DEPLOY" \ + -o jsonpath='{.spec.template.spec.containers[*].env[*].name}' \ + | grep -q OTEL_EXPORTER_OTLP_ENDPOINT; then + echo "WARNING: deploy/$DEPLOY has no OTEL_EXPORTER_OTLP_ENDPOINT env." >&2 + echo " Its app will not propagate trace context; its exchanges will land in" >&2 + echo " orphan traces instead of the caller's tree. If the app is OTel-capable:" >&2 + echo " kubectl set env -n $NAMESPACE deploy/$DEPLOY \\" >&2 + echo " OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector..svc.cluster.local:8335" >&2 + echo " (If it is NOT OTel-instrumented, use attach-lineage.sh — it adds the shim.)" >&2 +fi + +gen() { # $1 = EMIT mode; forwards the shared knobs to the one generator + EMIT="$1" NAME="$DEPLOY" SELF_ID="$SELF_ID" NAMESPACE="$NAMESPACE" \ + "${SCRIPT_DIR}/attach-lineage.sh" +} + +gen cm | kubectl apply -f - +kubectl patch deploy "$DEPLOY" -n "$NAMESPACE" --type strategic --patch "$(gen patch)" + +kubectl rollout status -n "$NAMESPACE" "deploy/$DEPLOY" --timeout=180s +echo ">> lineage sidecar attached to deploy/$DEPLOY (self_id=$SELF_ID, ns=$NAMESPACE)" From c21dbd6de8efb81f63eb70e4f533ed139b636c12 Mon Sep 17 00:00:00 2001 From: YehoshuaSagron Date: Fri, 14 Aug 2026 21:41:33 +0300 Subject: [PATCH 2/2] Docs: Add the lineage demo to the demos index Signed-off-by: YehoshuaSagron --- authbridge/demos/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/authbridge/demos/README.md b/authbridge/demos/README.md index 690a89be2..9ae3dd78e 100644 --- a/authbridge/demos/README.md +++ b/authbridge/demos/README.md @@ -23,6 +23,7 @@ more AuthBridge capabilities. | **[abctl Walkthrough](weather-agent/demo-with-abctl.md)** | Reference | Watch the AuthBridge plugin pipeline live with the `abctl` TUI | Tooling only | | **[IBAC](ibac/README.md)** | Intermediate | Intent-Based Access Control: LLM judge denies outbound HTTP that doesn't align with the user's recorded intent. Reproduces the email-poison / prompt-injection attack from `huang195/ibac`; chat with the agent through the rossoctl UI and see the exfiltration blocked, then `make show-result` for a pipeline-level forensic | UI + kubectl | | **[SPARC (finance)](finance-sparc/README.md)** | Intermediate | SPARC pre-tool reflection: the `sparc` plugin blocks a hallucinated/ungrounded tool argument (an invented transaction id) before it executes and transparently asks the user to clarify, then approves the corrected call. Complements IBAC — SPARC verifies argument grounding, IBAC verifies intent alignment | UI + kubectl | +| **[Lineage](lineage/README.md)** | Intermediate | Per-request data lineage: enable the `lineage-telemetry` plugin and every HTTP exchange becomes two facts-only spans (`request` + `response`, paired by `lineage.exchange.id`) sent to **any** OTLP consumer. Ships a propagate-only OTel shim that makes the pairing correct under concurrency, and both attach paths — generate a Deployment, or patch one you do not own | kubectl + scripts | | **[CPEX Bridge (HR)](hr-cpex/README.md)** | Advanced | CPEX/APL declarative policy: one route chains a coarse APL predicate, an embedded Cedar PDP, RFC 8693 token exchange with a post-check, PII redaction and audit plugins. Same request, different data per caller (Bob sees an SSN, Eve gets it redacted). Self-contained: its own kind cluster + namespace, deployed via `make` rather than operator injection | [kubectl (make)](hr-cpex/README.md#quick-start) | ## Recommended Path