diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bc38adc..6dbba41 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -22,3 +22,4 @@ jobs: - run: npm run test -w @autter/runtime-browser - run: npm run test -w @autter/otlp-ingester - run: npm run size -w @autter/runtime-browser + - run: python3 -m unittest discover -s adapters/python -v diff --git a/CHANGELOG.md b/CHANGELOG.md index cfd7604..4dbebf4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,29 @@ Packages: `otlp-ingester` 1.5.0, `runtime-node` 1.5.0, `runtime-next` 1.5.0, `ru - Browser: `captureException` and the global handlers send `code/why/fix/link/expected/requestId`; new `autterErrorFromResponse(response)`; failed fetch/XHR 5xx responses carry `x-request-id` (`runtime-browser`). - Experimental zero-code `logging.requests` hook mode, off by default (`runtime-node`). +### Security + +- Scrub secrets and PII from exception messages, stack traces and span status messages, not only custom attributes; re-scrub every span at export so third-party instrumentations, HTTP URLs (`?token=`) and `recordException` calls are covered; scrub `LlmCallHandle.setAttributes`, `instrumentLlmClient` attributes and provider errors, and module-level captures made before `initAutterServer` (`runtime-node`). +- New value patterns everywhere: Basic auth, `Authorization:`/`Cookie:`/`Set-Cookie:` header text, connection strings with empty usernames or `@` in the password, `sk-proj-`/`sk-ant-`, Stripe, GitHub fine-grained, GitLab, npm, SendGrid and Google API keys, `password=`/`?token=`/`?api_key=` assignments, Luhn-valid card numbers. New sensitive keys: any `…authorization` header key, `session`/`sessionid`/`sid`, `pwd`, `dsn`, bounded `ssn`. Custom value patterns now apply globally (`runtime-node`, `runtime-browser`, `otlp-ingester`). +- The browser relay scrubs message, stack, name and nested context (`redact` option) (`runtime-node`). +- Browser SDK scrubs messages, stacks and nested context; new `redact: { keys, values } | false` option and `scrubText` export (`runtime-browser`). +- Ingester scrubs occurrence message/stack/route, span names, browser context (deep) and LLM-call attributes before storage and the sink webhook; extra patterns via `AUTTER_REDACT_VALUE_PATTERNS` / `AUTTER_REDACT_KEY_PATTERNS` (`otlp-ingester`). +- Python adapter: new stdlib-only `adapters/python/redact.py` (`redact_text`, `redact_attributes`, `configure`); the caught-exception sampler scrubs tracebacks with it. +- Shared parity vectors in `test-vectors/redaction.json` run against every implementation. +- Export-time scrubbing also covers span link attributes, span event names and resource attributes on traces and metrics (`process.command_args` from NodeSDK's process detector, `OTEL_RESOURCE_ATTRIBUTES`). A batch with a span that could not be scrubbed is reported as a failed export instead of a silent partial success (`runtime-node`). +- The JWT pattern no longer takes seconds on long `eyJ-eyJ-…` runs (5 s per 64 KB in Node, 3.4 s in the ingester, 6 s in the browser); the ingester caps each scrubbed string at 64 KB and cuts context values to their stored size before scrubbing (`runtime-node`, `runtime-browser`, `otlp-ingester`, Python). +- Custom key/value patterns with `g` or `y` flags work on every key and every occurrence (a `/x/g` key pattern used to skip every other matching key); browser value patterns mask every match, not just the first (`runtime-node`, `runtime-browser`). +- Messages, names and stacks are scrubbed before they are cut to size, so a secret straddling the limit is masked whole (browser, relay, Python sampler). A browser context with a throwing getter or revoked Proxy is dropped instead of throwing into the app. The `card` key rule no longer masks keys like `discard` or `scorecard`. + +### Version compatibility check + +- One source of truth for feature requirements: `packages/otlp-ingester/src/compat-manifest.json`. It lists the minimum ingester, ClickHouse migrations, route and minimum SDK versions for each feature. See `docs/COMPATIBILITY.md`. +- New `GET /v1/compat` (public): ingester version, schema state and level, and per-feature availability. Optional `?features=…&sdk=name@version` returns evaluated issues. Every response carries `x-autter-ingester-version`, exposed to CORS on `/v1/browser`. The ingester records the SDK name and version each service sends with (`runtime_sdk_versions`) and its own version (`runtime_ingester_info`), through migration `0015-runtime-compat` (`otlp-ingester`). +- The SDK checks the ingester once, in the background, when a feature that needs a newer ingester is in use. It warns once per incompatible feature, naming both versions and the fix. Opt out with `compatCheck: false` or `AUTTER_COMPAT_CHECK=0`. The SDK reports itself as `telemetry.distro.name` and `telemetry.distro.version`, and the browser relay forwards the browser SDK version. Both check browser features against the ingester's version header (`runtime-node`, `runtime-next`). +- New `npx @autter/runtime-node doctor`: a one-shot SDK, ingester and schema report. Exits non-zero on a mismatch (`runtime-node`). +- The browser payload carries the SDK version (`sdk`). Older ingesters ignore it (`runtime-browser`). +- Python adapter: new stdlib-only `adapters/python/compat.py`, with `warn_if_incompatible` and `python3 compat.py doctor`. + ### Changes - Console output is a pretty tree outside production; production keeps JSON lines (`runtime-node`). @@ -40,10 +63,15 @@ Packages: `otlp-ingester` 1.5.0, `runtime-node` 1.5.0, `runtime-next` 1.5.0, `ru - `runtime-browser` no longer declares a `./dist/index.cjs` entry that the ESM-only build never produced. - `runtime-next` builds no longer race on `dist/` cleanup and drop `client.d.ts` / `edge.d.ts`. +### Behavior notes + +- Error messages that contained secrets now fingerprint on their scrubbed form, so such issues may regroup once (they previously split per secret value). + ### Upgrade -- Deploy otlp-ingester 1.5.0 first (migrations 0012–0014 run at boot), then SDKs. Older ingesters accept 1.5.0 SDK traffic but ignore the new fields. +- Deploy otlp-ingester 1.5.0 first (migrations 0012–0015 run at boot), then SDKs. Older ingesters accept 1.5.0 SDK traffic but ignore the new fields. - The `runtime_request_1m` rollup counts every insert, so it is approximate when exporters retry a batch. +- The compatibility tables (`runtime_sdk_versions`, `runtime_ingester_info`) are created by migration `0015-runtime-compat`, after 0012–0014. ## [1.3.1] - 2026-09-08 diff --git a/README.md b/README.md index 33ba597..cd892ab 100644 --- a/README.md +++ b/README.md @@ -182,6 +182,7 @@ all, and ad-blockers can't tell it apart from your own API traffic. - [Operation logging and diagnostic context](docs/OPERATION-LOGGING.md) - [Requests, coded errors and background work](docs/REQUESTS-AND-ERRORS.md) - [Continuous detection, profiles, and outcomes](docs/CONTINUOUS-DETECTION.md) +- [Version compatibility](docs/COMPATIBILITY.md): the built-in SDK/ingester/schema check and `npx @autter/runtime-node doctor` - [Roadmap](docs/PLAN.md) · [Releasing](docs/RELEASING.md) ## Contributing @@ -193,6 +194,11 @@ good first areas to pick up. ## Hosting the ingester +Upgrade the ingester before the SDKs. New features need its routes and +ClickHouse migrations. The SDK warns once at runtime when they don't match, +and `npx @autter/runtime-node doctor --endpoint ` prints a full +report. See [Version compatibility](docs/COMPATIBILITY.md). + **Autter cloud** hosts it at `otlp.autter.dev` (the SDKs' default endpoint). Deployment runbook + scripts for the AWS/ECS setup: [`deploy/aws`](deploy/aws). diff --git a/adapters/python/caught_exceptions.py b/adapters/python/caught_exceptions.py index 86de6df..c2cc460 100644 --- a/adapters/python/caught_exceptions.py +++ b/adapters/python/caught_exceptions.py @@ -4,18 +4,26 @@ ones. It has substantial overhead and must be explicitly enabled. Pass an ``emit(name, stack)`` callback that records a standard OTel exception event with ``autter.handled=true`` and ``autter.sampled=true`` on an always-on span. + +The formatted traceback includes ``str(exc)``, so it is scrubbed for secrets +and PII (connection strings, tokens, API keys, emails, …) with +:func:`redact.redact_text` before ``emit`` sees it. """ from __future__ import annotations import random -import re import sys import threading import time import traceback from collections.abc import Callable +try: # imported as part of a package + from .redact import redact_text +except ImportError: # copied next to the app as a plain module + from redact import redact_text + def install_caught_sampler( emit: Callable[[str, str], None], *, sample_rate: float = 0.01, max_per_minute: int = 10 @@ -43,8 +51,8 @@ def hook(frame, event, arg): return hook seen.add(identity) sent += 1 - stack = "".join(traceback.format_exception(exc_type, exc_value, tb, limit=8))[:8000] - stack = re.sub(r"[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}", "[redacted]", stack, flags=re.I) + # Scrub with slack, then cut: a secret straddling the cut stays whole. + stack = redact_text("".join(traceback.format_exception(exc_type, exc_value, tb, limit=8))[:8512])[:8000] try: emit(exc_type.__name__, stack) except Exception: diff --git a/adapters/python/compat.py b/adapters/python/compat.py new file mode 100644 index 0000000..f145060 --- /dev/null +++ b/adapters/python/compat.py @@ -0,0 +1,430 @@ +"""Version compatibility check for Python services sending to Autter Runtime. + +Plain OpenTelemetry SDKs talk to the Autter ingester directly, so nothing in +the SDK knows whether the ingester can store what it sends (OTLP logs need +ingester >= 1.4.0, memory metrics >= 1.3.3, ...). This module asks once: + +* ``warn_if_incompatible(endpoint, ["operation_logging"])`` - fire-and-forget + background check at startup; logs ONE warning per incompatible feature on + the ``autter.runtime`` logger, naming both versions and the fix. Never + raises, never blocks (daemon thread). Disabled by ``AUTTER_COMPAT_CHECK=0``. +* ``python3 compat.py doctor --endpoint https://ingest.example.com`` - one-shot + report; exit 0 compatible, 1 mismatch, 2 ingester unreachable. + +Stdlib only, like the rest of this adapter. ``MANIFEST`` is a copy of the +single source of truth, ``packages/otlp-ingester/src/compat-manifest.json``; +``test_compat.py`` fails if they drift. +""" + +from __future__ import annotations + +import json +import logging +import os +import re +import sys +import threading +import urllib.error +import urllib.request +from typing import Any, Iterable, Optional + +MANIFEST: dict[str, Any] = json.loads( + r""" +{ + "manifestVersion": 1, + "docs": "https://github.com/Autter-dev/autter-runtime/blob/main/docs/COMPATIBILITY.md", + "upgrade": { + "ingester": "docker pull ghcr.io/autter-dev/otlp-ingester:latest and restart it (ClickHouse migrations run at boot)", + "sdk": "npm install {package}@latest" + }, + "features": [ + { + "id": "llm_calls", + "title": "LLM call tracking", + "ingester": "1.1.0", + "migrations": [ + "0004-llm-calls" + ], + "sdks": { + "@autter/runtime-node": "1.1.0", + "@autter/runtime-next": "1.1.0" + } + }, + { + "id": "endpoint_latency", + "title": "Endpoint latency regression detection", + "ingester": "1.3.1", + "migrations": [ + "0005-latency-histograms" + ], + "sdks": { + "@autter/runtime-node": "1.3.0", + "@autter/runtime-next": "1.3.0" + } + }, + { + "id": "browser_network_events", + "title": "Browser network, timing and outcome capture", + "ingester": "1.3.2", + "migrations": [], + "sdks": { + "@autter/runtime-browser": "1.3.2", + "@autter/runtime-next": "1.3.2" + }, + "browserEvents": [ + "request_failure", + "timing", + "outcome" + ] + }, + { + "id": "profiles", + "title": "Profile ingestion", + "ingester": "1.3.2", + "migrations": [ + "0006-profile-samples" + ], + "route": "/v1/profiles", + "sdks": {} + }, + { + "id": "source_maps", + "title": "Source map upload", + "ingester": "1.3.2", + "migrations": [ + "0007-source-maps" + ], + "route": "/v1/sourcemaps", + "sdks": {} + }, + { + "id": "memory_metrics", + "title": "Memory pressure detection", + "ingester": "1.3.3", + "migrations": [ + "0008-memory-signals", + "0010-memory-temporality" + ], + "sdks": { + "@autter/runtime-node": "1.3.3", + "@autter/runtime-next": "1.3.3" + } + }, + { + "id": "platform_events", + "title": "Platform OOM/restart events", + "ingester": "1.3.3", + "migrations": [ + "0008-memory-signals" + ], + "route": "/v1/platform-events", + "sdks": {} + }, + { + "id": "csp_violations", + "title": "Browser CSP violation capture", + "ingester": "1.3.4", + "migrations": [], + "sdks": { + "@autter/runtime-browser": "1.3.3", + "@autter/runtime-next": "1.3.4" + }, + "browserEvents": [ + "csp_violation" + ] + }, + { + "id": "operation_logging", + "title": "Operation logging", + "ingester": "1.4.0", + "migrations": [ + "0011-runtime-logs" + ], + "route": "/v1/logs", + "sdks": { + "@autter/runtime-node": "1.4.0", + "@autter/runtime-next": "1.4.0" + } + }, + { + "id": "request_summaries", + "title": "Request summaries and the per-route request rollup", + "ingester": "1.5.0", + "migrations": [ + "0012-runtime-logs-requests", + "0014-runtime-request-1m" + ], + "route": "/v1/logs", + "sdks": { + "@autter/runtime-node": "1.5.0", + "@autter/runtime-next": "1.5.0", + "@autter/runtime-edge": "1.0.0" + } + }, + { + "id": "coded_errors", + "title": "Coded errors and code-based grouping", + "ingester": "1.5.0", + "migrations": [ + "0013-occurrence-codes" + ], + "sdks": { + "@autter/runtime-node": "1.5.0", + "@autter/runtime-next": "1.5.0", + "@autter/runtime-browser": "1.4.0", + "@autter/runtime-edge": "1.0.0" + } + } + ] +} +""" +) + +INGESTER_VERSION_HEADER = "x-autter-ingester-version" +PYTHON_FEATURES = ("llm_calls", "endpoint_latency", "memory_metrics", "operation_logging") +DEFAULT_ENDPOINT = "https://otlp.autter.dev" +LOGGER = logging.getLogger("autter.runtime") + +_VERSION_RE = re.compile(r"^v?(\d{1,6})\.(\d{1,6})\.(\d{1,6})(-[0-9A-Za-z.-]+)?") + + +def _parse(version: Optional[str]): + if not isinstance(version, str): + return None + match = _VERSION_RE.match(version.strip()) + if not match: + return None + return (int(match[1]), int(match[2]), int(match[3]), 0 if match[4] else 1) + + +def version_at_least(version: Optional[str], minimum: str) -> Optional[bool]: + """True/False when comparable, None when the version is unknown.""" + left, right = _parse(version), _parse(minimum) + if left is None or right is None: + return None + return left >= right + + +def _feature(feature_id: str) -> Optional[dict]: + return next((f for f in MANIFEST["features"] if f["id"] == feature_id), None) + + +def ingester_upgrade_hint() -> str: + return f"Upgrade the ingester: {MANIFEST['upgrade']['ingester']}. See {MANIFEST['docs']}" + + +def evaluate(features: Iterable[str], ingester: Optional[dict]) -> list[dict]: + """Incompatibilities for the features in use (same rules as compat.ts). + + ``ingester`` is ``{"version": str|None, "schema": {"status", "applied"}, + "routes": {route: bool}}``. Unknown never produces an issue. + """ + issues: list[dict] = [] + if not ingester: + return issues + seen: set[str] = set() + fix = ingester_upgrade_hint() + for feature_id in features: + if feature_id in seen: + continue + seen.add(feature_id) + feature = _feature(feature_id) + if feature is None: + continue + version = ingester.get("version") + at_least = version_at_least(version, feature["ingester"]) + if at_least is False: + issues.append({ + "feature": feature_id, "kind": "ingester_too_old", + "required": feature["ingester"], "actual": version, "fix": fix, + "message": f"{feature['title']} needs ingester >= {feature['ingester']}; yours is {version}. {fix}", + }) + continue + route = feature.get("route") + if version is None and route and (ingester.get("routes") or {}).get(route) is False: + actual = "1.4.0 or older (no /v1/compat)" + issues.append({ + "feature": feature_id, "kind": "ingester_too_old", + "required": feature["ingester"], "actual": actual, "fix": fix, + "message": f"{feature['title']} needs ingester >= {feature['ingester']}; yours is {actual} " + f"and has no {route} route, so this data is dropped. {fix}", + }) + continue + schema = ingester.get("schema") + if at_least and schema and feature["migrations"]: + applied = set(schema.get("applied") or []) + missing = [m for m in feature["migrations"] if m not in applied] + status = schema.get("status") + if status == "failed" or (status == "ready" and missing): + what = ", ".join(missing or feature["migrations"]) + schema_fix = ("Check the ingester's CLICKHOUSE_URL/credentials and its logs, then restart it so " + f"the boot migrations run. See {MANIFEST['docs']}") + issues.append({ + "feature": feature_id, "kind": "schema_not_applied", + "required": ", ".join(feature["migrations"]), "actual": status, "fix": schema_fix, + "message": f"{feature['title']} needs ClickHouse migration {what}, which ingester {version} " + f"has not applied (schema {status}). {schema_fix}", + }) + return issues + + +def _request(method: str, url: str, timeout: float, body: Optional[bytes] = None): + """(status, headers, text) or None on any failure.""" + req = urllib.request.Request(url, data=body, method=method, headers={ + "accept": "application/json", "user-agent": "autter-runtime-python-compat", + **({"content-type": "application/json"} if body is not None else {}), + }) + try: + with urllib.request.urlopen(req, timeout=timeout) as res: # noqa: S310 - user-configured endpoint + return res.status, res.headers, res.read(256 * 1024).decode("utf-8", "replace") + except urllib.error.HTTPError as err: + return err.code, err.headers, "" + except Exception: # noqa: BLE001 - never raise from a diagnostic + return None + + +def fetch_ingester_compat(endpoint: str, probe_routes: Iterable[str] = (), timeout: float = 3.0): + """``(info, error)``: what the ingester supports, or ``(None, reason)``. + + A 404 on /v1/compat means an ingester from before version reporting + (1.4.0 or older); ``probe_routes`` then learns which routes exist. + """ + base = endpoint.rstrip("/") + result = _request("GET", f"{base}/v1/compat", timeout) + if result is None: + return None, f"could not reach {base}/v1/compat" + status, headers, text = result + header_version = headers.get(INGESTER_VERSION_HEADER) if headers else None + if status == 200: + try: + report = json.loads(text) + version = report["ingester"]["version"] + if _parse(version) is None: + raise ValueError(version) + info: dict = {"version": version} + schema = report.get("schema") or {} + if schema.get("status") in ("ready", "pending", "failed", "unconfigured"): + info["schema"] = {"status": schema["status"], "applied": list(schema.get("applied") or [])} + return info, None + except Exception: # noqa: BLE001 + if _parse(header_version): + return {"version": header_version}, None + return None, "unrecognised /v1/compat response" + if status == 404: + if _parse(header_version): + return {"version": header_version}, None + routes = {} + for route in probe_routes: + probe = _request("POST", f"{base}{route}", timeout, b"{}") + if probe is not None: + routes[route] = probe[0] != 404 + return {"version": None, "legacy": True, "routes": routes}, None + return None, f"/v1/compat answered {status}" + + +def _routes_for(features: Iterable[str]) -> list[str]: + routes = [] + for feature_id in features: + route = (_feature(feature_id) or {}).get("route") + if route and route not in routes: + routes.append(route) + return routes + + +_warned: set[str] = set() +_lock = threading.Lock() + + +def _env_disabled() -> bool: + return os.environ.get("AUTTER_COMPAT_CHECK", "").strip().lower() in ("0", "false", "off") + + +def check_once(endpoint: str, features: Iterable[str], logger: Optional[logging.Logger] = None) -> list[dict]: + """Synchronous check; logs each new issue once per process. Never raises.""" + try: + features = list(features) + info, error = fetch_ingester_compat(endpoint, _routes_for(features)) + log = logger or LOGGER + if info is None: + log.debug("autter compat check skipped: %s", error) + return [] + issues = evaluate(features, info) + for issue in issues: + key = f"{issue['feature']}:{issue['kind']}" + with _lock: + if key in _warned: + continue + _warned.add(key) + log.warning("[autter-runtime] %s", issue["message"]) + return issues + except Exception: # noqa: BLE001 - never break the host app + return [] + + +def warn_if_incompatible( + endpoint: str, + features: Iterable[str], + *, + enabled: bool = True, + logger: Optional[logging.Logger] = None, +) -> Optional[threading.Thread]: + """Fire-and-forget background check (daemon thread). Returns the thread + (for tests) or None when disabled.""" + if not enabled or _env_disabled(): + return None + thread = threading.Thread( + target=check_once, args=(endpoint, list(features), logger), name="autter-compat-check", daemon=True + ) + thread.start() + return thread + + +def doctor(endpoint: str, features: Optional[list[str]] = None, timeout: float = 5.0) -> tuple[dict, int]: + """One-shot report and exit code (0 ok, 1 mismatch, 2 unreachable).""" + # What a Python service sends over plain OTLP: traces (LLM spans), + # metrics (latency histograms, process memory) and logs. + features = features or list(PYTHON_FEATURES) + info, error = fetch_ingester_compat(endpoint, _routes_for(features), timeout) + issues = evaluate(features, info) + report = {"endpoint": endpoint.rstrip("/"), "ingester": info, "error": error, "features": features, "issues": issues} + return report, 2 if info is None else 1 if issues else 0 + + +def main(argv: Optional[list[str]] = None) -> int: + import argparse + + parser = argparse.ArgumentParser(prog="compat.py", description="Autter Runtime version compatibility check") + sub = parser.add_subparsers(dest="command", required=True) + doc = sub.add_parser("doctor", help="check the ingester against the features you use") + doc.add_argument("--endpoint", default=os.environ.get("AUTTER_ENDPOINT") + or os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT") or DEFAULT_ENDPOINT) + doc.add_argument("--features", default="", help="comma-separated ids: " + + ", ".join(f["id"] for f in MANIFEST["features"])) + doc.add_argument("--json", action="store_true") + doc.add_argument("--timeout", type=float, default=5.0) + args = parser.parse_args(argv) + features = [f.strip() for f in args.features.split(",") if f.strip()] or None + report, code = doctor(args.endpoint, features, args.timeout) + if args.json: + print(json.dumps(report, indent=2)) + return code + info = report["ingester"] + print("Autter Runtime doctor (Python)") + print(f" endpoint {report['endpoint']}") + if info is None: + print(f" ingester unreachable or unidentified ({report['error']})") + elif info.get("version"): + print(f" ingester @autter/otlp-ingester@{info['version']}") + if info.get("schema"): + print(f" schema {info['schema']['status']}") + else: + print(" ingester 1.4.0 or older (no /v1/compat endpoint; upgrade for a full report)") + for issue in report["issues"]: + print(f" - {issue['message']}") + print({0: "Compatible.", 1: "Incompatible: see the problems above.", + 2: "Could not check the ingester. Pass --endpoint ."}[code]) + return code + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/adapters/python/redact.py b/adapters/python/redact.py new file mode 100644 index 0000000..be056cc --- /dev/null +++ b/adapters/python/redact.py @@ -0,0 +1,203 @@ +"""Secret/PII scrubbing for Python services reporting to Autter Runtime. + +Python exceptions routinely carry secrets in their *message* (``str(exc)``) +and therefore in every formatted traceback: ``psycopg2.OperationalError: +connection to postgres://admin:hunter2@db failed``, ``401 Incorrect API key +provided: sk-...``. Scrub before anything leaves the process: + + from redact import redact_text, redact_attributes + + span.add_event("exception", { + "exception.type": type(exc).__name__, + "exception.message": redact_text(str(exc)), + "exception.stacktrace": redact_text("".join(traceback.format_exception(exc))), + }) + span.set_attributes(redact_attributes(custom_context)) + +Same patterns as ``redactText``/``redactAttributes`` in @autter/runtime-node; +parity is enforced by the shared ``test-vectors/redaction.json``. Secure by +default; extend with :func:`configure` (your own value/key patterns). The +Autter ingester scrubs again server-side as a second line of defence. + +Stdlib only. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Mapping +from typing import Any + +MASK = "[redacted]" + +_PRIVATE_KEY_BLOCK = re.compile( + r"-----BEGIN [A-Z ]*PRIVATE KEY-----[\s\S]*?(?:-----END [A-Z ]*PRIVATE KEY-----|$)" +) +_PREFIXED_SECRET = re.compile( + r"\b(?:sk-[A-Za-z0-9_-]{20,}|(?:sk|rk)_(?:live|test)_[A-Za-z0-9]{10,}|whsec_[A-Za-z0-9]{10,}" + r"|gh[pousr]_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,}|glpat-[A-Za-z0-9_-]{20,}" + r"|xox[abposr]-[A-Za-z0-9-]{10,}|AIza[A-Za-z0-9_-]{30,}|(?:AKIA|ASIA)[0-9A-Z]{16}" + r"|npm_[A-Za-z0-9]{36}|autter_(?:rt|pat)_[A-Za-z0-9_-]{10,}" + r"|SG\.[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,})" +) +# Lookbehind, not \b: a \b start at every "eyJ" after "-" is quadratic. +_JWT = re.compile(r"(?]+)", + re.I, +) +_URL_CREDENTIALS = re.compile( + r"\b([a-z][a-z0-9+.-]{0,31}://)(?:[^\s/:@\"'<>]*:[^\s/\"'<>]*|[^\s/:@\"'<>]{16,})@", re.I +) +_CARD = re.compile( + r"\b(?:4\d{3}|5[1-5]\d{2}|2[2-7]\d{2}|3[47]\d{2}|6(?:011|5\d{2}))(?:[ -]?\d){9,15}\b" +) +_EMAIL = re.compile(r"[A-Z0-9._%+-]{1,64}@[A-Z0-9.-]{1,253}\.[A-Z]{2,63}", re.I) + +_SENSITIVE_KEYS = [ + re.compile(p) + for p in ( + r"e-?mail", + r"pass(word|wd|phrase)|^pass$|(^|[._-])pwd$", + r"token", + r"secret", + r"credential", + r"(api|access|secret|private|consumer|client|signing|encryption)-?[_.]?key", + r"authori[sz]ation|^auth(-|_|$)|bearer", + r"cookie", + r"(^|[._-])session$|^(j|php)?sess(ion)?id$|^sid$|connect\.sid", + r"phone|msisdn", + r"(^|[^a-z])ssn($|[^a-z])|social[-_ ]?security", + r"cvv|cvc|(^|[^a-z])card([-_. ]?(number|num|no))?$", + r"credit[-_.]?card", + r"connection[-_.]?string|(^|[._-])dsn$", + r"recovery[-_.]?code|\botp\b|magic[-_.]?link", + ) +] +# Numeric GenAI usage counters keep their value even though "token" matches. +_USAGE_KEY = re.compile(r"(^|\.)(input|output|total|prompt|completion)_?tokens$|token_count$", re.I) + +_extra_values: list[re.Pattern[str]] = [] +_extra_keys: list[re.Pattern[str]] = [] +_enabled = True + +_MAX_DEPTH = 8 +_MAX_ITEMS = 1000 + + +def configure( + *, + value_patterns: Iterable[str | re.Pattern[str]] = (), + key_patterns: Iterable[str | re.Pattern[str]] = (), + enabled: bool = True, +) -> None: + """Add your own patterns on top of the built-ins (replaces earlier extras). + + ``value_patterns`` are masked wherever they occur inside strings; + ``key_patterns`` mask whole values whose key matches (case-insensitive). + ``enabled=False`` turns scrubbing off (not recommended). + """ + global _extra_values, _extra_keys, _enabled + _extra_values = [p if isinstance(p, re.Pattern) else re.compile(p, re.I) for p in value_patterns] + _extra_keys = [p if isinstance(p, re.Pattern) else re.compile(p, re.I) for p in key_patterns] + _enabled = enabled + + +def _luhn_valid(candidate: str) -> bool: + digits = [int(c) for c in candidate if c.isdigit()] + if not 13 <= len(digits) <= 19: + return False + total = 0 + for i, d in enumerate(reversed(digits)): + if i % 2 == 1: + d *= 2 + if d > 9: + d -= 9 + total += d + return total % 10 == 0 + + +def _mask_assignment(match: re.Match[str]) -> str: + key, sep, value = match.group(1), match.group(2), match.group(3) + if value[:1] in ("'", '"'): + return f"{key}{sep}{value[0]}{MASK}{value[0]}" + return f"{key}{sep}{MASK}" + + +def redact_text(value: str) -> str: + """Scrub secrets/PII embedded in one string; the rest survives.""" + if not _enabled or not isinstance(value, str) or not value: + return value + out = _PRIVATE_KEY_BLOCK.sub(MASK, value) + out = _PREFIXED_SECRET.sub(MASK, out) + out = _JWT.sub(MASK, out) + out = _AUTH_SCHEME.sub(MASK, out) + out = _HEADER_VALUE.sub(lambda m: m.group(1) + MASK, out) + out = _SECRET_ASSIGNMENT.sub(_mask_assignment, out) + out = _URL_CREDENTIALS.sub(lambda m: m.group(1) + MASK + "@", out) + out = _CARD.sub(lambda m: MASK if _luhn_valid(m.group(0)) else m.group(0), out) + out = _EMAIL.sub(MASK, out) + for pattern in _extra_values: + out = pattern.sub(MASK, out) + return out + + +def is_sensitive_key(key: str) -> bool: + lowered = key.lower() + return any(p.search(lowered) for p in _SENSITIVE_KEYS) or any(p.search(key) for p in _extra_keys) + + +def _redact_value(key: str, value: Any, depth: int) -> Any: + if isinstance(value, bool) or value is None: + return value + if isinstance(value, (int, float)): + if _USAGE_KEY.search(key) and value >= 0: + return value + return MASK if is_sensitive_key(key) else value + if is_sensitive_key(key): + return MASK + if isinstance(value, str): + return redact_text(value) + if depth >= _MAX_DEPTH: + return MASK + if isinstance(value, Mapping): + return _redact_mapping(value, depth + 1) + if isinstance(value, (list, tuple)): + return [_redact_value("", item, depth + 1) for item in list(value)[:_MAX_ITEMS]] + try: + text = str(value) + except Exception: # a broken __str__ must not break the caller + return MASK + return redact_text(text) + + +def _redact_mapping(mapping: Mapping[Any, Any], depth: int) -> dict[str, Any]: + out: dict[str, Any] = {} + for i, (key, value) in enumerate(mapping.items()): + if i >= _MAX_ITEMS: + out["__redaction_truncated__"] = MASK + break + out[str(key)] = _redact_value(str(key), value, depth) + return out + + +def redact_attributes(attributes: Mapping[str, Any] | None) -> dict[str, Any]: + """Copy of ``attributes`` with sensitive keys masked (any depth) and + secrets scrubbed inside string values. Never mutates the input.""" + if not attributes: + return {} + if not _enabled: + return dict(attributes) + return _redact_mapping(attributes, 0) diff --git a/adapters/python/test_compat.py b/adapters/python/test_compat.py new file mode 100644 index 0000000..f0558d5 --- /dev/null +++ b/adapters/python/test_compat.py @@ -0,0 +1,141 @@ +"""Tests for the Python adapter's version compatibility check (stdlib only). + +Run: python3 -m unittest discover -s adapters/python -v +""" + +from __future__ import annotations + +import io +import json +import logging +import os +import sys +import threading +import time +import unittest +from contextlib import redirect_stdout +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import compat # noqa: E402 + +MANIFEST_PATH = os.path.join( + os.path.dirname(__file__), "..", "..", "packages", "otlp-ingester", "src", "compat-manifest.json" +) + + +def serve(routes): + """routes: {(method, path): (status, body, headers)}; default 404.""" + + class Handler(BaseHTTPRequestHandler): + def _reply(self): + length = int(self.headers.get("content-length") or 0) + if length: + self.rfile.read(length) + status, body, headers = routes.get((self.command, self.path), (404, "", {})) + self.send_response(status) + for key, value in headers.items(): + self.send_header(key, value) + self.end_headers() + self.wfile.write(body.encode()) + + do_GET = do_POST = _reply + + def log_message(self, *args): + pass + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + threading.Thread(target=server.serve_forever, daemon=True).start() + return server, f"http://127.0.0.1:{server.server_address[1]}" + + +def report(version, applied=None): + return json.dumps({"ingester": {"version": version}, "schema": {"status": "ready", "applied": applied or []}}) + + +class Manifest(unittest.TestCase): + def test_embedded_manifest_matches_the_source_of_truth(self): + with open(MANIFEST_PATH, encoding="utf-8") as fh: + self.assertEqual(compat.MANIFEST, json.load(fh)) + + def test_versions(self): + self.assertTrue(compat.version_at_least("1.4.0", "1.4.0")) + self.assertFalse(compat.version_at_least("1.4.0-rc.1", "1.4.0")) + self.assertTrue(compat.version_at_least("v1.10.0", "1.9.0")) + self.assertIsNone(compat.version_at_least("latest", "1.0.0")) + + +class Check(unittest.TestCase): + def setUp(self): + compat._warned.clear() + self.stream = io.StringIO() + self.logger = logging.getLogger(f"autter.test.{self.id()}") + self.logger.addHandler(logging.StreamHandler(self.stream)) + self.logger.setLevel(logging.DEBUG) + self.logger.propagate = False + + def test_old_ingester_warns_once_with_versions_and_fix(self): + server, url = serve({("GET", "/v1/compat"): (200, report("1.3.4"), {})}) + try: + for _ in range(2): + thread = compat.warn_if_incompatible(url, ["operation_logging"], logger=self.logger) + thread.join(5) + lines = [l for l in self.stream.getvalue().splitlines() if "Operation logging" in l] + self.assertEqual(len(lines), 1) + self.assertRegex(lines[0], r"Operation logging needs ingester >= 1\.4\.0; yours is 1\.3\.4\. Upgrade the ingester: docker pull") + finally: + server.shutdown() + + def test_no_warning_when_compatible(self): + applied = ["0004-llm-calls", "0005-latency-histograms", "0008-memory-signals", + "0010-memory-temporality", "0011-runtime-logs"] + server, url = serve({("GET", "/v1/compat"): (200, report("1.4.0", applied), {})}) + try: + issues = compat.check_once(url, compat.PYTHON_FEATURES, logger=self.logger) + self.assertEqual(issues, []) + self.assertNotIn("[autter-runtime]", self.stream.getvalue()) + finally: + server.shutdown() + + def test_legacy_ingester_probe(self): + server, url = serve({("POST", "/v1/traces"): (401, "", {})}) # /v1/compat and /v1/logs: 404 + try: + issues = compat.check_once(url, ["operation_logging"], logger=self.logger) + self.assertEqual(len(issues), 1) + self.assertIn("no /v1/logs route", issues[0]["message"]) + finally: + server.shutdown() + + def test_unreachable_never_raises_or_blocks(self): + started = time.monotonic() + thread = compat.warn_if_incompatible("http://127.0.0.1:9", ["operation_logging"], logger=self.logger) + self.assertLess(time.monotonic() - started, 0.5) + self.assertTrue(thread.daemon) + thread.join(5) + self.assertNotIn("[autter-runtime]", self.stream.getvalue()) + self.assertEqual(compat.check_once("not a url", ["operation_logging"]), []) + + def test_disabled_by_env(self): + os.environ["AUTTER_COMPAT_CHECK"] = "0" + try: + self.assertIsNone(compat.warn_if_incompatible("http://127.0.0.1:9", ["operation_logging"])) + finally: + del os.environ["AUTTER_COMPAT_CHECK"] + + def test_doctor_exit_codes(self): + server, url = serve({("GET", "/v1/compat"): (200, report("1.3.2"), {})}) + try: + out = io.StringIO() + with redirect_stdout(out): + code = compat.main(["doctor", "--endpoint", url, "--features", "memory_metrics"]) + self.assertEqual(code, 1) + self.assertIn("Memory pressure detection needs ingester >= 1.3.3; yours is 1.3.2", out.getvalue()) + finally: + server.shutdown() + with redirect_stdout(io.StringIO()): + self.assertEqual(compat.main(["doctor", "--endpoint", "http://127.0.0.1:9", "--timeout", "1"]), 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/adapters/python/test_redact.py b/adapters/python/test_redact.py new file mode 100644 index 0000000..bf37d75 --- /dev/null +++ b/adapters/python/test_redact.py @@ -0,0 +1,127 @@ +"""Tests for the Python adapter's redaction (stdlib unittest, no deps). + +Run: python3 -m unittest discover -s adapters/python -v +""" + +from __future__ import annotations + +import json +import os +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +import redact # noqa: E402 +from caught_exceptions import install_caught_sampler # noqa: E402 +from redact import MASK, configure, redact_attributes, redact_text # noqa: E402 + +VECTORS_PATH = os.path.join(os.path.dirname(__file__), "..", "..", "test-vectors", "redaction.json") +with open(VECTORS_PATH, encoding="utf-8") as fh: + VECTORS = json.load(fh) + + +def _join(value): + return "".join(value) if isinstance(value, list) else value + + +class SharedVectors(unittest.TestCase): + def test_text_vectors(self): + for vector in VECTORS["text"]: + if "python" in vector.get("skip", []): + continue + with self.subTest(vector["name"]): + self.assertEqual(redact_text(_join(vector["input"])), _join(vector["expect"])) + + def test_sensitive_keys_masked_at_any_depth(self): + for key in VECTORS["keys"]["sensitive"]: + with self.subTest(key): + out = redact_attributes({"outer": {"items": [{key: "raw"}]}}) + self.assertEqual(out["outer"]["items"][0][key], MASK) + + def test_safe_keys_survive(self): + for key in VECTORS["keys"]["safe"]: + with self.subTest(key): + self.assertEqual(redact_attributes({"outer": {key: "kept"}})["outer"][key], "kept") + + +class Attributes(unittest.TestCase): + def test_nested_values_and_usage_counters(self): + out = redact_attributes( + { + "db": {"dsn": "postgres://u:p@h/db", "args": ["redis://:pw@cache:6379"]}, + "gen_ai.usage.input_tokens": 12, + "max_tokens": 100, + "note": "contact jane@example.com", + } + ) + self.assertEqual(out["db"]["dsn"], MASK) + self.assertEqual(out["db"]["args"], [f"redis://{MASK}@cache:6379"]) + self.assertEqual(out["gen_ai.usage.input_tokens"], 12) + self.assertEqual(out["max_tokens"], MASK) + self.assertEqual(out["note"], f"contact {MASK}") + + def test_does_not_mutate_input(self): + original = {"password": "x", "nested": {"token": "y"}} + redact_attributes(original) + self.assertEqual(original, {"password": "x", "nested": {"token": "y"}}) + + def test_custom_patterns(self): + configure(value_patterns=[r"CUST-\d{4}"], key_patterns=[r"^tenant_ref$"]) + try: + self.assertEqual(redact_text("CUST-1234 / CUST-9999"), f"{MASK} / {MASK}") + self.assertEqual(redact_attributes({"tenant_ref": "t"}), {"tenant_ref": MASK}) + finally: + configure() + + def test_opt_out(self): + configure(enabled=False) + try: + self.assertEqual(redact_text("password=x"), "password=x") + finally: + configure() + self.assertTrue(redact._enabled) + + +class Hardening(unittest.TestCase): + def test_jwt_pattern_is_linear(self): + import time + + started = time.monotonic() + redact_text("eyJ-" * 16384) + self.assertLess(time.monotonic() - started, 0.5) + + def test_broken_str_is_masked_not_raised(self): + class Broken: + def __str__(self): + raise RuntimeError("no") + + self.assertEqual(redact_attributes({"x": Broken()}), {"x": MASK}) + + +class CaughtExceptionSampler(unittest.TestCase): + def test_traceback_message_is_scrubbed_before_emit(self): + emitted: list[tuple[str, str]] = [] + + def connect(): + raise ConnectionError("could not connect to postgres://admin:hunter2@db.internal:5432/app") + + stop = install_caught_sampler(lambda name, stack: emitted.append((name, stack)), sample_rate=1.0) + try: + try: + connect() + except ConnectionError: + pass + finally: + stop() + + self.assertTrue(emitted, "sampler emitted nothing") + name, stack = emitted[0] + self.assertEqual(name, "ConnectionError") + self.assertNotIn("hunter2", stack) + self.assertIn(f"postgres://{MASK}@db.internal:5432/app", stack) + self.assertIn("in connect", stack) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 273e035..792c60b 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -303,11 +303,20 @@ counters, not an analytics event store). Forbidden at the schema level (rejected/stripped): full URLs with query strings, cookies, DOM content, form values, request headers/bodies, emails. -Server-side custom attributes are guarded at the source instead: the Node -SDK masks email/token/credential-shaped values and sensitive-keyed -attributes before export (`redactAttributes`, on by default), so OTLP spans -never carry a stray `user.email` even though the OTLP schema itself accepts -free-form attributes. +Server-side telemetry is guarded at the source instead: the Node SDK masks +secret/PII-shaped values (tokens, keys, credentials in connection strings +and URLs, emails, card numbers) and sensitive-keyed attributes in +attributes, exception messages, stack traces and status messages, and +re-scrubs every span at export (`redactAttributes`, on by default). The +browser SDK, the relay and the Python adapter apply the same patterns. + +Because old SDKs and plain OTel senders scrub nothing, the ingester scrubs +again before storage and before the sink webhook (which feeds downstream +LLM-assisted fixing): occurrence message/stack/route, span names, all +attribute bags, and LLM-call attributes (`src/redact.ts`; extend with +`AUTTER_REDACT_VALUE_PATTERNS` / `AUTTER_REDACT_KEY_PATTERNS`, JSON arrays +of regex sources). Messages containing secrets therefore fingerprint on +their scrubbed form. ## Sink webhook (v1) diff --git a/docs/COMPATIBILITY.md b/docs/COMPATIBILITY.md new file mode 100644 index 0000000..7d4ef7d --- /dev/null +++ b/docs/COMPATIBILITY.md @@ -0,0 +1,118 @@ +# Version compatibility + +Some Runtime features need a minimum ingester version (it adds the route or +the ClickHouse table they write to) and a minimum SDK version (it starts +sending them). When the ingester is older than the feature, the data is +dropped: the SDK gets a 404 or 400 and retries quietly. You no longer have to +match versions by hand. Runtime checks for you in three places: + +1. **The SDK warns once at runtime.** `@autter/runtime-node` and + `@autter/runtime-next` check the ingester in the background when a + feature that needs a newer ingester is in use. Each incompatible feature + gets one warning: + + ```text + [autter-runtime] Operation logging needs ingester >= 1.4.0; yours is 1.3.4. Upgrade the ingester: docker pull ghcr.io/autter-dev/otlp-ingester:latest and restart it (ClickHouse migrations run at boot). See https://github.com/Autter-dev/autter-runtime/blob/main/docs/COMPATIBILITY.md + ``` + + The check never throws, never delays startup or exit, and prints nothing + when versions match or can't be determined. It makes one `GET /v1/compat` + request per process. It also reads the `x-autter-ingester-version` header + on responses the SDK already receives (log export, browser relay). Turn it + off with `initAutterServer({ compatCheck: false })` or `AUTTER_COMPAT_CHECK=0`. + `debug: true` (or `AUTTER_DEBUG=1`) logs the result even when everything + matches. The browser relay (`createBrowserRelayHandler`, `createAutterRelayRoute`) + checks browser features such as CSP violation capture on the server side. + The browser bundle sends only its version. + +2. **`doctor` gives a one-shot report.** + + ```bash + npx @autter/runtime-node doctor --endpoint https://ingest.example.com + # also verify a key: --key "$AUTTER_RUNTIME_KEY" machine-readable: --json + ``` + + It lists the Autter SDK versions installed in the current project, the + ingester version and its ClickHouse schema level, and every feature those + SDKs can use. Each feature is marked `ok`, `FAIL` (with the fix), or `??` + (an ingester older than version reporting can't tell). Exit codes: `0` + compatible, `1` mismatch or rejected key, `2` ingester unreachable. The + endpoint defaults to `$AUTTER_ENDPOINT`, then `$OTEL_EXPORTER_OTLP_ENDPOINT`, + then `https://otlp.autter.dev`. Use it in CI or after a deploy. + + For Python services, copy `adapters/python/compat.py` next to your app: + + ```bash + python3 compat.py doctor --endpoint https://ingest.example.com --features operation_logging + ``` + + ```python + from compat import warn_if_incompatible # stdlib only; daemon thread, never raises + warn_if_incompatible(os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"], ["operation_logging", "memory_metrics"]) + ``` + +3. **The Autter dashboard shows the status.** Repository → Settings → Runtime + shows the ingester version, its schema level, the SDK and version each + service sends with, and a warning with the fix for any feature your + versions don't support. + +## The ingester endpoint + +`GET /v1/compat` is public, like `/healthz`, and returns no tenant data. It +returns the ingester version, the schema state (`ready`, `pending`, `failed`, +`unconfigured`), the applied migrations, and every feature in the manifest +with `available` set. Add `?features=operation_logging&sdk=@autter/runtime-node@1.3.0` +to get the evaluated `issues` too. Every response, including ingest +responses, carries `x-autter-ingester-version`. + +The ingester records the SDK name and version each service sends with in +`runtime_sdk_versions`, and its own version and schema level in +`runtime_ingester_info` (migration `0015-runtime-compat`). The SDK name and +version come from the OTLP resource attributes `telemetry.distro.name` and +`telemetry.distro.version`, which the Node and Next.js SDKs set. Plain OTel +SDKs are recorded as `opentelemetry-` from `telemetry.sdk.*`. The +browser SDK sends an `sdk` field. Writes are deduplicated to one row per +service, SDK and version per hour. + +Ingesters released before `/v1/compat` (1.4.0 and older) answer it with a +404. The SDK and `doctor` then probe the feature's own route instead. For +example, `POST /v1/logs` without a key returns 401 on 1.4.0 and 404 on older +ingesters. That still catches the most common mismatch: operation logging +against a pre-1.4.0 ingester. + +## Feature matrix + +The single source of truth is +[`packages/otlp-ingester/src/compat-manifest.json`](../packages/otlp-ingester/src/compat-manifest.json). +The SDKs bundle it, the ingester serves it, and the Python adapter embeds a +copy that a test keeps identical. + +| Feature (`id`) | Ingester | Migrations | Route | SDKs | +| --- | --- | --- | --- | --- | +| LLM call tracking (`llm_calls`) | 1.1.0 | `0004-llm-calls` | | node/next 1.1.0 | +| Endpoint latency regression detection (`endpoint_latency`) | 1.3.1 | `0005-latency-histograms` | | node/next 1.3.0 | +| Browser network, timing and outcome capture (`browser_network_events`) | 1.3.2 | | | browser/next 1.3.2 | +| Profile ingestion (`profiles`) | 1.3.2 | `0006-profile-samples` | `/v1/profiles` | | +| Source map upload (`source_maps`) | 1.3.2 | `0007-source-maps` | `/v1/sourcemaps` | | +| Memory pressure detection (`memory_metrics`) | 1.3.3 | `0008`, `0010` | | node/next 1.3.3 | +| Platform OOM/restart events (`platform_events`) | 1.3.3 | `0008-memory-signals` | `/v1/platform-events` | | +| Browser CSP violation capture (`csp_violations`) | 1.3.4 | | | browser 1.3.3, next 1.3.4 | +| Operation logging (`operation_logging`) | 1.4.0 | `0011-runtime-logs` | `/v1/logs` | node/next 1.4.0 | + +Notes: + +- Ingester 1.3.0 created the latency table but rejected numeric delta + temporality, so endpoint latency needs 1.3.1. +- Ingesters older than 1.3.2 (network events) or 1.3.4 (CSP) reject the + **whole** browser batch with a 400 when it contains a newer event type. + Errors in the same batch are lost too. Upgrade the ingester before the + browser SDK. +- Upgrade order: ingester first (its migrations run at boot), then SDKs. + +## Adding a feature + +Append an entry to `compat-manifest.json`. Give the first ingester release +that stores the feature, its migrations, its route (if it has its own), and +the first SDK releases that emit it. The ingester's `compat.test.ts` checks +ids, migrations, routes and versions against the code. The Python parity test +fails until `adapters/python/compat.py` gets the same manifest. diff --git a/docs/CONTINUOUS-DETECTION.md b/docs/CONTINUOUS-DETECTION.md index 7b13c45..f1c7271 100644 --- a/docs/CONTINUOUS-DETECTION.md +++ b/docs/CONTINUOUS-DETECTION.md @@ -14,7 +14,10 @@ Existing OTLP exporters benefit from server-side detection without an SDK upgrade when the ingester is 1.3.2 or later. Install `@autter/runtime-node`, `@autter/runtime-browser`, or `@autter/runtime-next` version 1.3.2 or later to use the new outcome helpers -and browser network and timing capture in an application. +and browser network and timing capture in an application. Ingesters older +than 1.3.2 reject browser batches containing the new event types, so upgrade +the ingester first. [Version compatibility](COMPATIBILITY.md) explains the +built-in check and `doctor` command. ## Report a failed outcome without throwing @@ -119,6 +122,14 @@ OTel `exception` event with `autter.handled=true` and overhead, applies to the current and newly created threads, and should be enabled only for targeted diagnosis. Other languages can send the same event. +The sampler scrubs each traceback (which includes `str(exc)`) for secrets +and PII before calling `emit`, using `adapters/python/redact.py` (stdlib +only; copy it next to `caught_exceptions.py`). Use the same module for your +own `record_exception` calls — `redact_text(str(exc))`, +`redact_text(traceback_text)`, `redact_attributes(context)` — and +`configure(value_patterns=[...], key_patterns=[...])` for custom patterns. +The ingester scrubs again server-side. + ```python from opentelemetry import trace from caught_exceptions import install_caught_sampler diff --git a/docs/ENDPOINT-REGRESSIONS.md b/docs/ENDPOINT-REGRESSIONS.md index 336ad9d..a15b41a 100644 --- a/docs/ENDPOINT-REGRESSIONS.md +++ b/docs/ENDPOINT-REGRESSIONS.md @@ -26,7 +26,7 @@ The SDK adds a process instance ID and explicit request histogram bounds through Send explicit-bucket histograms with delta temporality: -Use ingester 1.3.1 or later. Version 1.3.0 does not correctly accept numeric delta temporality. OTLP delta is `1` (or `AGGREGATION_TEMPORALITY_DELTA`); cumulative is `2` and is not accepted for detection. +Use ingester 1.3.1 or later (`npx @autter/runtime-node doctor` checks it; see [Version compatibility](COMPATIBILITY.md)). Version 1.3.0 does not correctly accept numeric delta temporality. OTLP delta is `1` (or `AGGREGATION_TEMPORALITY_DELTA`); cumulative is `2` and is not accepted for detection. - `http.server.request.duration` uses seconds. - `http.server.duration` uses milliseconds. diff --git a/docs/INTEGRATIONS.md b/docs/INTEGRATIONS.md index 51dd82a..65a25de 100644 --- a/docs/INTEGRATIONS.md +++ b/docs/INTEGRATIONS.md @@ -15,6 +15,7 @@ profile and source-map uploads use the server key. | `POST /v1/profiles` | symbolized pprof (server key only) | | `POST /v1/sourcemaps` | release-keyed source map JSON (server key only) | | `POST /v1/platform-events` | ECS/Kubernetes OOM and restart JSON (server key only) | +| `GET /v1/compat` | ingester version, schema level and supported features (public; see [Version compatibility](COMPATIBILITY.md)) | Any language with an OpenTelemetry SDK can send server telemetry — point its OTLP/HTTP exporter at the ingester and add the key as a header. diff --git a/docs/MEMORY-PRESSURE.md b/docs/MEMORY-PRESSURE.md index fd2899b..74f22a9 100644 --- a/docs/MEMORY-PRESSURE.md +++ b/docs/MEMORY-PRESSURE.md @@ -7,7 +7,9 @@ automatically; other languages must enable their runtime/process instrumentation or emit the portable metrics below. An OTel exporter by itself does not measure process memory. Redeploy the application after changing its instrumentation. Self-hosted ingesters need version 1.3.3+ for memory sample -storage and `/v1/platform-events`. +storage and `/v1/platform-events`. The Node SDK warns once if the ingester is +older, and `npx @autter/runtime-node doctor` checks it; see +[Version compatibility](COMPATIBILITY.md). ## What must be running diff --git a/docs/OPERATION-LOGGING.md b/docs/OPERATION-LOGGING.md index 3ca9af3..10fa605 100644 --- a/docs/OPERATION-LOGGING.md +++ b/docs/OPERATION-LOGGING.md @@ -3,6 +3,16 @@ Operation logging starts in `@autter/runtime-node` and `@autter/runtime-next` version **1.4.0**. Update the ingester to **1.4.0** before upgrading SDKs: it creates `runtime_logs` through migration `0011-runtime-logs` and accepts `/v1/logs`. +You don't need to check this by hand. The SDK warns once at runtime when your +ingester is too old, and `npx @autter/runtime-node doctor` reports every +mismatch. See [Version compatibility](COMPATIBILITY.md). + +**1.5.0** (ingester 1.5.0 first) adds request summaries, `runtimeContext`, inline +messages, `kind` on every summary, coded errors, `fork`/carriers, the AI rollup, +`waitUntil`, sinks and logger-only mode — all additive; a golden test proves a +1.4.0 operation still exports the same OTLP JSON apart from the two new summary +keys. Request-level features are documented in +[REQUESTS-AND-ERRORS.md](REQUESTS-AND-ERRORS.md). **1.5.0** (ingester 1.5.0 first) adds request summaries, `runtimeContext`, inline messages, `kind` on every summary, coded errors, `fork`/carriers, the AI rollup, diff --git a/package-lock.json b/package-lock.json index 6582c0b..379f377 100644 --- a/package-lock.json +++ b/package-lock.json @@ -4869,6 +4869,9 @@ "@opentelemetry/sdk-trace-base": "^1.30.1", "@opentelemetry/semantic-conventions": "^1.28.0" }, + "bin": { + "autter-runtime": "dist/cli.js" + }, "devDependencies": { "@autter/runtime-core": "0.0.0", "@types/node": "^22.10.0", diff --git a/packages/otlp-ingester/README.md b/packages/otlp-ingester/README.md index 2e85a65..0a7ba7e 100644 --- a/packages/otlp-ingester/README.md +++ b/packages/otlp-ingester/README.md @@ -14,6 +14,8 @@ one per-repo signal model, fingerprints errors, and writes ClickHouse. | `POST /v1/platform-events` | JSON, server key | ECS/Kubernetes OOM kills and restarts → memory incident correlation | | `POST /v1/profiles` | Symbolized pprof, server key | CPU/in-use heap profile samples | | `POST /v1/browser` | Browser payload `version: 1` | Errors/rejections → occurrences; session pings → rollups | +| `POST /v1/logs` | OTLP logs, server key | Structured logs and operation summaries → `runtime_logs` | +| `GET /v1/compat` | — | Version, schema level and supported features (public) | | `GET /healthz` | — | Liveness + ClickHouse reachability | Auth on every ingest route: `Authorization: Bearer `, @@ -46,6 +48,17 @@ AUTTER_INGEST_KEYS='[ The validator webhook may return the same extra fields: `{ orgId, repositoryId, scope?, allowedOrigins? }`. +### Version compatibility + +`GET /v1/compat` (public, no tenant data) returns this ingester's version, +its ClickHouse schema state, and which features from +`src/compat-manifest.json` it supports. Add +`?features=operation_logging&sdk=@autter/runtime-node@1.3.0` to get the evaluated +incompatibilities. Every response carries `x-autter-ingester-version`. The +ingester records the SDK name and version each service sends with +(`runtime_sdk_versions`) and its own version (`runtime_ingester_info`), and +the Autter dashboard reads both. See [docs/COMPATIBILITY.md](../../docs/COMPATIBILITY.md). + ## Configuration | Env | Default | Description | @@ -69,6 +82,8 @@ The validator webhook may return the same extra fields: | `OCCURRENCE_TTL_DAYS` / `SPAN_TTL_DAYS` / `METRICS_TTL_DAYS` | `14` / `7` / `90` | ClickHouse TTLs (applied at table creation) | | `LLM_CALL_TTL_DAYS` | `90` | Retention for `runtime_llm_calls` rows | | `LOG_TTL_DAYS` | `14` | Retention for `runtime_logs` (request summaries are always kept, never sampled, so this is the main volume knob). Unlike the other TTLs it is also applied to **existing** tables: at boot the ingester compares it with the table's TTL and runs `ALTER TABLE … MODIFY TTL` only when they differ; a failure is logged and retried next boot | +| `AUTTER_REDACT_VALUE_PATTERNS` | — | JSON array of extra regex sources masked in stored/forwarded text and attribute values (built-in secret/PII patterns always apply) | +| `AUTTER_REDACT_KEY_PATTERNS` | — | JSON array of extra regex sources for attribute keys whose values are masked wholesale | ## Coded errors and request summaries (1.5.0) diff --git a/packages/otlp-ingester/src/clickhouse.ts b/packages/otlp-ingester/src/clickhouse.ts index ea4ba91..0fd8ac9 100644 --- a/packages/otlp-ingester/src/clickhouse.ts +++ b/packages/otlp-ingester/src/clickhouse.ts @@ -13,6 +13,12 @@ import { requestRollupTableDDL, type RuntimeLogRecord, } from "./logs.js"; +import { + ingesterInfoTableDDL, + sdkVersionTableDDL, + type SdkVersionRow, +} from "./sdk-versions.js"; +import type { SchemaStatus } from "./compat.js"; import { MIGRATIONS, OCCURRENCE_CODE_COLUMNS, @@ -49,6 +55,8 @@ function chDateTime(date: Date): string { export class ClickHouseStore { private client: ClickHouseClient | null = null; private ensurePromise: Promise | null = null; + private schemaState: Exclude = "pending"; + private appliedIds: string[] = []; constructor(private readonly config: IngesterConfig) {} @@ -93,6 +101,8 @@ export class ClickHouseStore { // existing runtime_logs lacks until migrations run, so the view is // created by migration 0014 alone (migrations also run on fresh DBs). requestRollupTableDDL(db), + sdkVersionTableDDL(db, metricsTtlDays), + ingesterInfoTableDDL(db), `CREATE TABLE IF NOT EXISTS ${db}.runtime_error_occurrences ( org_id String, repository_id String, @@ -236,8 +246,11 @@ export class ClickHouseStore { console.log(`clickhouse migration applied: ${migration.id}`); } await this.applyLogTtl(); + this.appliedIds = migrations.map((migration) => migration.id); + this.schemaState = "ready"; })().catch((err) => { this.ensurePromise = null; + this.schemaState = "failed"; throw err; }); return this.ensurePromise; @@ -321,6 +334,51 @@ export class ClickHouseStore { ); } + /** Schema state as of the last ensureSchema attempt — never triggers DDL + * (it backs the unauthenticated /v1/compat endpoint). */ + schemaStatus(): { status: SchemaStatus; applied: string[] } { + if (!this.configured) return { status: "unconfigured", applied: [] }; + return { status: this.schemaState, applied: [...this.appliedIds] }; + } + + async insertSdkVersions(ctx: IngestContext, rows: SdkVersionRow[]): Promise { + if (!rows.length || !this.configured) return; + await this.ensureSchema(); + await this.getClient().insert({ + table: this.table("runtime_sdk_versions"), + format: "JSONEachRow", + clickhouse_settings: INSERT_SETTINGS, + values: rows.map((row) => ({ + org_id: ctx.orgId, + repository_id: ctx.repositoryId, + service: row.service, + environment: row.environment, + sdk_name: row.sdkName, + sdk_version: row.sdkVersion, + sdk_language: row.sdkLanguage, + ingester_version: row.ingesterVersion, + last_seen: row.seenAt.toISOString(), + })), + }); + } + + /** Record this ingester's version and schema level (global, not tenant data). */ + async insertIngesterInfo(info: { version: string; schemaLevel: string; report: unknown }): Promise { + if (!this.configured) return; + await this.ensureSchema(); + await this.getClient().insert({ + table: this.table("runtime_ingester_info"), + format: "JSONEachRow", + clickhouse_settings: INSERT_SETTINGS, + values: [{ + ingester_version: info.version, + schema_level: info.schemaLevel, + report: JSON.stringify(info.report), + reported_at: new Date().toISOString(), + }], + }); + } + async ping(): Promise { if (!this.configured) return false; const result = await this.getClient().ping(); diff --git a/packages/otlp-ingester/src/coded-errors.test.ts b/packages/otlp-ingester/src/coded-errors.test.ts index 8bb4764..081d65c 100644 --- a/packages/otlp-ingester/src/coded-errors.test.ts +++ b/packages/otlp-ingester/src/coded-errors.test.ts @@ -402,14 +402,15 @@ test("migrations are uniquely ordered and 0012–0014 are additive and idempoten assert.equal(new Set(ids).size, ids.length); const serials = ids.map((id) => Number(id.slice(0, 4))); assert.deepEqual(serials, serials.map((_, i) => i + 1)); - assert.deepEqual(ids.slice(-3), ["0012-runtime-logs-requests", "0013-occurrence-codes", "0014-runtime-request-1m"]); - for (const migration of MIGRATIONS.slice(-3)) { + const requestMigrations = MIGRATIONS.filter((m) => /^001[2-4]-/.test(m.id)); + assert.deepEqual(requestMigrations.map((m) => m.id), ["0012-runtime-logs-requests", "0013-occurrence-codes", "0014-runtime-request-1m"]); + for (const migration of requestMigrations) { for (const statement of migration.statements) { assert.match(statement, /IF NOT EXISTS/, statement); assert.doesNotMatch(statement, /DROP|MODIFY/, statement); } } - const [logs, occ, rollup] = MIGRATIONS.slice(-3).map((m) => m.statements.join("\n")); + const [logs, occ, rollup] = requestMigrations.map((m) => m.statements.join("\n")); for (const column of ["kind LowCardinality(String) DEFAULT ''", "request_id String DEFAULT ''", "route String DEFAULT ''", "status_code UInt16 DEFAULT 0", "error_code String DEFAULT ''", "ai_cost_usd Float64 DEFAULT 0", "ai_calls UInt32 DEFAULT 0", "INDEX IF NOT EXISTS idx_logs_request_id request_id TYPE bloom_filter GRANULARITY 4"]) { diff --git a/packages/otlp-ingester/src/compat-manifest.json b/packages/otlp-ingester/src/compat-manifest.json new file mode 100644 index 0000000..16f02b2 --- /dev/null +++ b/packages/otlp-ingester/src/compat-manifest.json @@ -0,0 +1,94 @@ +{ + "manifestVersion": 1, + "docs": "https://github.com/Autter-dev/autter-runtime/blob/main/docs/COMPATIBILITY.md", + "upgrade": { + "ingester": "docker pull ghcr.io/autter-dev/otlp-ingester:latest and restart it (ClickHouse migrations run at boot)", + "sdk": "npm install {package}@latest" + }, + "features": [ + { + "id": "llm_calls", + "title": "LLM call tracking", + "ingester": "1.1.0", + "migrations": ["0004-llm-calls"], + "sdks": { "@autter/runtime-node": "1.1.0", "@autter/runtime-next": "1.1.0" } + }, + { + "id": "endpoint_latency", + "title": "Endpoint latency regression detection", + "ingester": "1.3.1", + "migrations": ["0005-latency-histograms"], + "sdks": { "@autter/runtime-node": "1.3.0", "@autter/runtime-next": "1.3.0" } + }, + { + "id": "browser_network_events", + "title": "Browser network, timing and outcome capture", + "ingester": "1.3.2", + "migrations": [], + "sdks": { "@autter/runtime-browser": "1.3.2", "@autter/runtime-next": "1.3.2" }, + "browserEvents": ["request_failure", "timing", "outcome"] + }, + { + "id": "profiles", + "title": "Profile ingestion", + "ingester": "1.3.2", + "migrations": ["0006-profile-samples"], + "route": "/v1/profiles", + "sdks": {} + }, + { + "id": "source_maps", + "title": "Source map upload", + "ingester": "1.3.2", + "migrations": ["0007-source-maps"], + "route": "/v1/sourcemaps", + "sdks": {} + }, + { + "id": "memory_metrics", + "title": "Memory pressure detection", + "ingester": "1.3.3", + "migrations": ["0008-memory-signals", "0010-memory-temporality"], + "sdks": { "@autter/runtime-node": "1.3.3", "@autter/runtime-next": "1.3.3" } + }, + { + "id": "platform_events", + "title": "Platform OOM/restart events", + "ingester": "1.3.3", + "migrations": ["0008-memory-signals"], + "route": "/v1/platform-events", + "sdks": {} + }, + { + "id": "csp_violations", + "title": "Browser CSP violation capture", + "ingester": "1.3.4", + "migrations": [], + "sdks": { "@autter/runtime-browser": "1.3.3", "@autter/runtime-next": "1.3.4" }, + "browserEvents": ["csp_violation"] + }, + { + "id": "operation_logging", + "title": "Operation logging", + "ingester": "1.4.0", + "migrations": ["0011-runtime-logs"], + "route": "/v1/logs", + "sdks": { "@autter/runtime-node": "1.4.0", "@autter/runtime-next": "1.4.0" } + }, + { + "id": "request_summaries", + "title": "Request summaries and the per-route request rollup", + "ingester": "1.5.0", + "migrations": ["0012-runtime-logs-requests", "0014-runtime-request-1m"], + "route": "/v1/logs", + "sdks": { "@autter/runtime-node": "1.5.0", "@autter/runtime-next": "1.5.0", "@autter/runtime-edge": "1.0.0" } + }, + { + "id": "coded_errors", + "title": "Coded errors and code-based grouping", + "ingester": "1.5.0", + "migrations": ["0013-occurrence-codes"], + "sdks": { "@autter/runtime-node": "1.5.0", "@autter/runtime-next": "1.5.0", "@autter/runtime-browser": "1.4.0", "@autter/runtime-edge": "1.0.0" } + } + ] +} diff --git a/packages/otlp-ingester/src/compat.test.ts b/packages/otlp-ingester/src/compat.test.ts new file mode 100644 index 0000000..f2b0ae7 --- /dev/null +++ b/packages/otlp-ingester/src/compat.test.ts @@ -0,0 +1,137 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { test } from "node:test"; +import { + buildCompatReport, + COMPAT_MANIFEST, + compareVersions, + evaluateCompat, + featuresForBrowserEvents, + ingesterInfoFromReport, + versionAtLeast, +} from "./compat.js"; +import { MIGRATIONS } from "./migrations.js"; +import { browserPayloadSchema } from "./normalize-browser.js"; + +const ALL = MIGRATIONS.map((m) => m.id); +const pkgVersion = (dir: string) => + (JSON.parse(readFileSync(new URL(`../../${dir}/package.json`, import.meta.url), "utf8")) as { version: string }).version; + +test("version comparison handles v-prefixes, prereleases and junk", () => { + assert.equal(compareVersions("1.4.0", "1.4.0"), 0); + assert.equal(compareVersions("v1.10.0", "1.9.9"), 1); + assert.equal(compareVersions("1.3.4", "1.4.0"), -1); + assert.equal(compareVersions("1.4.0-rc.1", "1.4.0"), -1); + assert.equal(compareVersions("latest", "1.4.0"), null); + assert.equal(versionAtLeast(null, "1.0.0"), null); + assert.equal(versionAtLeast("2.0.0", "1.4.0"), true); +}); + +test("manifest is consistent with migrations, routes and released package versions", () => { + const ids = new Set(); + const server = readFileSync(new URL("./server.ts", import.meta.url), "utf8"); + for (const feature of COMPAT_MANIFEST.features) { + assert.ok(!ids.has(feature.id), `duplicate ${feature.id}`); + ids.add(feature.id); + assert.ok(compareVersions(feature.ingester, pkgVersion("otlp-ingester"))! <= 0, `${feature.id}: ingester ${feature.ingester} is unreleased`); + for (const migration of feature.migrations) assert.ok(ALL.includes(migration), `${feature.id}: unknown migration ${migration}`); + if (feature.route) assert.ok(server.includes(`app.post("${feature.route}"`), `${feature.id}: no route ${feature.route}`); + for (const [pkg, min] of Object.entries(feature.sdks)) { + const dir = pkg.replace("@autter/", ""); + assert.ok(compareVersions(min, pkgVersion(dir))! <= 0, `${feature.id}: ${pkg} ${min} is unreleased`); + } + for (const type of feature.browserEvents ?? []) { + const parsed = browserPayloadSchema.safeParse({ + version: 1, service: "s", environment: "e", + events: [{ type, timestamp: new Date().toISOString() }], + }); + assert.ok(parsed.success, `${feature.id}: browser event ${type} not accepted`); + } + } +}); + +test("operation logging on an older ingester names both versions and the fix", () => { + const issues = evaluateCompat({ + features: ["operation_logging", "memory_metrics"], + ingester: { version: "1.3.4" }, + }); + assert.equal(issues.length, 1); + const [issue] = issues; + assert.equal(issue!.kind, "ingester_too_old"); + assert.equal(issue!.required, "1.4.0"); + assert.equal(issue!.actual, "1.3.4"); + assert.match(issue!.message, /^Operation logging needs ingester >= 1\.4\.0; yours is 1\.3\.4\. Upgrade the ingester: .*otlp-ingester/); + assert.match(issue!.message, /COMPATIBILITY\.md/); +}); + +test("no issues when compatible, unknown, or for unknown feature ids", () => { + const ready = { status: "ready" as const, applied: ALL }; + assert.deepEqual(evaluateCompat({ features: ["operation_logging", "csp_violations"], ingester: { version: "1.4.0", schema: ready } }), []); + assert.deepEqual(evaluateCompat({ features: ["operation_logging"], ingester: null }), []); + assert.deepEqual(evaluateCompat({ features: ["operation_logging"], ingester: { version: null, legacy: true } }), []); + assert.deepEqual(evaluateCompat({ features: ["no_such_feature"], ingester: { version: "0.1.0" } }), []); + assert.deepEqual( + evaluateCompat({ features: ["operation_logging"], ingester: { version: "1.4.0", schema: { status: "pending", applied: [] } } }), + [], + ); +}); + +test("legacy ingesters are flagged only when a route probe proved the route missing", () => { + const issues = evaluateCompat({ + features: ["operation_logging", "memory_metrics"], + ingester: { version: null, legacy: true, routes: { "/v1/logs": false } }, + }); + assert.equal(issues.length, 1); + assert.match(issues[0]!.message, /1\.4\.0 or older .*no \/v1\/logs route/); + assert.deepEqual( + evaluateCompat({ features: ["operation_logging"], ingester: { version: null, legacy: true, routes: { "/v1/logs": true } } }), + [], + ); +}); + +test("unapplied or failed migrations are reported with a schema fix", () => { + const missing = evaluateCompat({ + features: ["operation_logging"], + ingester: { version: "1.4.0", schema: { status: "ready", applied: ALL.filter((id) => id !== "0011-runtime-logs") } }, + }); + assert.equal(missing[0]!.kind, "schema_not_applied"); + assert.match(missing[0]!.message, /0011-runtime-logs/); + const failed = evaluateCompat({ + features: ["memory_metrics"], + ingester: { version: "1.4.0", schema: { status: "failed", applied: [] } }, + }); + assert.match(failed[0]!.message, /schema failed.*CLICKHOUSE_URL/); +}); + +test("SDK too old for a feature is reported against that SDK package", () => { + const issues = evaluateCompat({ + features: ["operation_logging"], + ingester: { version: "1.4.0" }, + sdk: { name: "@autter/runtime-next", version: "1.3.4" }, + }); + assert.equal(issues.length, 1); + assert.equal(issues[0]!.kind, "sdk_too_old"); + assert.match(issues[0]!.message, /npm install @autter\/runtime-next@latest/); +}); + +test("browser event types map to their features", () => { + assert.deepEqual(featuresForBrowserEvents(["exception", "csp_violation"]), ["csp_violations"]); + assert.deepEqual(featuresForBrowserEvents(["timing"]), ["browser_network_events"]); +}); + +test("compat report marks availability from version and schema state, and round-trips", () => { + const pending = buildCompatReport({ version: "1.4.0", schemaStatus: "pending", applied: [], allMigrations: ALL }); + const logs = pending.features.find((f) => f.id === "operation_logging")!; + assert.equal(logs.ingesterSupported, true); + assert.equal(logs.available, false); + assert.equal(pending.features.find((f) => f.id === "csp_violations")!.available, true); + const ready = buildCompatReport({ version: "1.5.0", schemaStatus: "ready", applied: ALL, allMigrations: ALL }); + assert.equal(ready.schema.level, ALL[ALL.length - 1]); + assert.ok(ready.features.every((f) => f.available)); + assert.deepEqual(ingesterInfoFromReport(JSON.parse(JSON.stringify(ready))), { + version: "1.5.0", + schema: { status: "ready", applied: ALL }, + }); + assert.equal(ingesterInfoFromReport({ ingester: { version: "