feat(otel): client-side redaction of PII and/or secrets

[simonvdk-mistral]

What

Adds a client-side redaction layer for OpenTelemetry spans so PII and secrets never leave the machine. The core primitive is reusable by any OTEL application, and the Mistral SDK installs it automatically when it owns the exporter.

Also add documentation and examples for the observability features of the SDK.

The primitive

RedactingSpanExporter wraps any SpanExporter and redacts each span before delegating the actual export:

from mistralai.extra.observability.redaction import RedactingSpanExporter

exporter = RedactingSpanExporter(OTLPSpanExporter(...))          # default policy
provider.add_span_processor(BatchSpanProcessor(exporter))

Redaction covers the whole span surface: attributes, events, links, resource attributes, span name, and status description.

The module requires the optional OpenTelemetry SDK (the telemetry extra) to run, but not to import — so it stays importable in environments without the extra.

Why an exporter wrapper (and not span-creation-time redaction)

Redaction is deliberately placed in a SpanExporter decorator, invoked at export time, rather than when spans/attributes are created. In dedicated mode the SDK wires:

BatchSpanProcessor(RedactingSpanExporter(OTLPSpanExporter(...)))

This has real, concrete benefits:

• Off the request hot path. With BatchSpanProcessor (what the SDK installs), export runs on a dedicated background thread, in batches. The regex scanning and span rebuild happen there — not on the threads serving the user's chat/embeddings calls — so redaction adds no latency to application requests. Cost is amortized across a batch and absorbed by the exporter thread; under extreme load it manifests as export backpressure, never as slower API calls.
• Last line of defense. Sitting at the very edge, right before bytes leave the process, it sees every span regardless of which instrumentation produced it. Nothing is exported without passing through redaction (fail-closed).
• Composable and reusable. It's a plain SpanExporter decorator with no Mistral coupling, so any OpenTelemetry application can wrap its own exporter — which is exactly what users do in global/custom-provider mode.

Caveat: the "off the hot path" property depends on an asynchronous processor. BatchSpanProcessor (installed by the SDK) gives it; a SimpleSpanProcessor would export — and therefore redact — synchronously on span end.

Policies

Policy	Strategy	Trade-off
RegexRedactionPolicy (default, redaction=True)	Content-oriented: keeps keys and structure, redacts matched substrings (secret tokens plus PII — emails, card-like sequences, IPv4).	Redacts most sensitive data while preserving observability value; may miss free-form PII or secrets not in the pattern set.
AttributeRedactionPolicy	Key-oriented: redacts whole values for sensitive keys (explicit set, fragment match, or non-primitive value), then scans kept values for secret token patterns.	Very conservative, but erases most prompt/response content.
CallbackRedactionPolicy (redaction=<callable>)	Your (key, value) -> value | None masker per attribute; return None to drop the attribute.	Full control; you own the logic.

SDK integration

configure_telemetry gains a redaction argument:

• True (default) — default (regex) policy
• False — redaction disabled
• a RedactionPolicy instance (e.g. AttributeRedactionPolicy())
• a (key, value) -> value | None callback

Redaction only applies in dedicated provider mode, where the SDK owns the exporter. In global/custom-provider modes the application owns the export pipeline, so the argument is ignored and a warning is logged — wrap your own exporter with RedactingSpanExporter in that case.

Tests

• New test_redaction.py covering the policies, span rebuild, and the exporter
  wrapper.
• Extended test_telemetry.py for the redaction wiring and the
  ignored-argument warnings.

[gitguardian-eu]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[simonvdk-mistral]

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id GitGuardian status Secret Commit Filename
289842 Triggered Shopify Generic App Token e8f0122 src/mistralai/extra/tests/test_redaction.py View secret
289840 Triggered PostHog Project API Key e8f0122 src/mistralai/extra/tests/test_redaction.py View secret
🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

These are all fake/dummy secrets to test redaction

[rbarbadillo]

generally lgtm! just a small edge-case to cover