Add in-enclave moderation of image requests via OpenAI omni-moderation

[adambalogh]

Why

OpenAI notified us that child sexual abuse material has been attempted through our API key. Prompts are only visible in plaintext inside the enclave (the OHTTP path is encrypted browser↔TEE), so this is the only place content moderation can run.

Scope: image requests only

This PR moderates image requests only — image generation, image editing (reference images ride the same request), and the inline-image chat models. Plain text chat carries no moderation code path at all: no scoring call, no verdict, no headers, no added latency. Extending moderation to text chat is a deliberate future PR; the one gate to widen is the should_moderate_model predicate in moderation.py (everything downstream — relay strike policy, app warnings — keys off the flag headers and needs no change).

What

In-scope requests are scored against OpenAI's free omni-moderation-latest endpoint (tee_gateway/moderation.py, modeled on web_search.py) before any provider is called. The check covers the newest user turn: its prompt text plus up to 5 attached images (data URIs / URLs).

• Blocking: a request flagged for a category in BLOCKED_CATEGORIES (default: sexual/minors only) is refused with HTTP 451 + code: "moderation_blocked" and never forwarded to a model provider. All other flagged categories are reported but still served — the relay's strike policy handles repeat offenders.
• Sealed verdict: the full result (flagged / blocked / categories / scores) rides inside the encrypted response body under a new moderation key — non-streaming responses and the final SSE frame — outside the signed output hash, exactly like images and usage, so existing signature verifiers are unaffected.
• Relay signal: flagged requests additionally carry content-free X-Moderation-Flagged / X-Moderation-Categories / X-Moderation-Blocked headers, forwarded through the OHTTP path (new x-moderation forward prefix). This is a deliberate, narrow exception to the "relay learns nothing" stance: the relay operates the user-facing service and needs a per-request abuse bit to run its strike policy (see the companion chat-api PR). The headers never carry prompt content or scores, and clean traffic is byte-identical to before.
• Fail-open: no OpenAI key or a moderation-endpoint failure means the request proceeds unscored; a positive verdict always comes from a real moderation response. /health and the /v1/keys response report moderation_enabled.
• Billing: the moderation endpoint is free; no cost-block changes. Blocked (451) requests produce no opengradient block and are never settled.

Dedicated moderation client reuses the injected OpenAI key with a tight (15s total) timeout. The added round-trip lands only on image requests, which already take multiple seconds.

Testing

• tee_gateway/test/test_moderation.py (23 tests): availability, input extraction, verdicts, header shapes, image-only scope, all failure modes fail-open, controller 451 short-circuit, OHTTP header forwarding
• tee_gateway/test/test_moderation_e2e.py (11 wire tests + 2 opt-in live): real connexion app end to end — image requests moderated (clean/flagged/blocked/outage/streaming), text chat proven untouched (moderation endpoint never consulted) on both the direct and sealed OHTTP paths; live tests against the real moderation endpoint gated on RUN_PROVIDER_INTEGRATION_TESTS=1
• Full CI suite green (383 passed); make lint (ruff + mypy) clean

Companion PRs

• opengradient/chat-api — strike counting + blacklist (ban enforcement off by default: observation mode)
• opengradient/chat-app — warning banners + suspended-account state

[Copilot]

Pull request overview

Adds an in-enclave, fail-open moderation pre-flight for image-scope /v1/chat/completions requests using OpenAI’s omni-moderation-latest, surfacing a sealed verdict to clients and a minimal content-free abuse signal to the relay via forwarded headers.

Changes:

• Introduces tee_gateway/moderation.py (client config, scope gate, newest-user-turn extraction, verdict + headers) and wires it into chat + image-generation response paths (streaming and non-streaming).
• Extends OHTTP header forwarding to propagate X-Moderation-* headers and adds /health + /v1/keys reporting for moderation availability.
• Adds comprehensive unit + wire/E2E tests and updates OpenAPI schema + repo documentation.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tee_gateway/moderation.py	New moderation client + scope predicate + request extraction + verdict/headers logic.
tee_gateway/controllers/chat_controller.py	Runs moderation pre-flight for in-scope models; blocks on configured categories; attaches verdict/headers to responses.
tee_gateway/image_generation.py	Threads moderation verdict/headers through image-generation response helpers (JSON + SSE).
tee_gateway/controllers/ohttp_controller.py	Forwards x-moderation* response headers through the relay.
tee_gateway/llm_backend.py	Configures/tears down moderation HTTP client alongside provider clients on key injection.
tee_gateway/__main__.py	Exposes moderation_enabled in /health and /v1/keys response.
tee_gateway/openapi/openapi.yaml	Documents optional moderation block on chat completion responses.
tee_gateway/models/create_chat_completion_response.py	Extends minimal response holder to include moderation.
tee_gateway/test/test_moderation.py	Unit tests for moderation module behavior + controller short-circuiting.
tee_gateway/test/test_moderation_e2e.py	End-to-end tests across direct + streaming + OHTTP paths (including live opt-in).
CLAUDE.md	Documents moderation behavior, scope, headers, and failure/billing semantics.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.