feat(google): pass Gemini agentic video through instead of flattening it

docs-site/src/content/docs/reference/adapters.md

@@ -282,6 +282,31 @@ freeform call echoed without its `</function>` close counts as complete once
   `functionResponse` per representable call. Interrupted histories receive an explicit missing-result marker;
   duplicate or standalone results are preserved as marked text (and image siblings) rather than
   emitted as invalid unpaired `functionResponse` parts.
+- **Video input and agentic processing.** The OpenAI-compatible content part
+  `{"type": "video_url", "video_url": {"url": "…", "processing": "agentic"}}` is accepted on both
+  the Chat and Responses ingress routes. `url` is required; `processing` is optional and is
+  upper-cased onto the Gemini part as `media_processing` (`STATIC` is Gemini's default, `AGENTIC`
+  requests agentic video understanding). The field sits on the **part**, beside `inline_data` or
+  `file_data`, so it applies to inline bytes and fetched URIs alike — it is not the Interactions
+  API's `processing`. A request that omits `processing` gains no field, so existing callers are
+  unchanged.
+
+  Three URL forms are handled, and only three:
+
+  | `url` | sent as |
+  | --- | --- |
+  | `data:` URL | `inline_data` with the data URL's own media type |
+  | YouTube watch URL (`youtube.com`, `youtu.be`, `m.`/`music.`/`-nocookie` variants) | `file_data.file_uri` |
+  | `https://generativelanguage.googleapis.com/<version>/files/<id>` — the Files API resource form, where `<version>` is `v1`, `v1beta` or `v1alpha` | `file_data.file_uri` |
+
+  Any other remote URL is kept as the text marker `[video: <url>]`, because the adapter has no
+  media type for it and no evidence Gemini will fetch it. The allowlist is matched on the parsed
+  URL's host and path over HTTPS — not on a substring — so a look-alike host does not become a
+  `file_data` reference the proxy asks Gemini to fetch. The path is anchored at the start, so the
+  resumable-upload endpoint (`/upload/<version>/files/<id>`) is *not* accepted: it is not a
+  readable resource, and `file_data.file_uri` asks Gemini to dereference what it is given. `file_data` carries `file_uri` only; no
+  guessed `mime_type` is attached.
+
 - **Inline image output:** when the model is one of the explicit image-capable chat IDs
   (`gemini-3.1-flash-image`, `gemini-2.0-flash-preview-image-generation`, or
   `gemini-3-pro-image-preview`), the adapter sends `responseModalities: ["TEXT", "IMAGE"]`.

src/adapters/google.ts

@@ -231,6 +231,72 @@ function toolResultImageParts(content: string | OcxContentPart[]): unknown[] {
  * surfaced on Claude-on-Antigravity; the guard lives here because this is where the parts are
  * built. Mirrors the Anthropic adapter's own empty-block guard (src/adapters/anthropic.ts).
  */
+/**
+ * A video URI Gemini fetches on its own behalf, as a `file_data` reference.
+ *
+ * Deliberately an allowlist of the forms Google documents, not "anything that is
+ * not a data: URL". `file_data` tells Gemini to go and get the bytes; pointing it
+ * at an arbitrary host would either fail upstream or make the proxy the reason a
+ * caller's private URL got dereferenced by Google. Anything not matched here
+ * keeps the existing `[video: …]` text marker.
+ *
+ * Returns the uri alone: the documented REST example for a YouTube part carries
+ * `file_data.file_uri` and nothing else, and the Files API knows the type of what
+ * it stored. An invented `mime_type` would be a guess on both paths.
+ *
+ * https://ai.google.dev/gemini-api/docs/generate-content/video-understanding
+ */
+function geminiFetchableVideoUri(url: string): string | null {
+  let parsed: URL;
+  try {
+    parsed = new URL(url);
+  } catch {
+    return null;
+  }
+  if (parsed.protocol !== "https:") return null;
+
+  const host = parsed.hostname.toLowerCase();
+  const youtubeHosts = new Set([
+    "youtube.com",
+    "www.youtube.com",
+    "m.youtube.com",
+    "music.youtube.com",
+    "youtu.be",
+    "www.youtube-nocookie.com",
+    "youtube-nocookie.com",
+  ]);
+  if (youtubeHosts.has(host)) return url;
+
+  // The Files API resource form, https://generativelanguage.googleapis.com/v1beta/files/<id>.
+  // Anchored at the start so the resumable-upload endpoint (/upload/v1beta/files/<id>) does
+  // not match: that URL is not a readable resource, and passing it as `file_data.file_uri`
+  // would have Gemini dereference something it cannot read. The version segment stays loose
+  // because this service is reachable as v1, v1beta and v1alpha.
+  if (host === "generativelanguage.googleapis.com" && /^\/v1[a-z0-9]*\/files\/[^/]+$/.test(parsed.pathname)) {
+    return url;
+  }
+
+  return null;
+}
+
+/**
+ * The caller's requested video mode as GenerateContent spells it.
+ *
+ * `media_processing` sits on the part beside `inline_data`/`file_data` and takes
+ * `STATIC` (the default) or `AGENTIC`. `processing: "agentic"` — the spelling in
+ * the original request and in Google's Interactions API — is a different API and
+ * is ignored here, so forwarding it verbatim would have looked like a
+ * pass-through while agentic mode never actually engaged.
+ *
+ * Upper-cased and forwarded rather than checked against our own copy of the enum:
+ * that list is Google's to extend, and a stale allowlist here would silently
+ * downgrade a caller using a newer mode. An unrecognized value fails upstream
+ * naming the field, which is a better failure than us dropping it.
+ */
+function geminiMediaProcessing(processing: string | undefined): string | undefined {
+  return processing ? processing.toUpperCase() : undefined;
+}
+
 const GEMINI_EMPTY_PLACEHOLDER = "(empty)";
 const GEMINI_EMPTY_TOOL_OUTPUT_PLACEHOLDER = "(empty tool output)";
 const GEMINI_MISSING_TOOL_RESULT = "[missing tool_result for this tool_use in history]";
@@ -341,10 +407,34 @@ function messagesToGeminiFormat(
               continue;
             }
             if (p.type === "video") {
+              // `media_processing` rides on the PART, so it applies to inline bytes
+              // exactly as it does to a fetched uri — emitting it on only one of the
+              // two would silently drop the mode for data: URLs.
+              const mediaProcessing = geminiMediaProcessing(p.processing);
+              const processingPart = mediaProcessing ? { media_processing: mediaProcessing } : {};
+
+              // Gemini accepts inline video bytes in the same Part union as images.
               const data = parseDataUrl(p.videoUrl);
-              // Gemini accepts inline video bytes in the same Part union as images. Arbitrary
-              // remote URLs are not valid fileData references, so retain only a short marker.
-              parts.push(data ? { inline_data: { mime_type: data.mediaType, data: data.base64 } } : { text: `[video: ${p.videoUrl}]` });
+              if (data) {
+                parts.push({
+                  inline_data: { mime_type: data.mediaType, data: data.base64 },
+                  ...processingPart,
+                });
+                continue;
+              }
+              // Two URI forms Gemini fetches itself: a YouTube watch URL and a Files API
+              // uri. Those ARE valid file_data references (#3271), and flattening them to
+              // a text marker was the whole reason agentic video could not be reached —
+              // the video never arrived as a video. Every other remote URL keeps the
+              // marker: we have no mime type for it and no evidence Gemini can fetch it.
+              const fileUri = geminiFetchableVideoUri(p.videoUrl);
+              if (fileUri) {
+                // Emitted only when the caller asked for a mode, so no existing
+                // request gains a field it did not have.
+                parts.push({ file_data: { file_uri: fileUri }, ...processingPart });
+                continue;
+              }
+              parts.push({ text: `[video: ${p.videoUrl}]` });
               continue;
             }
             if (p.type === "document") {

src/chat/inbound.ts

@@ -57,11 +57,21 @@ function contentToText(content: unknown): string {
 // route-eligibility predicate and this translator cannot drift apart again.
 const imageUrlFromPart = chatImageUrlFromPart;
 
-function videoUrlFromPart(part: Rec): string | null {
+/**
+ * A caller's video part, with the `processing` mode Gemini's agentic video
+ * understanding is requested by (#3271). The object form is the only one that
+ * can carry it — `video_url` as a bare string has nowhere to put it.
+ */
+function videoFromPart(part: Rec): { url: string; processing?: string } | null {
   if (part.type !== "video_url") return null;
   const videoUrl = part.video_url;
-  if (typeof videoUrl === "string" && videoUrl.length > 0) return videoUrl;
-  if (isRec(videoUrl) && typeof videoUrl.url === "string" && videoUrl.url.length > 0) return videoUrl.url;
+  if (typeof videoUrl === "string" && videoUrl.length > 0) return { url: videoUrl };
+  if (isRec(videoUrl) && typeof videoUrl.url === "string" && videoUrl.url.length > 0) {
+    const processing = typeof videoUrl.processing === "string" && videoUrl.processing.length > 0
+      ? videoUrl.processing
+      : undefined;
+    return { url: videoUrl.url, ...(processing ? { processing } : {}) };
+  }
   return null;
 }
 
@@ -91,9 +101,13 @@ function userContentToBlocks(content: unknown): Rec[] {
       });
       continue;
     }
-    const videoUrl = videoUrlFromPart(raw);
-    if (videoUrl) {
-      blocks.push({ type: "input_video", video_url: videoUrl });
+    const video = videoFromPart(raw);
+    if (video) {
+      blocks.push({
+        type: "input_video",
+        video_url: video.url,
+        ...(video.processing ? { processing: video.processing } : {}),
+      });
       continue;
     }
     const file = fileFromPart(raw);

src/responses/parser-content.ts

@@ -9,7 +9,7 @@ type InputBlock =
   | { type: "input_text"; text: string }
   | { type: "text"; text: string }
   | { type: "input_image"; image_url?: string; file_id?: string; detail?: string }
-  | { type: "input_video"; video_url?: string }
+  | { type: "input_video"; video_url?: string; processing?: string }
   // codex-rs protocol/src/models.rs sends audio as input_audio with an audio_url.
   | { type: "input_audio"; audio_url?: string; format?: string }
   | { type: "input_file"; file_id?: string; filename?: string; file_data?: string };
@@ -62,7 +62,8 @@ export function inputContentParts(blocks: unknown): string | OcxContentPart[] {
       // the request never carried, which is worse than dropping malformed input.
     } else if (block.type === "input_video") {
       const videoUrl = nonEmptyString(block.video_url);
-      if (videoUrl) parts.push({ type: "video", videoUrl });
+      const processing = nonEmptyString((block as { processing?: string }).processing);
+      if (videoUrl) parts.push({ type: "video", videoUrl, ...(processing ? { processing } : {}) });
     } else if (block.type === "input_audio") {
       // Upstream Codex sends input_audio with an audio_url (codex-rs
       // protocol/src/models.rs). The IR has no audio carrier and no adapter consumes

src/responses/schema.ts

@@ -14,6 +14,10 @@ const inputImageBlockSchema = z.object({
 const inputVideoBlockSchema = z.object({
   type: z.literal("input_video"),
   video_url: z.string().min(1),
+  // Gemini agentic video understanding (#3271). z.object() strips unknown keys,
+  // so without declaring it here the mode is dropped before any adapter sees it
+  // and the request silently degrades to frame-by-frame decoding.
+  processing: z.string().min(1).optional(),
 });
 const inputFileBlockSchema = z.object({
   type: z.literal("input_file"),

src/types/request.ts

@@ -221,8 +221,17 @@ export interface OcxImageContent {
 
 export interface OcxVideoContent {
   type: "video";
-  /** A base64 `data:` URL from an OpenAI-compatible `video_url` part. */
+  /**
+   * A base64 `data:` URL from an OpenAI-compatible `video_url` part, or a URI
+   * the upstream can fetch itself (a YouTube watch URL, a Files API uri).
+   */
   videoUrl: string;
+  /**
+   * Gemini's agentic video mode, carried verbatim from the caller's
+   * `video_url.processing` (#3271). Absent for every request that does not ask
+   * for it, so no existing traffic gains a field.
+   */
+  processing?: string;
 }
 
 /**

structure/providers/google.md

@@ -150,3 +150,29 @@ per-turn ordinal lists; the serialized ceiling, held at half `MAX_DEBUG_LINE_BYT
 turn detail from the tail until the summary fits. Without the second, a worst case inside the
 first serializes past the debug buffer's per-line cap, and the buffer truncates at a byte
 boundary: the consumer gets unparseable JSON whose retained prefix still reads `truncated: false`.
+
+## Video part boundary and agentic media processing
+
+The inbound contract is the OpenAI-compatible content part
+`{ type: "video_url", video_url: { url, processing? } }`, normalized by both the Chat and
+Responses ingress into an internal `{ type: "video", videoUrl, processing? }` part. `processing`
+is caller-supplied and optional; nothing infers it.
+
+The outbound contract is a GenerateContent `contents[].parts[]` entry. `media_processing` is a
+**Part** field, not a request field and not the Interactions API's `processing`, so it is emitted
+beside `inline_data` and beside `file_data` alike — attaching it to only the fetched-URI branch
+would silently drop the mode for `data:` URLs, which is the shape the first revision of #3271 had.
+`geminiMediaProcessing` upper-cases the caller's value and returns `undefined` when absent, so a
+request that did not ask for a mode gains no field.
+
+`geminiFetchableVideoUri` is the trust boundary: it decides which URLs opencodex will ask Gemini
+to **fetch on its own behalf**. It parses the URL and requires HTTPS, then admits exactly two
+families — the YouTube watch hosts (`youtube.com`, `www.`/`m.`/`music.` variants, `youtu.be`, and
+the `-nocookie` forms) and `generativelanguage.googleapis.com` with a path matching
+`/files/<id>`. Matching is on the parsed host and pathname, never a substring of the URL, so a
+look-alike host cannot become a `file_data` reference. Everything else keeps the
+`[video: <url>]` text marker: without a media type there is nothing correct to send, and a
+fetchable reference the proxy cannot vouch for is the SSRF-shaped half of this feature.
+
+`file_data` carries `file_uri` only. An earlier revision guessed a `mime_type` for it; the Files
+API already knows the type of what it stores, and a wrong guess is worse than no guess.