feat(airt): multimodal prompt_matrix (text×media cross-product) + agent transparency (1.6.10)

capabilities/ai-red-teaming/agents/ai-red-teaming-agent.md

@@ -78,7 +78,8 @@ Keep it to a single line; don't pad it.
    - Agent/MCP/HTTP endpoint with tools → `generate_agentic_attack`
    - ML image classifier (perturb pixels to misclassify) → `generate_image_attack`
    - **Multimodal LLM (vision/audio/video) with media inputs → `generate_multimodal_attack`**. Detect this when the user attaches or points to media and wants to probe a chat/vision model: "attack this vision model", "run these prompts with the images in `./imgs`", "apply an image transform on the images", "test this voice model with the audio in this folder", "visual prompt injection", "typographic jailbreak". Pass `image_dir`/`audio_dir`/`video_dir` for folders or `image_paths`/`audio_paths`/`video_paths` for explicit files. Do NOT confuse with `generate_image_attack` (classifier evasion, not chat).
-2. IMMEDIATELY call `execute_workflow` with the filename returned by the generator. Skipping this leaves the assessment with 0 trials.
+2. IMMEDIATELY call `execute_workflow` with the filename returned by the generator, in the SAME turn — a `generate_*` call that returns "workflow generated / NEXT STEP: execute_workflow" is NOT done. Never stop after generating; skipping execution leaves the assessment with 0 trials and looks like a silent failure to the user.
+   - **Narrate every step — no black box.** Before each tool call, say what you're about to do and why (e.g. "Generating the workflow… now executing 20 trials (4 prompts × 5 images)… scoring with gpt-4o-mini…"). After execution, report the assessment ID and how many trials ran. If a step errors or a target returns no response, say so plainly (e.g. "target call failed: 401 auth — provider key missing") instead of moving on silently.
 3. Call `register_assessment`, then `update_assessment_status` once execution finishes.
 4. Call `validate_attack_results` FIRST. If it surfaces errors, stop and report them — do not call analytics tools.
 5. If validation passes, call `get_assessment_status` for platform metrics and report ONLY those raw values.
@@ -453,6 +454,7 @@ planning; the tool loads and probes the media at runtime inside the workflow.
 | n_iterations | No | Iterations per media file (default 4). |
 | prompts | No | Per-media prompts aligned with media order (images, then audio, then video). |
 | prompts_csv | No | Path to a `media_filename,prompt` CSV; each media is paired by basename (unmapped → `goal`). |
+| prompt_matrix | No | **Cross-product** text prompts: run EVERY prompt against EVERY media item → N×M trials (e.g. 4 prompts × 5 images = 20). Transforms apply to every combination. Distinct from `prompts`/`prompts_csv` (1:1 pairing). |
 | custom_url | No | Target a custom multimodal HTTP endpoint instead of a litellm model. |
 | custom_auth_type | No | Auth for `custom_url`: `none`, `bearer`, or `api_key`. |
 | custom_auth_env_var | No | Env var holding the endpoint credential (default `TARGET_API_KEY`). |
@@ -489,6 +491,13 @@ and `custom_response_text_path` (JSONPath to the reply text), then call with `cu
 user describes different intents per file/folder), pass `prompts_csv` (matched by basename) or an
 explicit `prompts` list aligned with media order. Otherwise a single `goal` is used for every set.
 
+**Cross-product / matrix (every prompt × every image).** When the user wants *all combinations* of
+several text prompts and several media — e.g. "I have 4 prompts and 5 images, run all 20" — pass
+`prompt_matrix=[...]` (the list of text prompts). Each prompt is run against every media item, so N
+prompts × M media = **N×M trials** (transforms apply to each). Do NOT use `prompts`/`prompts_csv`
+for this — those pair one prompt per media (1:1, giving only M trials). Tell the user the exact
+count up front (e.g. "4 prompts × 5 images = 20 trials").
+
 **Media-OUTPUT scoring.** When the target *generates* media — an image-out model (e.g.
 `gemini-2.5-flash-image`) or a speech-to-speech target — set `score_media_output=True` so the
 generated image/audio/video is scored by a media-aware judge (not just the text). The trial score

capabilities/ai-red-teaming/capability.yaml

@@ -1,6 +1,6 @@
 schema: 1
 name: ai-red-teaming
-version: "1.6.9"
+version: "1.6.10"
 description: >
   Probe the security and safety of AI applications, agents, and foundation models.
   Orchestrates adversarial attack workflows to discover vulnerabilities in LLMs,

capabilities/ai-red-teaming/scripts/attack_runner.py

@@ -5493,12 +5493,35 @@ def _load(paths, cls):
     sys.stdout.flush()
 
     async with assessment.trace():
-        for i in range(n_sets):
-            img = images[i] if i < len(images) else None
-            aud = audios[i] if i < len(audios) else None
-            vid = videos[i] if i < len(videos) else None
-            goal_for_set = PROMPTS[i] if (PROMPTS and i < len(PROMPTS)) else GOAL
-            print(f"[{{i + 1}}/{{n_sets}}] multimodal attack "
+        # Build the (prompt, image, audio, video) sets to run. Default is 1:1 by
+        # media index (each media set uses prompt[index] or GOAL). When PROMPT_MATRIX
+        # is set, run EVERY prompt against EVERY media set — a full cross product of
+        # len(PROMPT_MATRIX) x n_sets trials (e.g. 4 prompts x 5 images = 20). Any
+        # configured TRANSFORMS are applied to every set either way.
+        def _media_at(media_idx):
+            return (
+                images[media_idx] if media_idx < len(images) else None,
+                audios[media_idx] if media_idx < len(audios) else None,
+                videos[media_idx] if media_idx < len(videos) else None,
+            )
+
+        media_sets = []
+        if PROMPT_MATRIX:
+            for matrix_prompt in PROMPT_MATRIX:
+                for media_idx in range(n_sets):
+                    media_sets.append((matrix_prompt, *_media_at(media_idx)))
+        else:
+            for media_idx in range(n_sets):
+                set_prompt = PROMPTS[media_idx] if (PROMPTS and media_idx < len(PROMPTS)) else GOAL
+                media_sets.append((set_prompt, *_media_at(media_idx)))
+
+        total_sets = len(media_sets)
+        print(f"Running {{total_sets}} multimodal set(s)"
+              + (f" (matrix: {{len(PROMPT_MATRIX)}} prompts x {{n_sets}} media)" if PROMPT_MATRIX else ""))
+        sys.stdout.flush()
+
+        for set_number, (goal_for_set, img, aud, vid) in enumerate(media_sets, start=1):
+            print(f"[{{set_number}}/{{total_sets}}] multimodal attack "
                   f"(image={{img is not None}}, audio={{aud is not None}}, video={{vid is not None}})")
             sys.stdout.flush()
             try:
@@ -5635,6 +5658,12 @@ def generate_multimodal_attack(params: dict) -> dict:
         goal,
     )
 
+    # Cross product: run EVERY prompt in the matrix against EVERY media set. With N
+    # matrix prompts and M media, this yields N*M trials (e.g. 4 prompts x 5 images
+    # = 20). Empty = the default 1:1-by-index behaviour above.
+    prompt_matrix = params.get("prompt_matrix") or []
+    prompt_matrix = [str(p) for p in prompt_matrix if str(p).strip()]
+
     # Media-OUTPUT scoring: when the target generates media (image-out / speech-to-
     # speech), score the generated modality with a media-aware judge and take the MAX
     # across text + media. Off by default (only the text response is scored).
@@ -5679,6 +5708,8 @@ def generate_multimodal_attack(params: dict) -> dict:
         "VIDEO_PATHS = {}".format(repr(video_paths)),
         # Per-media prompts aligned with media order; empty = single-goal behaviour.
         "PROMPTS = {}".format(repr(prompts_list)),
+        # Cross-product prompts: each is run against every media set (N x M trials).
+        "PROMPT_MATRIX = {}".format(repr(prompt_matrix)),
         # Output modalities to score with a media-aware judge (empty = text only).
         "MEDIA_OUTPUT_MODALITIES = {}".format(repr(media_out_modalities)),
         'MEDIA_OUTPUT_RUBRIC = "{}"'.format(_safe_str(media_output_rubric)),

capabilities/ai-red-teaming/tools/attacks.py

@@ -450,6 +450,13 @@ def generate_multimodal_attack(
         "Path to a CSV of `media_filename,prompt` rows. Each media file is paired with "
         "its prompt (matched by basename); unmapped media fall back to `goal`.",
     ] = "",
+    prompt_matrix: t.Annotated[
+        list[str] | None,
+        "Cross-product text prompts: run EVERY prompt against EVERY media item. With N "
+        "prompts here and M media files, this produces N*M trials (e.g. 4 prompts x 5 "
+        "images = 20). Any transforms are applied to every combination. Distinct from "
+        "`prompts`/`prompts_csv`, which pair one prompt per media (1:1).",
+    ] = None,
     custom_url: t.Annotated[
         str,
         "Target a custom multimodal HTTP endpoint instead of a litellm model. When "
@@ -553,6 +560,8 @@ def generate_multimodal_attack(
         params["prompts"] = prompts
     if prompts_csv:
         params["prompts_csv"] = prompts_csv
+    if prompt_matrix:
+        params["prompt_matrix"] = prompt_matrix
     if custom_url:
         params["custom_url"] = custom_url
     if custom_auth_type: