OpenAI Responses API is sent Chat Completions content parts (`type: "text"`): live vision examples 400, fall back, then fail with `invalid_image_url`

[Dhivya-Bharathy]

Executive Summary

Live runs of the in-tree vision examples (examples/python/agents/image-agent.py, image-to-text-agent.py) against gpt-4o-mini hit this provider error before Chat Completions fallback:

Responses API failed, falling back to Chat Completions: Error code: 400 - {
  "error": {
    "message": "Invalid value: 'text'. Supported values are: 'input_text', 'input_image', 'input_audio', 'output_text', 'refusal', 'input_file', 'computer_screenshot', 'summary_text', and 'encrypted_content'.",
    "type": "invalid_request_error",
    "param": "input[0].content[0].type",
    "code": "invalid_value"
  }
}

Then Chat Completions failed on the same turn:

Failed to download image from image.jpg. Image URL is invalid.

OpenAIClient._use_responses_api("gpt-4o-mini") is True (official OpenAI, no custom base_url, SDK has responses). _build_responses_input maps system messages and tool calls, then appends the remaining Chat Completions messages unchanged:

else:
    input_items.append(msg)

Chat Completions multimodal parts use "type": "text" and "type": "image_url". The Responses API requires "input_text" and "input_image". The 400 is exactly that mismatch (param: input[0].content[0].type).

Fallback does not save the example: a local filename image.jpg is not an HTTP URL, so Chat Completions returns invalid_image_url. The process can still exit 0 while printing stacked Error panels — the corpus harness counted these as “pass.”

Validated on main @ 43bea02, live OpenAI, example wall times 37.3s and 40.2s of failed API round-trips.

Workaround today: disable Responses API (custom base_url, or a model _use_responses_api rejects) and pass a public https:// image URL. Neither is the documented vision example.

Problem Statement

Vision / multimodal agents must:

1. Send user text + image to the active OpenAI API surface
2. Use the part types that surface documents
3. Fail loudly (non-zero) when the image cannot be read
4. Not burn ~40s on a guaranteed-400 mapping bug

gpt-4o-mini on the official API goes through Responses first. The transform is lossy for any message whose content is a list of parts. Every vision agent, screenshot tool, and “analyze this file” flow on OpenAI models is on this path.

Live execution evidence

From image-agent.py (live, gpt-4o-mini):

WARNING Responses API failed, falling back to Chat Completions: Error code: 400 -
Invalid value: 'text'. Supported values are: 'input_text', 'input_image', ...
param: input[0].content[0].type

ERROR Error creating async completion: Error code: 400 -
Failed to download image from image.jpg. Image URL is invalid.

ERROR Unified chat completion failed: ... invalid_image_url

The same pair of errors repeated (two chat attempts). Wall clock 37.251s, process exit 0.

image-to-text-agent.py: same Responses 400, same image.jpg URL error, plus DuckDuckGo search is not available (missing extra). Wall 40.171s, exit 0.

This is not a missing-key problem. The key was valid; the payload schema was wrong, then the image locator was a local path.

Expected vs actual

Expected	Actual
Responses input[].content[].type is input_text / input_image	text (Chat Completions name) → 400
Local file becomes input_image with bytes or data URL	Passed through as image.jpg string → invalid_image_url
Example exits non-zero on provider 400	Exit 0, rich Error panels
Fallback is rare	Fallback is every multimodal Responses call with list content

Repro

# Minimal shape that _build_responses_input currently forwards:
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image"},
            {"type": "image_url", "image_url": {"url": "image.jpg"}},
        ],
    }
]

Run any Agent vision example that builds that shape against gpt-4o-mini with a real OPENAI_API_KEY.

python examples/python/agents/image-agent.py

Expected: one Responses (or Completions) call that accepts the image.
Actual: Responses 400 on type=text, fallback, Completions 400 on image.jpg.

Impact analysis

Developer impact

Vision examples on the default OpenAI model look “runnable” (exit 0) and are broken. Debugging requires noticing a warning (Responses API failed, falling back) rather than a hard error. Time-to-green for multimodal is ~40s of failed HTTP per example.

Business impact

OpenAI’s current default API for these models is Responses. Shipping a transform that 400s on the first content part means official-API vision is not production-ready. Customers on Azure/custom base_url may not see it (_use_responses_api returns False) — so CI that mocks or uses a proxy never catches it.

User impact

“Analyze this screenshot / PDF page / photo” agents fail with invalid_image_url or a cryptic Invalid value: 'text'. Users think their file is bad. It is the SDK mapping.

Root cause hypothesis

def _build_responses_input(...):
    for msg in messages:
        role = msg.get("role", "")
        if role in ("system", "developer"):
            ...  # → instructions
        elif role == "assistant" and msg.get("tool_calls"):
            ...  # function_call items
        elif role == "tool":
            ...  # function_call_output
        else:
            input_items.append(msg)  # ← Chat Completions message, untranslated

User/assistant messages with content: str happen to work (Responses accepts string content in some cases). User messages with part lists do not:

Chat Completions	Responses
{"type": "text", "text": "..."}	{"type": "input_text", "text": "..."}
{"type": "image_url", "image_url": {"url": "https://..."}}	{"type": "input_image", "image_url": "https://..."} (flat URL)

param: input[0].content[0].type in the live 400 is the first part of the first forwarded user message.

Second failure: even a perfect Responses mapper will 400/fail if the URL is a relative filesystem path. Examples pass image.jpg. Chat Completions tries to GET it as a URL. There is no Path.read_bytes() → data URL (or input_file) step on this path.

Sequence

sequenceDiagram
    participant E as Vision example
    participant A as Agent
    participant C as OpenAIClient
    participant R as Responses API
    participant CC as Chat Completions

    E->>A: image.jpg + prompt
    A->>C: acreate_completion (gpt-4o-mini)
    C->>C: _use_responses_api True
    C->>R: input[0].content[0].type = "text"
    R-->>C: 400 invalid_value
    C->>C: warning, fall through
    C->>CC: image_url url=image.jpg
    CC-->>C: 400 invalid_image_url
    C-->>A: exception in chat
    A-->>E: Error panel, exit 0

Loading

Proposed fix (with code)

1. Map content parts in _build_responses_input

def _content_to_responses_parts(content):
    if content is None:
        return []
    if isinstance(content, str):
        return [{"type": "input_text", "text": content}]
    parts = []
    for p in content:
        if not isinstance(p, dict):
            parts.append({"type": "input_text", "text": str(p)})
            continue
        t = p.get("type")
        if t in ("text", "input_text"):
            parts.append({"type": "input_text", "text": p.get("text") or p.get("content") or ""})
        elif t in ("image_url", "input_image"):
            url = p.get("image_url")
            if isinstance(url, dict):
                url = url.get("url")
            url = _ensure_fetchable_image_url(url)
            parts.append({"type": "input_image", "image_url": url})
        else:
            parts.append(p)
    return parts

For the else: input_items.append(msg) branch:

item = {
    "role": role,
    "content": _content_to_responses_parts(msg.get("content")),
}
input_items.append(item)

Do not forward raw Chat Completions messages into params["input"].

2. Local files → data URLs (both APIs)

from pathlib import Path
import base64
import mimetypes

def _ensure_fetchable_image_url(url: str | None) -> str:
    if not url:
        raise ValueError("empty image url")
    if url.startswith(("http://", "https://", "data:")):
        return url
    path = Path(url)
    if not path.is_file():
        raise FileNotFoundError(url)
    mime = mimetypes.guess_type(path.name)[0] or "image/jpeg"
    b64 = base64.standard_b64encode(path.read_bytes()).decode("ascii")
    return f"data:{mime};base64,{b64}"

Use this in both Responses mapping and Chat Completions image_url construction so fallback does not 400 on image.jpg.

3. Examples: exit non-zero on chat failure

Vision examples should sys.exit(1) when the agent returns an Error panel / exception. Exit 0 with printed errors hides this from CI.

4. Tests

def test_responses_input_rewrites_text_parts():
    client = OpenAIClient.__new__(OpenAIClient)  # or a thin helper extract
    params = OpenAIClient._build_responses_input(
        client,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "hi"},
                {"type": "image_url", "image_url": {"url": "https://example.com/a.png"}},
            ],
        }],
        model="gpt-4o-mini",
    )
    parts = params["input"][0]["content"]
    assert parts[0]["type"] == "input_text"
    assert parts[1]["type"] == "input_image"
    assert "text" != parts[0]["type"]

def test_local_image_becomes_data_url(tmp_path):
    img = tmp_path / "image.jpg"
    img.write_bytes(b"\xff\xd8\xff")  # jpeg magic
    url = _ensure_fetchable_image_url(str(img))
    assert url.startswith("data:image/jpeg;base64,")

A test that only mocks responses.create success will not catch this; the live 400 is on request shape.

Architecture

Agent multimodal message
        │
        ▼
  Chat Completions shape (type=text | image_url)
        │
        ├── if Responses: MAP parts → input_text / input_image
        │         └── local path → data: URL
        └── if Completions: keep image_url
                  └── local path → data: URL (same helper)

Today the MAP step is missing; local-path promotion is missing.

Performance evidence

Example	Wall	HTTP	Exit
image-agent.py	37.251s	Responses 400 + Completions 400 (×2)	0
image-to-text-agent.py	40.171s	same + missing ddgs warning	0

~40s is failed round-trips, not inference. A correct mapper is one successful call.

Before vs after

Before: Default OpenAI vision → Responses 400 on type=text → fallback → invalid_image_url on local files → exit 0.

After: Parts mapped; local files inlined; examples fail CI if the provider 400s; unit tests lock the Responses schema.

Out of scope

• Installing opencv-python / ddgs extras (separate example deps)
• Non-OpenAI vision providers
• ACP image prompt blocks (ACP advertises image: false today)

Related

• This is not the Harbor 10-tool-call cap and not the code -p constructor crash.
• _use_responses_api skipping when base_url is set explains why some environments never see the 400.

[praisonai-triage-agent]

👋 Thanks for opening this issue, @Dhivya-Bharathy!

A maintainer will review this shortly. In the meantime:

• Make sure you've included steps to reproduce (for bugs)
• Check existing issues for duplicates
• Review the documentation for related guides

Bug and enhancement reports are auto-triaged with Claude. Questions and other issues: a maintainer can comment @claude to trigger analysis.

Routing: Agent-callable tools → PraisonAI-Tools; lifecycle plugins (tracing/hooks/guardrails) → PraisonAI-Plugins.

[MervinPraison]

@claude

NOTE: This is an issue (not a PR). Create a new fix branch per STEP 3 below.

You are working on the PraisonAI SDK. Follow AGENTS.md strictly.

STEP 0 — SETUP GIT IDENTITY & AUTH (GLOBAL — required for all repos):
git config --global user.name "MervinPraison"
gh auth setup-git

STEP 1 — READ GUIDELINES:
Read AGENTS.md ...
SCOPE: Focus ONLY on Python packages (praisonaiagents, praisonai). Do NOT modify praisonai-rust or praisonai-ts.
AIM: LIGHTWEIGHT AND POWERFUL — reject scope creep ...

STEP 2 — ARCHITECTURE VALIDATION & ROUTING:

• Core SDK → praisonaiagents/
• Wrapper → praisonai/
• Tools → MervinPraison/PraisonAI-Tools (STEP 3-ALT)
• Docs → PraisonAIDocs
• Plugins → PraisonAI-Plugins (STEP 3-ALT)
  If checks fail → comment and close issue, no PR.

STEP 3 — IMPLEMENT (for changes in THIS repo):

• git checkout -b claude/issue-$ISSUE_NUMBER-$(date +%Y%m%d)
• minimal fix, commit, push
• MUST run: gh pr create --title "fix: <title>" --body "Fixes #$ISSUE_NUMBER" ...

STEP 3-ALT — IMPLEMENT IN EXTERNAL REPO (Tools/Docs/Plugins):

• gh repo clone → /tmp/ → branch → commit → push → gh pr create -R ...

STEP 4 — TEST:

• pytest tests/ for SDK changes

STEP 5 — CREATE PR (must use gh pr create automatically)

[praisonai-triage-agent]

Claude finished @MervinPraison's task —— View job • claude/issue-3934-20260817-0642 • [Create PR ➔](https://github.com/MervinPraison/PraisonAI/compare/main...claude/issue-3934-20260817-0642?quick_pull=1&title=Issue%20%233934%3A%20Changes%20from%20Claude&body=This%20PR%20addresses%20issue%20%233934%0A%0AGenerated%20with%20%5BClaude%20Code%5D(https%3A%2F%2Fclaude.ai%2Fcode)

---

Fixed: OpenAI Responses API multimodal content parts

PR created: #3998

• Read AGENTS.md guidelines
• Locate _build_responses_input/_build_responses_params
• Map content parts in LLM _build_responses_params
• Add unit tests for LLM multimodal mapping
• Run tests (16 passed)
• Commit, push, create PR

Root cause

The OpenAIClient path (bare gpt-4o-mini) was already fixed in PR #3991 — it maps text→input_text, image_url→input_image and inlines local files as data URLs.

But the LLM class path (openai/gpt-4o-mini, i.e. Path P1 in the tests) still had the exact bug from the issue at llm.py:5890:

else:
    input_items.append(msg)   # Chat Completions message forwarded untranslated

So multimodal messages (content parts type: "text" / image_url) still hit the 400 invalid_value on input[0].content[0].type.

Fix (minimal, no new API surface)

Reuse the existing OpenAIClient._build_responses_content classmethod in LLM._build_responses_params to translate parts to input_text/input_image and inline local image paths as data: URLs. Plain-string content passes through unchanged.

else:
    from .openai_client import OpenAIClient
    item = dict(msg)
    item["content"] = OpenAIClient._build_responses_content(msg.get("content", ""))
    input_items.append(item)

Tests

Added 4 unit tests to tests/integration/test_responses_api.py:

• multimodal parts → input_text/input_image
• local image path → data URL
• plain-string passthrough

Result: 16 passed (non-live tests).

Scope notes

• Kept the fix lightweight — reused the existing helper rather than duplicating logic or adding params/modules (per AGENTS.md).
• Did not modify the example files (image-agent.py etc.) sys.exit(1) behaviour — that is example harness/UX scope creep beyond the SDK mapping bug, and the core SDK fix is what makes vision work on the official Responses API.
  • Branch