FIX: warn instead of raising on reasoning-model token truncation (Chat and Responses targets)

[romanlutz]

Description

Reasoning models spend completion tokens on hidden reasoning before emitting any visible output. When the token budget (max_completion_tokens for Chat Completions, max_output_tokens for the Responses API) is set too low, the API can return a truncated response with empty visible content: finish_reason="length" for Chat, or status="incomplete" with reason max_output_tokens for Responses.

Previously both OpenAI targets treated that empty content as a hard failure and raised EmptyResponseException (surfaced with a misleading "Status Code: 204"), which also triggered the full retry loop even though retrying with the same token budget cannot succeed.

Since a low token budget can be a deliberate configuration choice, this changes the behavior to warn rather than raise, consistently across the Chat and Responses targets:

• On token-limit truncation, log a warning explaining that reasoning models consume tokens on hidden reasoning and that the token limit may need to be increased.
• Preserve any reasoning or partial text the response did contain.
• When the truncation left no visible content, return a graceful empty response (error="empty") so the run continues instead of raising and retrying.
• Mark truncated responses with prompt_metadata["truncated"] = True on the first piece so downstream scorers and attacks can distinguish a truncated-empty message from a normal empty one.
• Capture token usage from the response into the first piece's metadata, including on the truncated path where usage is most valuable (the whole budget may have been spent on hidden reasoning with no visible answer).
• Responses that are empty for other reasons still raise EmptyResponseException as before.

Refactoring

• _validate_response is now validation-only: it inspects the response and raises on genuinely invalid ones, while constructing the resulting Message (including the graceful empty path) is the responsibility of _construct_message_from_response_async. The base-class contract changed from returning Message | None to None.
• Extracted a shared build_empty_truncated_response helper in prompt_target/common/utils.py, used by both targets instead of duplicating the empty-response recipe.
• Narrowed the Responses target's _validate_response and _construct_message_from_response_async parameters from Any to Response, and typed the Chat validators with ChatCompletion.
• Added token_usage_from_responses to parse the Responses API usage shape, giving the Responses target token-usage parity with Chat.
• Fixed misleading "chat" wording in Responses API empty-response error messages.

Tests and Documentation

• Added and updated unit tests for both targets covering the truncation warning, the graceful empty-response path, the truncated metadata marker, and token-usage capture (including on the truncated path).
• Added unit tests for the shared build_empty_truncated_response helper and for token_usage_from_responses.
• The full tests/unit/prompt_target unit suite passes.

No documentation changes needed. JupyText not run (no doc or notebook changes).

[Copilot]

Pull request overview

This PR adjusts OpenAI prompt targets to treat token-limit truncation as a valid but incomplete response: it logs a clear warning when truncation is detected and, when truncation produces no visible output, it returns a graceful empty response (error="empty") instead of raising EmptyResponseException (avoiding futile retries with the same token budget).

Changes:

• Update the OpenAI target validation contract so _validate_response raises on invalid responses (no longer returns an error Message).
• For Chat Completions (finish_reason="length") and Responses API (status="incomplete", reason="max_output_tokens"), warn on truncation and allow construction to produce partial output or a graceful empty message.
• Add/extend unit tests to assert warnings are emitted and truncated-empty responses yield error="empty" with prompt_metadata["truncated"]=True.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pyrit/prompt_target/openai/openai_target.py	Simplifies the validation flow: _validate_response is now “raise-only” and no longer returns an error Message.
pyrit/prompt_target/openai/openai_chat_target.py	Detects finish_reason="length" truncation, warns instead of raising, and returns a graceful empty piece when truncated output has no visible content.
pyrit/prompt_target/openai/openai_response_target.py	Detects Responses API truncation via status/incomplete_details, warns, tolerates empty sections on truncation, and appends a graceful empty text piece when needed.
pyrit/prompt_target/common/chat_completions_response_parser.py	Adds helper functions to read finish_reason and to build a graceful empty response for truncated Chat Completions.
tests/unit/prompt_target/target/test_openai_chat_target.py	Adds coverage for truncation warnings and truncated-empty construction behavior (including token usage capture).
tests/unit/prompt_target/target/test_openai_response_target.py	Adds coverage for Responses API truncation detection, warning behavior, and truncated message construction edge-cases (empty output, partial text, tool-call skipping).
tests/unit/prompt_target/target/test_chat_completions_helpers.py	Adds unit tests for new shared helper functions used by Chat Completions targets.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.