Skip to content

FIX: Retry adversarial replies with non-object JSON roots#2247

Open
romanlutz wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-fix-adversarial-json-parsing
Open

FIX: Retry adversarial replies with non-object JSON roots#2247
romanlutz wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-fix-adversarial-json-parsing

Conversation

@romanlutz

Copy link
Copy Markdown
Contributor

Description

Adversarial chat models can return valid JSON whose root is an array or scalar. _parse_adversarial_reply previously called .items() unconditionally, raising AttributeError outside the InvalidJsonException retry contract and aborting the attack instead of retrying on clean conversation history.

This change explicitly rejects non-object JSON roots as InvalidJsonException, including the decoded root type in the error message. It does not broadly catch exceptions or coerce malformed response shapes.

Tests and Documentation

  • Added parametrized parser coverage for array, null, string, and number roots.
  • Added a manager-level regression test proving an array response is retried and a subsequent valid object succeeds.
  • Ran focused parser, manager, and clean-history retry tests: 13 passed, plus 6 exact regression cases passed.
  • Ran the changed-file pre-commit suite, including Ruff format/check and ty: passed.
  • Documentation: N/A.
  • JupyText: N/A.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: cf88d9a1-35b5-4edf-8bd3-c9d5fc82c174
Copilot AI review requested due to automatic review settings July 22, 2026 04:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes adversarial-chat JSON parsing/retry behavior by ensuring _parse_adversarial_reply treats non-object JSON roots (array/scalar/null) as InvalidJsonException, so the existing send_json_with_retry_async clean-history retry contract applies instead of aborting on an AttributeError.

Changes:

  • Reject non-dict JSON roots in _parse_adversarial_reply and include the decoded root type in the error message.
  • Add parser unit coverage for array/null/string/number JSON roots.
  • Add a manager-level regression test verifying an array-root response is retried and a subsequent valid object succeeds.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
pyrit/executor/attack/component/adversarial_conversation_manager.py Adds an explicit non-object JSON-root check that raises InvalidJsonException, preserving retry semantics.
tests/unit/executor/attack/component/test_adversarial_conversation_manager.py Adds unit + regression coverage to ensure non-object roots trigger InvalidJsonException and are retried successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants