Skip to content

[ML-70507] [m] Preserve Gemini tool signatures in LangChain and OpenAI clients - #750

Draft
jamesbxwu wants to merge 4 commits into
mainfrom
codex/ml-70507-gemini-signatures
Draft

jamesbxwu wants to merge 4 commits into
mainfrom
codex/ml-70507-gemini-signatures

Conversation

@jamesbxwu

@jamesbxwu jamesbxwu commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Gemini 3.8 Flash returns a thoughtSignature alongside each tool call. LangChain drops it when rebuilding a request from parsed tool calls, and OpenAI Agents expects Google's extra_content.google.thought_signature representation. Generated agents therefore execute the first tool and then receive HTTP 400 for the missing signature.

Preserve the opaque signature in both connector paths:

  • ChatDatabricks restores the field by call ID within each assistant message, including accumulated stream fragments and history restored from JSON checkpoints. This follows the approach from closed, unmerged databricks-langchain: preserve Gemini thought signatures in tool calls #270.
  • Databricks OpenAI clients expose the gateway field in Google's standard extension during JSON/SSE parsing, and restore thoughtSignature when the Agents SDK replays that extension. Existing provider fields and stored request history are preserved. The SDK still owns stream iteration, closing, and errors.

Require openai-agents>=0.6.5, the first tagged version that preserves Google thought signatures in both Chat Completions conversion and streaming tool history. The previous 0.5.0 floor discards provider metadata before the next client request, so the connector cannot restore it without stateful recovery. This minimum dependency change keeps the native SDK regressions active instead of skipping incompatible versions.

This is a partial fix for ML-70507. Provider discovery for neutral aliases, GPT reasoning/API selection, and gateway tool-call identity remain open dependencies. The gateway reuses IDs across distinct Gemini invocations; current OpenAI Agents correctly rejects that reuse. This PR preserves IDs and does not add invocation caches or change the templates' API selection. Connector releases and scaffold dependency updates are needed before newly generated agents receive the changes.

Validation:

  • Python 3.10 lowest-direct OpenAI suites with openai-agents==0.6.5: 182 passed, 1 skipped without extras; 220 passed with memory extras. The same native JSON/SSE two-tool regressions remain unconditional. Earlier current-SDK validation also passed all 220 cases.
  • LangChain chat-model suite plus 11 new regression cases: 119 passed, 2 skipped. All 11 new cases fail against unchanged main; six of the seven new OpenAI cases fail on main, including native SDK two-tool loops through actual JSON/SSE decoding.
  • Ruff check/format and focused ty checks pass for both connectors.
  • Live DEFAULT/E2-dogfood with system.ai.gemini-3-8-flash: unchanged-main native LangGraph executes the first tool and then fails for missing signature; the patch executes echo("one"), then echo("two"), then reaches a final answer in both async nonstream and message-streaming modes.
  • Live OpenAI Agents 0.20.0 now gets past the signature failure. A streaming control with two distinct local tools completes both calls and a final answer. Reusing the same tool still fails because the gateway repeats its ID; the SDK rejects that second invocation with ModelBehaviorError. A nonstream distinct-tool control executes both calls, then exposes another gateway contract defect: final assistant content is an array where the SDK requires a string. A separate raw HTTP 200 response confirmed a list of text blocks containing thoughtSignature; this PR does not flatten or discard those blocks. One repeated-tool streaming attempt timed out before executing tools and is not counted as a successful run.
  • GPT 6.1 Sol's default Chat Completions path still returns HTTP 400 for reasoning plus tools, requiring Responses or explicit reasoning opt-out.

Only local echo tools and in-memory history were used; no platform resources were created.

Public CI baseline: the framework job fails the same six memory-backed parallel-history ordering cases as current main (expected last/first, observed first/last). Confirmed independently against main run https://github.com/databricks/databricks-ai-bridge/actions/runs/38662891535/job/116188048864; no forced skips. Other checks are tracked separately.

Combined private cache validation: https://github.com/databricks-eng/ai-oss-integration-tests-runner/actions/runs/38698567494 passed all 20 live caching tests (OpenAI 8, LangChain 4, template adapters 8) at e231ca2. This source combines 11 PR heads; it is not an isolated run of this branch. Models cover Claude Sonnet 4.5/4.6 in the template adapters and Gateway/FMAPI routes in the connectors. The Apps/tool matrix runs separately.

When combining #727 and #750, compose the Gemini signature and cache normalizers in each shared OpenAI response hook. Keep #746 usage accounting with the cache regressions in the LangChain merge. The test-only combined commit records those resolutions; individual PRs retain independent boundaries.

Bug-bash integration validation

The test-only combined branch at be2fa8ee8d6d3b3e011ecfa285e6455b90f9433a contains this exact PR head and eleven other reviewed heads. Its Agent Bricks unit suite passed 2,154 tests, 24 skipped. This is combined-source validation, not an isolated run of this PR; the branch must not be merged into main.

Private run 38698567494 passed all 20 live Claude cache tests at e231ca2 (OpenAI 8, LangGraph 4, generated adapters 8). Run 38701390693 at 53f9db8 passed OpenAI 8, LangGraph 4 and 3 adapter cases; the remaining 5 adapter cases returned HTTP403 because the shared test-workspace Gateway budget reached its $10 limit. Production source is identical across those runs and the final combined source; later changes affect only matrix fixtures, cleanup, regression tests and documentation. The rerun is not all green, and no budgets were changed.

The corrected Apps test passed 20/20 matrix rows, two grant snapshots, transitive-permission controls and repeat-deploy idempotency. Its evidence confirms clean source and wheel/source matching. That live run predates the final cleanup correction: the harness recorded twelve successful cleanup operations but omitted the implicit Runtime Store/role for direct-authoring Apps. #752 now includes ownership-checked cleanup for every App and safe retry/evidence handling, covered by focused regressions and the final unit suite. No new full model matrix was run for those cleanup-only changes.

Independent read-only run 38708044949 confirmed all 13 exact prior-run resources absent, including three consecutive typed404 observations for the omitted Runtime Store, dedicated database and role. One separate current-window database/role pair remains present and unattributed: SCIM did not recover its App identity and one bounded audit SELECT failed. That pair was left intact; full cleanup of the current matrix run is not established. No resources were deleted using name/time correlation alone.

The Apps matrix covers LangGraph, CLI/direct authoring, dev/deployed runtime and five tool kinds. Separate cache tests cover both LangGraph and OpenAI Agents. These checks do not establish fresh-account provisioning rights or freshly consented deployed Genie OBO acceptance.

All public checks finished. #694/#727 have 54 green; the other ten PRs have 53 green and the same framework job red. Its six memory-backed parallel-history ordering failures also occur on current main. No forced skips or weakened grant assertions were applied.

@jamesbxwu jamesbxwu changed the title Preserve Gemini tool signatures in LangChain and OpenAI clients [ML-70507] Preserve Gemini tool signatures in LangChain and OpenAI clients Oct 15, 2026
@jamesbxwu jamesbxwu changed the title [ML-70507] Preserve Gemini tool signatures in LangChain and OpenAI clients [ML-70507] [m] Preserve Gemini tool signatures in LangChain and OpenAI clients Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant