test: expect flattened metadata keys for v5 by hassiebp · Pull Request #2001 · langfuse/langfuse-python

hassiebp · 2026-10-14T13:18:01Z

What does this PR do?

CI on main and on the v5.0.0 / v5.0.1 release commits fails in three e2e tests with KeyError: 'context' / KeyError: 'response_format'. The release commits only bump the version from 4.18.0 to 5.0.0. For Python SDK 5 and later, the server JSON-decodes per-key metadata and stores nested objects as flattened dotted paths (langfuse/langfuse#18436). The observations API returns those paths as-is, so nested metadata comes back as {"context.nested": ...}, not {"context": {"nested": ...}}. Before the release, pre-release CI still ran as 4.x and never hit this path.

This PR changes tests only; there are no SDK changes.

tests/e2e/test_media.py, tests/e2e/test_decorators.py: read and poll metadata["context.nested"].
tests/live_provider/test_openai.py: assert the string-valued paths response_format.type and response_format.json_schema.name, and drop expand_metadata="response_format", which no longer matches a top-level key. Arrays and booleans are JSON-stringified at their path, so the test doesn't assert on them.
tests/e2e/test_core_sdk.py: remove the xfail marker on test_create_score_with_custom_timestamp. The platform release it waited for has shipped, and the test XPASSes on current CI.

Failing run this fixes: https://github.com/langfuse/langfuse-python/actions/runs/38553271207

Type of change

Verification

uv run --frozen ruff check .                       # All checks passed!
uv run --frozen ruff format --check .              # 575 files already formatted
uv run --frozen mypy langfuse --no-error-summary   # clean

I didn't run e2e or live-provider tests locally: the local server's seeded API key was rejected and no OPENAI_API_KEY was available. This PR's CI E2E shards and live-provider job are the verification.

Checklist

I self-reviewed the diff using code_review.md.
I added or updated tests for behavior changes.
I updated docs, examples, or .env.template if needed.
I did not hand-edit generated files; if generated files changed, I used the upstream regeneration path.
I did not commit secrets or credentials.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking gap in the OpenAI schema checks.

Summary

Updates tests to expect flattened observation metadata for SDK v5.

Media checks read nested metadata through dotted keys.
OpenAI checks read response format from flattened metadata paths.
The custom-timestamp score test runs as a regular test.

One non-blocking concern remains: the OpenAI test no longer checks any fields inside the actual response schema.

hassiebp explicitly states that array and boolean assertions were intentionally dropped because those values are JSON-stringified.

_{Reviews (1) · Last reviewed commit: "test: expect flattened metadata keys for..." · Reviewed by Greptile}

Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com>

github-actions · 2026-10-14T13:18:09Z

@claude review

claude

Claude Code Review

No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.

_{Tip: disable this comment in your organization's Code Review settings.}

chatgpt-codex-connector · 2026-10-14T13:18:16Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T13:20:03.518706Z	`4d19003`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d190037b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-14T13:20:00Z

+    assert generation[0].metadata["response_format.type"] == "json_schema"
+    assert generation[0].metadata["response_format.json_schema.name"] == "math_response"


Preserve the full response-format assertion

Assert the flattened equivalent of the complete response_format, not only its type and name. If the OpenAI instrumentation drops or corrupts strict, the schema properties, required fields, or additionalProperties, this test now passes despite previously verifying the entire object; checking all expected flattened leaf paths and serialized values would retain that regression coverage.

AGENTS.md reference: AGENTS.md:L32-L34

Useful? React with 👍 / 👎.

claude

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

claude · 2026-10-14T13:21:40Z

+    assert generation[0].metadata["response_format.type"] == "json_schema"
+    assert generation[0].metadata["response_format.json_schema.name"] == "math_response"


🟡 nit (optional): Maintainers lose coverage of the recorded json_schema body. The new assertions would still pass if the SDK dropped or mangled schema in response_format metadata. Only response_format.type and response_format.json_schema.name are checked now, though the flattened schema still has string leaves that can be asserted. Fix: also assert a few string-valued schema paths, e.g. metadata["response_format.json_schema.schema.type"] == "object" and metadata["response_format.json_schema.schema.properties.final_answer.type"] == "string".

Why this was flagged

On the base branch, tests/live_provider/test_openai.py compared generation[0].metadata["response_format"] with the full {"type": "json_schema", "json_schema": json_schema}. So a regression in _get_structured_output_metadata / _serialize_openai_value (langfuse/openai.py:250-274) that lost or changed the nested schema would fail the test. After the change, lines 1007-1008 check only two top-level string leaves. Such a regression now passes silently. The PR says arrays and booleans are JSON-stringified at their path, which explains skipping strict and required. It does not explain skipping string leaves like schema.type or schema.properties.final_answer.type, which come back as plain strings at their dotted paths.

Verification: The base test asserted generation[0].metadata["response_format"] == {"type": "json_schema", "json_schema": json_schema}, covering the full schema (test_openai.py lines 961-984). Lines 1007-1008 now assert only response_format.type and response_format.json_schema.name, so a regression that lost the schema would pass silently.

greptile-apps · 2026-10-14T13:22:07Z

+    assert generation[0].metadata["response_format.type"] == "json_schema"
+    assert generation[0].metadata["response_format.json_schema.name"] == "math_response"


Actual schema goes unchecked

Replacing the full response_format comparison with only type and name lets this test pass if the captured metadata drops json_schema.schema entirely. Keep assertions for short string-valued leaves such as response_format.json_schema.schema.type and response_format.json_schema.schema.properties.final_answer.type. These do not need array or boolean decoding.

Prompt To Fix With AI

This is a comment left during a code review. Path: tests/live_provider/test_openai.py Line: 1007-1008 Comment: **Actual schema goes unchecked** Replacing the full `response_format` comparison with only `type` and `name` lets this test pass if the captured metadata drops `json_schema.schema` entirely. Keep assertions for short string-valued leaves such as `response_format.json_schema.schema.type` and `response_format.json_schema.schema.properties.final_answer.type`. These do not need array or boolean decoding. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

test: expect flattened metadata keys for v5

4d19003

Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com>

claude Bot reviewed Oct 14, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Oct 14, 2026

View reviewed changes

claude Bot reviewed Oct 14, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 14, 2026

View reviewed changes

hassiebp merged commit 4aa69a3 into main Oct 14, 2026
22 checks passed

hassiebp deleted the hassiebbot/flattened-metadata-tests-254d branch October 14, 2026 13:41

Navigation Menu

Repository navigation

test: expect flattened metadata keys for v5 - #2001

test: expect flattened metadata keys for v5#2001
hassiebp merged 1 commit into
mainfrom
hassiebbot/flattened-metadata-tests-254d

hassiebp commented Oct 14, 2026 •

edited by greptile-apps Bot

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

claude Bot left a comment

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 14, 2026

Uh oh!

claude Bot left a comment

Uh oh!

claude Bot Oct 14, 2026

Uh oh!

greptile-apps Bot Oct 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		assert generation[0].metadata["response_format.type"] == "json_schema"
		assert generation[0].metadata["response_format.json_schema.name"] == "math_response"

Conversation

hassiebp commented Oct 14, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Type of change

Verification

Checklist

Confidence Score: 4/5

Summary

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hassiebp commented Oct 14, 2026 •

edited by greptile-apps Bot

Loading

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading