feat(batch-evaluation)!: evaluate observations via the v2 observations API

[hassiebp]

What does this PR do?

This PR now carries three changes, because #1935 and #1937 were squash-merged into this branch.

Batch evaluation on the v2 observations API (this PR's original change)

• run_batched_evaluation reads only through api.observations.get_many (v2), with cursor pagination. The v3 GET /traces and v1 GET /observations paths are removed (they return 404 on Langfuse v4). It builds on community PR fix(batch_evaluation): route batch_evaluation through v2 observations API #1898.
• It evaluates observations only. The scope parameter and the root-observation / trace mode are removed. Every observation matching filter is scored on the observation.
• fields (v2 field groups, default "core,basic,io,metadata") replaces fetch_trace_fields. Mappers receive ObservationV2, where input/output are raw strings.
• BatchEvaluationResumeToken carries the v2 cursor. Resuming reuses the token's filter and rejects a different one. Duplicate rows that the events table briefly returns for one observation are evaluated once.
• _additional_trace_tags is dropped, since it relied on trace-create ingestion. Scores still go through the existing create_score path.

Remove trace-level IO setters (from #1935)

• Langfuse.set_current_trace_io(), span.set_trace_io() and the langfuse.trace.input/output attributes and media handling are removed. Set IO on the root observation instead.

Test suites on v4, CI on events_only (from #1937)

• New v4 test helpers read through v2 observations and scores_v3. All e2e and live-provider suites are ported, and CI drops the dual-write override.

Base: #1934's branch. Merge after #1934, then #1929 can merge at any point.

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor
• Documentation update
• Tooling, CI, or repo maintenance

Verification

uv run --frozen ruff check .                       # All checks passed!
uv run --frozen mypy langfuse --no-error-summary   # clean
uv run --frozen pytest -n auto tests/unit          # 751 passed, 2 skipped
# against a local Langfuse server (events_only):
uv run --frozen pytest -n 4 tests/e2e/test_batch_evaluation.py  # 7 passed

Checklist

• I self-reviewed the diff using code_review.md.
• I added or updated tests for behavior changes.
• I updated docs, examples, or .env.template if needed.
• I did not hand-edit generated files; if generated files changed, I used the upstream regeneration path.
• I did not commit secrets or credentials.

  

Confidence Score: 2/5

The PR is not yet safe to merge because resumed runs can miss observations and root-scope runs can score a trace more than once.

Summary

The PR moves batch evaluation reads to the v2 observations API, adds root-observation scope and field selection, and replaces page-based fetching with cursor pagination.

• Resume handling needs to preserve the original filter and handle tied timestamps when a cursor is unavailable.
• Root-observation filtering does not by itself guarantee a single trace-level evaluation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Scope and filter] --> B[v2 observations API]
  B --> C[Cursor-paginated pages]
  C --> D[Mapper and evaluators]
  D --> E{Scope}
  E -->|observations| F[Observation scores]
  E -->|root_observations| G[Trace scores]
  C --> H[Resume token]

Loading

_{Reviews (1) · Last reviewed commit: "test(batch-evaluation): rewrite e2e suit..."}

[github-actions]

@claude review

[claude]

Claude Code Review

No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.

_{Tip: disable this comment in your organization's Code Review settings.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 638034e310

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T20:17:31.870335Z	4ea561d	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

I reviewed this PR and didn't find any bugs. Because it's a large, breaking rework of the batch-evaluation read path (new v2 observations API, cursor-based pagination, and a redesigned resume token), a human look would still be worthwhile.

What was reviewed:

• Cursor-pagination/resume loop and max_items clamping in batch_evaluation.py
• Filter construction (v2 JSON-array validation, scope/resume conditions) and scope-based score attachment (observations vs root_observations)
• Ruled out: observations with no trace_id now fail per-item rather than being silently scored; cursor-less resume fallback uses strict "<" on startTime so same-timestamp boundary items could be skipped; max_items<=0 reports has_more_items=True despite fetching nothing

Extended reasoning...

The change reworks langfuse/batch_evaluation.py and langfuse/_client/client.py to move batch evaluation off the legacy v1 traces/observations APIs onto the v2 observations API, replacing timestamp-based pagination with cursor-based pagination and a redesigned resume token; tests were updated accordingly (unit suite grew substantially, e2e suite was heavily trimmed). It doesn't touch auth/crypto, but it changes the request/filter contract sent to the API and the pagination/resume correctness guarantees of a data-processing feature. It's a large (1000+ line), self-declared breaking change with real design tradeoffs (cursor vs. timestamp resume, root-observation semantics replacing traces), which is why a human look is still warranted even though no bugs were found.

[hassiebp]

remove this option, let's keep it simple to only observations for now

[cursor]

Done in 8299f52: batch evaluation now runs on observations only.

• The scope parameter is removed from run_batched_evaluation and the runner. Removing it entirely rather than keeping a one-value option means a scope can come back later as an additive change.
• Root-observation mode is gone: the isRootObservation filter, the one-root-per-trace dedupe and trace-level score attachment. Every matching observation is scored on the observation (plus the trace via the private _add_observation_scores_to_trace, as before).
• BatchEvaluationResumeToken no longer has a scope field. The filter check on resume stays.
• Docstrings and examples are updated. The root-scope unit and e2e tests are removed, and the remaining e2e tests now cover all observations. Unit: 750 passed; batch-eval e2e against an events_only server: 7 passed.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8063f8467f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4ea561df36

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".