fix(experiments)!: run synchronous tasks in worker threads

[po-et]

What does this PR do?

Synchronous experiment tasks currently block the event loop and serialize blocking work despite max_concurrency. Run them in worker threads while preserving context propagation; keep native async tasks, decorated async functions, partials and async callable objects on the experiment's loop, invoking the original callable.

This is a v5 breaking execution-context change, following the review on this PR. It targets prepare-v5-release, not a patch release. Synchronous tasks must not capture caller-thread-only resources such as a default SQLite connection, and shared state may now be accessed concurrently. max_concurrency=1 does not restore caller-thread execution.

MIGRATION_V5_SYNC_TASKS.md explains resource ownership, per-task connection creation/closure, async-client migration, and the existing item-error policy. Add real SQLite tests through run_experiment: a caller-owned connection records item failures, while worker-owned connections preserve both outputs. The migration source is separate from generated reference documentation.

Integrates v5 changes including #1934 while preserving published branch history. Relative to v5, generated API files and unrelated SDK/test paths are unchanged.

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor
• Documentation update
• Tooling, CI, or repo maintenance

Verification

Python 3.12, dependencies installed with uv sync --locked:

uv run --frozen pytest -q tests/unit/test_experiment.py
uv run --frozen pytest -n 4 --dist worksteal tests/unit
uv run --frozen ruff check .
uv run --frozen ruff format --check .
uv run --frozen mypy langfuse --no-error-summary

• Experiment suite: 50 passed. Selected unchanged-v5 baseline tests fail for concurrency and the caller-owned SQLite execution-context difference; the async-loop case passes.
• Complete unit suite: 809 passed, 2 skipped, 18 setup errors. All 18 are existing test_prompt.py initialization errors (Langfuse client is not initialized); unmodified v5 produces the same 18 errors (8 prompt tests pass). These files and generated API code are identical to v5.
• Ruff lint and mypy pass. The two changed Python files pass format checking. Repository-wide format checking reports the unchanged v5 tests/unit/test_media.py; the same warning occurs on the unmodified v5 file.
• Separate automated review performed. No server E2E or live-provider tests were run; no remote CI or maintainer approval is implied.

Checklist

• I self-reviewed the diff using code_review.md.
• I added or updated tests for behavior changes.
• I updated docs/examples for the breaking execution context.
• Relative to the v5 target, generated API files are unchanged.
• I did not commit secrets or credentials.

AI assistance was used for implementation, tests and automated review.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 4 committers have signed the CLA.

✅ po-et
❌ hassiebp
❌ langfuse-bot
❌ niklassemmler
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[chrikrah]

@po-et this fixes the serialization: sync tasks overlap now, and two of your new cases plus the concurrency case fail on main. Your docs note already says thread-affine tasks should move to async. I'd ship it in v5 and not in a patch, because the cost of missing that note is every item.

$ pytest -q tests/unit/test_experiment.py   # uv sync --locked, Python 3.12
36 passed                  # head 47c4ee4
3 failed, 33 passed        # langfuse/experiment.py from bf11ec1, your tests kept
FAILED ...::TestExperimentObservationTree::test_sync_tasks_run_concurrently_with_trace_context

$ python lf_sync_task_sqlite.py   # sync task reads a sqlite3 connection opened by the caller, max_concurrency=2
outputs: ['A', 'B'] items: 2   # bf11ec1
Item 0 failed: SQLite objects created in a thread can only be used in that same thread. ...
outputs: [] items: 0           # head 47c4ee4
# not run: the e2e suites against a Langfuse server

non-blocking: called from a plain script, run_experiment returns zero item results for that task and raises nothing. The failure shows only as an ERROR log line and an error span in the trace. In a notebook, run_async_safely already moves the loop off the caller's thread, so scripts are what changes. With the bug-fix box ticked it would ship with no migration note. I'd retarget this to prepare-v5-release with a ! in the title and tick "Breaking change". #1934 landed experiment.py changes there, so expect a rebase.

@hassiebp does this belong on prepare-v5-release?

[cursor]

Thanks a lot for this, the implementation looks solid. Context propagation into the worker threads and the async detection both hold up well in review.

We've decided not to include it in v5.0, which ships tomorrow. Changing the default execution context the day before a major release is too risky for us. A few things also need more design work first:

• Tasks that hold thread-bound resources, such as sqlite connections or some client sessions, would start failing, and those items are only logged.
• The default executor caps effective concurrency at min(32, cpus + 4). For example, max_concurrency=50 on 8 CPUs ran only 12 tasks at once. A dedicated ThreadPoolExecutor(max_workers=max_concurrency) would avoid that.
• Ctrl-C can't stop tasks that are already running in threads.

We'd be happy to ship this in a v5.x minor release behind an opt-in flag, so the default doesn't change. Once v5 is out, could you retarget this PR to main and add the flag? The migration notes can then go into the regular docs, instead of the MIGRATION_V5_SYNC_TASKS.md and README additions.