[bofenghuang]

Summary

This PR adds a new environment: mentalhealthbench.

MentalHealthBench is a benchmark from OpenAI (paper, announcement). It has 1,215 synthetic mental health conversations and 5,262 rubric criteria. Licensed clinicians wrote the criteria.

Why a new environment, not a HealthBench subset

MentalHealthBench uses the same score formula as HealthBench. But most other parts are different:

Part	HealthBench	MentalHealthBench
Judge prompt	HealthBench prompt. The judge sees the points of each criterion.	A new prompt. The judge does not see the points.
Judge output	criteria_met (true or false)	satisfaction (0 or 1)
Prompt layout	One conversation block, with the response as the last turn	Three blocks: conversation, response, criterion
Metadata	Theme, axis, consensus criteria	Acuity, user profile, language, behavior axis (10 values)
Length adjustment	Published by OpenAI	Not published
Data source	Hugging Face datasets	Official OpenAI zip file

A subset in healthbench would need a condition at most of these points. That makes the HealthBench code harder to read and to maintain.

A separate environment also gives the benchmark its own name. Thus, processed results and win rates show MentalHealthBench separately from HealthBench.

The cost: the new environment copies the judge loop and the retry logic from healthbench (about 100 lines). If a third rubric benchmark comes, we can move this shared code into medarc_verifiers.

What the environment does

1. It downloads the dataset from the official OpenAI zip file.
2. It checks the SHA-256 of the data file. If the file changes upstream, loading stops with an error.
3. It removes the canary string. The model and the judge do not see it.
4. It sends each conversation to the model without changes. Some conversations start with their own system message. The environment keeps that message and adds no other system prompt.
5. A judge grades each criterion separately, as 0 or 1.
6. The environment calculates the score with the formulas from the paper.

Judge prompt

The judge prompt is a word-for-word copy of the paper's prompt (Appendix B.2).

This prompt is different from the HealthBench prompt:

• The judge does not see the points of the criterion. It only decides if the behavior occurred.
• The judge returns {"explanation", "satisfaction": 0|1}.
• The conversation, the response and the criterion go in three separate blocks.

If the judge call fails, or the output is not valid JSON, the environment tries again (3 times by default). If all attempts fail, the criterion counts as "not met".

Scores

All scores use the total positive points of the task as the denominator.

Name	Meaning
reward	The signed score, with a lower limit of 0. This is the main score in the paper.
signed_score	Points of all met criteria, divided by the total positive points. This value can be negative.
positive_points_earned	The share of positive points that the response earns.
normalized_penalty_burden	The penalty points that the response gets, as a share of the total positive points.

An empty response gets 0 points. The judge does not grade it. The paper does the same (Appendix B.1).

Samples per task

The paper uses 4 responses for each task. To do the same, use -r 4. This costs 4 times more than -r 1.

More samples do not change the expected score. They only make the score less noisy. The paper does not compare 1 sample with 4 samples.

Note: the default of -r in medarc-eval is 3. Always set -r.

Differences from the paper

OpenAI did not publish evaluation code. These choices are ours:

• Conversation format for the judge. The paper does not say how to write the conversation into the prompt. We use Role: content blocks with a blank line between them.
• Judge model. The paper uses GPT-5.6 Sol with high reasoning effort. Our default is openai/gpt-5-mini. Thus, our scores are not directly comparable to the paper.
• Breakdowns. The environment does not calculate scores by axis, acuity, user profile or language. The data for these breakdowns is in info.

How to test

uv run pytest tests/test_environments/test_mentalhealthbench.py
medarc-eval mentalhealthbench -m gpt-4.1-mini -n 5 -r 1

The 6 new tests use synthetic data and a fake judge. They do not use real benchmark examples, because the dataset authors ask people not to publish them.

Not done yet

• A live run against a real model.
• A parity check against the paper's scores.

🤖 Generated with Claude Code