AMHART Quickstart Guide

AMHART (Adolescent Mental Health Assessment & Rating Tool) is a pipeline for evaluating LLM performance on adolescent mental health conversations.

The complete pipeline includes 3 primary steps:

1. Vignette Creation: Clinical vignettes are generated based on user-specified characteristics using few-shot prompting with prior examples
2. Conversation: An LLM acts as the patient from the vignette and converses with the LLM being evaluated
3. Grading: Multiple LLM judges grade the conversations against the AMHART criteria

Quick Start

1. Setup Environment

# Create your conda environment
conda create -n amhart
conda activate amhart
conda install pip

# Install dependencies
pip install -r requirements.txt

# Configure your .env file with API keys
cp .env.sample .env
# Edit .env with your API keys

2. Run the Complete Pipeline

Option A: Python Script

python main.py

Option B: Bash Script

./run_pipeline.sh

This will:

1. ✅ Generate clinical vignettes using few-shot prompting
2. ✅ Run conversations where an LLM acts as the patient
3. ✅ Grade conversations using multiple LLM judges against AMHART criteria

---

Pipeline Steps (Run Individually)

Each step can be configured in two ways:

1. YAML Configuration Files (recommended): Define all parameters in a config file for reproducibility. You may look at the configs/*/default.yaml for an example.

2. Command-Line Arguments: Override specific parameters or run without a config file

Step 1: Generate Vignettes

Create clinical vignettes based on user-specified characteristics using few-shot prompting.

python generators/generate_vignettes.py --config configs/vignette_generation/default.yaml

Output: data/vignette_bank/vignette_bank.csv

Step 2: Run Conversations

An LLM acts as the patient from the vignette and converses with the LLM being evaluated.

python generators/converse.py --config configs/conversations/default.yaml

Output: data/conversations/<model_name>/conversation_*.csv

Step 3: Grade Conversations

Evaluate conversations against the AMHART criteria (14 therapeutic dimensions).

Multi-Judge Grading (Recommended)

Grades conversations using multiple LLM judges simultaneously:

python graders/AMHARTGrader.py --config configs/grading/default.yaml

Single-Judge Grading (Alternative)

Individual judge evaluation scripts are also available:

# Claude-based grading
python graders/conversation_grader_by_claude.py /path/to/input_csv_dir /path/to/output.csv

# GPT-based grading
python graders/conversation_grader_by_gpt.py /path/to/input_csv_dir /path/to/output.csv

# Gemini-based grading (Jupyter notebook)
# Open and run graders/gemini_batch_prep.ipynb

Output: data/graded_conversations.csv

---

Configuration

All configurations are YAML files in configs/:

Vignette Generation

• configs/vignette_generation/default.yaml
• Customize: model, number of rounds, diversity threshold, vignette characteristics

Conversations

• configs/conversations/default.yaml
• Customize: models (A & B), system prompts, number of rounds

Grading

• configs/grading/default.yaml - Full conversation grading

---

Project Structure

AMHART/
├── main.py                      # Pipeline runner (Python)
├── run_pipeline.sh              # Pipeline runner (Bash)
├── generators/
│   ├── generate_vignettes.py    # Generate clinical vignettes
│   └── converse.py              # Run model conversations
├── graders/
│   ├── AMHARTGrader.py              # Multi-judge grading (recommended)
│   ├── conversation_grader_by_claude.py  # Single-judge Claude grading
│   ├── conversation_grader_by_gpt.py     # Single-judge GPT grading
│   ├── gemini_batch_prep.ipynb      # Single-judge Gemini grading
│   └── grading_prompts/             # AMHART criteria prompts
├── configs/
│   ├── vignette_generation/
│   ├── conversations/
│   └── grading/
└── data/
    ├── vignette_bank/           # Generated vignettes
    ├── conversations/           # Model conversations
    └── prompts_for_generating_vignettes/

---

Custom Runs

Use Different Configs

# Generate vignettes with custom config
python generators/generate_vignettes.py --config configs/vignette_generation/custom.yaml

# Run conversations with specific models
python generators/converse.py --config configs/conversations/gpt4_vs_claude.yaml

Command-Line Overrides

All scripts support both YAML configs and command-line arguments. Command-line args override YAML settings when both are provided.

# Generate vignettes with command-line args (no config file)
python generators/generate_vignettes.py \
  --model_name gpt-5 \
  --num_rounds 10 \
  --generate_n_vignettes 20

# Run conversations with CLI args
python generators/converse.py \
  --ModelA gpt-5 \
  --ModelB claude-sonnet-4.5 \
  --num_rounds 5

# Grade conversations with CLI args
python graders/AMHARTGrader.py \
  --config configs/grading/default.yaml

# Mix YAML config with CLI overrides
python graders/AMHARTGrader.py \
  --config configs/grading/default.yaml \
  --temperature 0.5  # Override just the temperature

---

Output Files

Vignette Bank

data/vignette_bank/vignette_bank.csv

• Columns: id, title, vignette

Conversations

data/conversations/<model>/conversation_<id>_<model>_iter_<n>.csv

• Columns: turn, role, content

Graded Results

data/graded_conversations.csv

• Columns: vignette_id, criteria1-14, reasoning1-14, YN_NA1-14, score, percent_score

---

Troubleshooting

API Key Issues

Make sure your .env file contains:

AZURE_API_BASE_URL=your_azure_endpoint
AZURE_OPENAI_API_KEY=your_azure_key
OPENAI_API_KEY=your_openai_key

Module Not Found

Run from the AMHART root directory:

cd /path/to/AMHART
python main.py

Permission Denied

Make bash script executable:

chmod +x run_pipeline.sh

Analysis

Regression

Use regressions/long_data.py to take .csvs that have been graded by graders/AMHARTGrader.py and convert them into long-form for regression analysis.

Visualizations

Use visualizations/charts.Rmd to create figures with this long data.

Copyright and License

Copyright (C) 2026 RAND Corporation. This code is made available under the MIT license.