The sentiment-bench application validates labelled sentiment data, runs model
comparisons and stores their results. It is the earlier tooling developed during the
dissertation. To reproduce the final study, use the submission guide.
For a complete walkthrough without provider credentials, start with the offline portfolio demo. It validates the benchmark and runs the separate historical strategy pipeline on synthetic fixtures.
With Python 3.12 and uv installed, run from the repository root:
uv sync --locked
uv run --locked sentiment-bench validate-dataThis check runs locally. No credentials are needed.
The committed dataset has 5,947 rows: 2,884 neutral, 2,082 positive and 981 negative.
The validation table should report zero duplicate sentence groups and zero conflicting
labels. The dataset card records its Financial PhraseBank/FiQA sources
and the corruption corrected in the older Data/data.csv merge.
If the derived CSV is missing, rebuild it with python scripts/build_labeled_dataset.py.
To install development dependencies and run all benchmark checks, use
make benchmark-setup and make benchmark-check instead.
Install without uv, including Windows PowerShell
On macOS or Linux:
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
sentiment-bench validate-dataOn Windows PowerShell:
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"
sentiment-bench validate-dataThis pip fallback resolves dependencies from pyproject.toml rather than uv.lock.
With an activated virtual environment, omit uv run --locked from the commands below.
Model runs require a configured provider. For OpenRouter, add your key to a local .env
file, which Git ignores:
OPENROUTER_API_KEY=your-openrouter-keyList the available models:
uv run --locked sentiment-bench list-models --provider openrouter --limit 5Replace MODEL_ID below with an available model ID. This command sends benchmark text
to OpenRouter and incurs the selected model's usage charges:
uv run --locked sentiment-bench run --provider openrouter --models MODEL_ID --mode pilotPilot mode samples 30 rows per class with seed 42. Results are saved locally in
results/sentiment_benchmark.sqlite by default. For Ollama, Cerebras or a shared
Turso/libSQL store, follow the provider guide.
Find the run ID, then use it in place of 1:
uv run --locked sentiment-bench runs
uv run --locked sentiment-bench results --run-id 1 --confusion
uv run --locked sentiment-bench export --run-id 1Results include macro-F1, balanced accuracy, MCC, per-class scores, invalid-output and
API-error counts. The export contains response tables, metrics and prompt/run metadata
under results/exports/run_1/. See results and exports for the
file schemas and the optional plotting extra.
uv run --locked sentiment-bench tuiThe TUI uses the same services as the CLI and saves preferences in
results/tui_session.json. The TUI guide covers its tabs, keyboard
shortcuts and logs. News collection is a separate workflow with its own provider
credentials, described in news sourcing.