evalgate is an eval gate for LLM apps. You write a small set of questions with known answers, and every pull request has to hold its own against the base branch before it merges. The gate runs the same cases on the base commit and on the PR commit, compares them, and posts one comment on the PR.
Results for every gate run: https://ramaraihan.is-a.dev/evalgate/
I'm building it in small milestones. The first real user is text-to-sql-chat.
- Cases are lines in a
.jsonlfile:id,questionand eithergold_sqlor"expect": "no_sql"for a question the data can't answer. An answer counts as right when the rows match the gold query's rows (row order only matters for top-N queries). - The model is a small Ollama model on the GitHub runner, pinned by tag and digest. The job fails if the pulled digest isn't the full pinned one.
- Base and PR run as two parallel jobs on separate runners, each with its own Postgres, its own
setupfrom that commit, and its own Ollama, so they never share a prompt cache. The cases come from the PR merge commit for both sides. - Accuracy is shown per side with a Wilson 95% interval. The comparison is paired on the same
cases:
b= right on base and wrong on the PR,c= the other way round. The gate fails when the PR is less accurate and the exact one-sided McNemar p-value (a binomial onb + c) is below 0.05. - A wrong SQL, a blocked query or a model crash counts as a wrong answer. Ollama or Postgres being down fails the job as an error, not as a regression.
- It is a reusable workflow,
.github/workflows/gate.yml. Call it from apull_requestworkflow (notpull_request_target). On fork PRs the token can't comment, so the comment is skipped and the job summary has the report.
Honest limits: with about 50 cases only large drops are detectable. Even if nothing improved, at least 5 cases have to flip from right to wrong before the gate fails. A small local model is also not the model an app runs in production, so this tests the prompt, the schema notes and the pipeline, not the production model's quality.
The second app is docs-rag-assistant, a
question answering app over PDF, Markdown and DOCX files. Its gate is the reusable workflow
.github/workflows/gate-rag.yml and the adapter src/evalgate/adapters/docsrag.py.
- Cases are lines in a
.jsonlfile:id,question,fileandpageorsection, which is where the answer is, or"file": nullfor a question the documents don't answer. - Each side gets its own runner with pgvector, Ollama and the app checked out at that commit. The
samples are indexed from that checkout with that checkout's code, so chunking, retrieval and the
prompt are all part of what is tested. Then the app answers every question the way its
/api/askdoes. - No model grades the answers. For each case the gated result is:
- answerable: the expected passage is among the retrieved sources, and the answer cites at least one source number, every cited number exists, and at least one cited source is the expected passage;
- unanswerable: the answer says the documents don't cover it and cites no retrieved source. A
clause counts when it refers to the documents, has a negation and doesn't attribute the claim
to them ("according to the documents, ..." does not count). A short list of English and
Indonesian phrases, in
ragmetrics.py, with tests.
- The comment also shows, per side and not gated: retrieval hit rate, MRR, citation rate, citations per answer, the share of citations pointing at a file other than the expected one, citations to sources that don't exist, and the abstain rate. The comment also lists dataset cases the PR removed, changed or added.
- The comparison is the same as M1: paired on the cases, exact one-sided McNemar, fail when the PR is less accurate and p < 0.05.
- The gold is the expected file and page in the dataset, read from the merge commit for both
sides, and the grading code lives here. The gate guards against honest regressions. The PR's
own code runs on the head side, so changes to
.github/workflows/andevals/need a human review.
The chat model is small on purpose so one side finishes in about 7 to 12 minutes on a 4 vCPU runner, install included
(qwen3:1.7b for answers and qwen3-embedding:0.6b for retrieval, the app's own embedding
model, both pinned by digest). I picked it by running the real 40 cases on a few candidates:
| chat model | passed | citation rate | abstain rate | one side, whole job |
|---|---|---|---|---|
| qwen2.5:1.5b | 8/40 | 12% | 4/6 | 7 min |
| qwen2.5:3b | 27/40 | 71% | 3/6 | 8 min |
| llama3.2:3b | 27/40 | 62% | 6/6 | 11 min |
| qwen3:1.7b | 37/40 | 91% | 6/6 | 7 min |
(Retrieval hit rate was 100% for all of them. The 1.5B model answered correctly but almost never cited a source, so it would have measured its habits and not the app.) With about 40 cases only large drops are detectable, and a 1.7B model is not the model the live app uses, so this watches chunking, retrieval and the prompt, not the quality of the production model's answers.
Runners are not all the same: in my timing runs the same job landed on an AMD EPYC 7763, an EPYC 9V74 and a Xeon 6975P, and the 3B model took 26 s per question on the first and about 9 s on the last. Both sides record the CPU and the comment warns when they differ.
Gate artifacts expire after 90 days, so the history would disappear. evalgate collect-runs
reads the gate-report artifacts of the apps through the GitHub API (read-only), keeps the numbers
from each verdict.json plus the PR number, title and state, and merges them into
docs/data/runs.json. Old records are never dropped, so an expired artifact doesn't erase a run.
No case questions or answers are stored.
docs/ is a static page (plain HTML, CSS and JS, no external requests) that reads that file: the
latest run of the deliberately bad PR 2 in text-to-sql-chat at the top, then a chart and a table
of gate runs per app, and a section on what the gate can't catch. I run the collector by hand and
commit the result; nothing is scheduled.
jobs:
gate:
uses: ramaraihan12/evalgate/.github/workflows/gate.yml@v0.1.0
permissions:
contents: read
pull-requests: write
with:
app_repo: ${{ github.repository }}
pr_number: ${{ github.event.pull_request.number }}
model: qwen2.5-coder:3b
model_digest: f72c60cabf6237b07f6e632b2c48d533cef25eda2efbd34bed21c5e9c01e6225
evalgate_ref: v0.1.0The PR side is the merge commit GitHub builds for the PR and the base side is its first parent, so new commits on the base branch don't count against the PR. Both sides record the CPU, the vCPU count and the Ollama version (pinned, 0.40.16 by default); the comment warns when they differ. The gold rows come from the adapter's own Postgres connection, not from the app under test.
For documents, call gate-rag.yml the same way and also pass embed_model and
embed_model_digest. See docs-rag-assistant's .github/workflows/eval-gate.yml.
The adapter in src/evalgate/adapters/sqlchat.py knows how to run text-to-sql-chat. A new app
needs an adapter with preflight, describe and either ask and gold_rows (SQL) or
run_case (the adapter grades a case itself, like docsrag.py).
Before building the gate I measured how fast small Ollama models run the pipeline on a plain
GitHub runner (4 vCPU, no GPU). bench/m0.py and the m0-bench workflow are still here.
For qwen2.5-coder:3b, 12 questions took about 95 seconds.
The harvest workflow turns real traffic into candidates for the eval sets. The code is public
here, but the review queue, the weekly schedule and the Langfuse read secrets live in a private
repo that calls this workflow, so redacted visitor questions are never published. The workflow is
reusable only (workflow_call); it has no schedule of its own.
What it reads. The last 7 days of root answer generations from Langfuse Cloud
(environment=production, v2 observations API, read-only Viewer key), kept only when the app
set metadata.flags (for example no_sources, no_citation, slow, error). The apps are
docs-rag and sql-chat. Questions already in the caller's review queue, in
evals/rag_cases.jsonl (docs-rag-assistant) or in evals/sql_cases.jsonl (text-to-sql-chat) are
skipped, matched on the lowercased, whitespace-collapsed question. At most 50 rows per app per
run are kept and the log says how many were dropped. Test and synthetic: traces are never
written.
What it writes. review-queue/<app>/<year>-W<week>.jsonl in the caller repo, one row per
question: trace_id, observation_id, app, question, answer_excerpt (300 characters at
most), flags, and empty label, expected, note for a human to fill in. New rows go on a
harvest/<year>-W<week> branch (harvest/<year>-W<week>-<run number> if that branch already
exists) and a pull request. With nothing new there is no PR.
Privacy. The apps redact emails, phone numbers, long digit runs and token-like strings before a trace leaves them, so Langfuse only ever holds redacted text. Rows carry no IPs or headers, and they are written only to the private caller repo.
Calling it. In the private repo, .github/workflows/harvest.yml:
name: weekly harvest
on:
schedule: [{cron: "0 3 * * 1"}]
workflow_dispatch:
jobs:
harvest:
permissions: {contents: write, pull-requests: write}
uses: ramaraihan12/evalgate/.github/workflows/harvest.yml@<tag>
with: {evalgate_ref: <tag>}
secrets: inheritThe private repo needs the secrets LANGFUSE_HOST, LANGFUSE_READ_PUBLIC_KEY and
LANGFUSE_READ_SECRET_KEY, and "Allow GitHub Actions to create pull requests" turned on. Locally:
uv run evalgate harvest --out review-queue with the same three variables set.
uv sync
uv run pytest
# refresh the dashboard data (needs GH_TOKEN or a logged-in gh)
uv run evalgate collect-runs --repo ramaraihan12/text-to-sql-chat=sql-chat --repo ramaraihan12/docs-rag-assistant=docs-rag --out docs/data/runs.json