Goal: be able to evaluate math competition/textbook/past papers

Edit this issue to research what kinds of mathematical expressions a math student from primary education all the way to tertiary education may encounter, whether in textbooks, past papers or competitions. Proceed to extract examples and make sure we can solve them (albeit with nuances that might not be mentioned at lower education levels, that need to be mentioned specifically). New competitions, textbooks, past papers come out all the time, monitor and track them.

Source survey, and a workflow that can actually reach full coverage

This started from someone asking whether Humanity's Last Exam is a target for
this goal. Short answer: almost none of the famous AI-era math benchmarks are, and the ones that
matter for us are older, duller and far larger. The long answer is a survey, a licensing line we
should not cross, and a loop.

Everything below was checked against the source rather than recalled; where a fact could not be
confirmed it says so.

1. The distinction that decides which sources are useful

An LLM math benchmark scores an answer to a natural-language question. Our unit of work is
an expression and a transformation of it, and the thing that makes a corpus valuable to us is not
its difficulty but whether the result is checkable in-process — substitute the root back,
differentiate the antiderivative back, sample the simplification numerically. That is what
work/casbench and work/rootcheck already do, and it is why they can be re-run on every commit
while a leaderboard cannot.

So every candidate source gets two questions, not one:

Is an item mechanically expressible as Solve / Integrate / Limit / Simplify / Differentiate?
Can its answer be verified without trusting a human-written key?

A source that fails (1) is a mining source at best: we read it to learn which shapes students
meet, then build our own instances. A source that passes both is a corpus we can run.

2. Tier A — CAS-native test suites. Run these.

These are aimed at computer algebra systems, not at language models, and they are the highest-value
material by a wide margin.

Rubi's integration test-suite. Over 72,000 integration problems, organised by the form of the
integrand, each stored with the integration variable and an optimal antiderivative valid for all
real and complex values — and published in Axiom, Maple, Mathematica and Maxima syntax as zip files:
https://rulebasedintegration.org/testProblems.html. The top-level split (algebraic functions,
exponentials, logarithms, trig, inverse trig, hyperbolic, …) is a ready-made taxonomy for
int: categories in coverage.md, which currently has 13 hand-written ones. The site calls the
suite "freely available" but states no licence, so: usable out-of-tree in work/ today,
not vendored into an MIT repo until someone gets that in writing.

Nasser Abbasi's Computer Algebra Independent Integration Tests,
https://www.12000.org/my_notes/CAS_integration_tests/index.htm — last compiled 2025-08-31, running
the suite above against Maple, Mathematica, Rubi, Fricas, Sympy, Maxima, XCas and Giac at the same
time. This gives us something no self-made corpus can: a per-problem external signal of whether an
integral is reasonably solvable at all. An integral that six systems get and we miss is a defect
with a known ceiling; one that nobody gets is not our bug. There is a sibling suite for ODEs
(https://www.12000.org/my_notes/CAS_ode_tests/index.htm) which becomes relevant the day we have an
ODE solver.

Wester's CAS capability review (1999), the paper the whole practice descends from:
http://www.math.unm.edu/~wester/cas/book/Wester.pdf. SymPy encodes it as
test_wester.py
with a documentation page at https://docs.sympy.org/latest/modules/polys/wester.html. Its value here
is breadth over difficulty: it enumerates areas a CAS is expected to have at all, so it doubles as
the gap list for #717 and as evidence for #746's argument about what a platform owes its callers.
(I did not verify the often-quoted problem count, so I am not quoting it.)

Licensing trap, stated once. AngouriMath is MIT. Maxima's test suite is GPL and SymPy's is
BSD-3; neither's test data can be lifted into this repo without dragging its terms along. Reading
them to learn what to test is fine. Copying them in is not.

3. Tier B — permissively licensed problem sets. Mine these.

Source	Size	Licence (as published)	Use to us
MATH (Hendrycks)	12,500 competition problems	MIT	Its algebra / intermediate-algebra / precalculus subsets are the closest large pool to our shape
Omni-MATH	4,428 olympiad problems, 33+ sub-domains, 10 difficulty levels	Apache-2.0	Upper bound of "competition"; mostly proof-shaped, mine sparingly
NuminaMath	~860k problems with worked solutions	Apache-2.0	Largest mining pool; the solutions name the manipulation steps, which is what we want
U-MATH	1,100 university-level problems, six subjects, ~20% multimodal	MIT	The best match for the "tertiary education" half of this issue
DeepMind `mathematics_dataset`	56 generator modules	Apache-2.0	See below — this is the important one

mathematics_dataset is not a corpus, it is a generator: modules for calculus__differentiate,
polynomials__expand, polynomials__simplify_power, algebra__linear_1d and 50-odd more, each able
to emit unbounded instances with the answer known by construction. That is exactly the shape that has
worked in this repo already — work/README.md records that rootcheck can see missing roots
because it multiplies polynomials up from known factors, and that this found #744 and the
incompleteness behind #272, which substitution-checking structurally cannot see. A generated corpus
scales, cannot be contaminated, and carries no copyright. This should be the spine of the goal, and
the fixed corpora should be the ribs.

4. Tier C — the AI benchmarks proper. Mostly not for us.

HLE — 2,500 questions, MIT-licensed, text and image, published at
https://huggingface.co/datasets/cais/hle. Mathematics is its largest slice, but the items are
research-level short-answer and multiple-choice questions whose difficulty lives in knowing which
mathematics applies, not in carrying out a manipulation. There is close to nothing here a CAS can be
scored on. It also ships a canary string and asks explicitly never to be included in training
corpora — worth respecting even for a test corpus, so: reference it, don't vendor it. Verdict:
interesting, not a target.

FrontierMath (Epoch AI) — 300 problems in tiers 1–3 plus 50 at tier 4, deliberately held out and
not public: https://epoch.ai/frontiermath/tiers-1-4/about. Unusable directly, but instructive: Epoch's
own 2026 audit corrected errors in a large fraction of the set (their write-up puts it at 42% of
problems, https://epoch.ai/benchmarks/frontiermath-tiers-1-3-v2). A hand-built key is a liability at
scale. Every point in our corpus should be machine-verifiable, and where it cannot be, it should be
labelled as trusted-by-hand.

MathArena — https://matharena.ai/, evaluating recurring competitions (AIME, HMMT, USAMO, BRUMO,
SMT) immediately after each one runs, precisely to stay uncontaminated. For us its value is the
calendar: it is a maintained feed of which competitions have just published, which is the
monitoring input this issue asks for and which nobody should be re-discovering by hand.

Formal-proof benchmarks — PutnamBench (1,709 Putnam problems formalised in Lean 4 / Isabelle,
some in Coq), miniF2F (244 + 244), ProofNet (371 undergraduate analysis/algebra/topology problems).
Not scoreable by us today. Worth knowing they exist, because a machine-readable statement of a
theorem is the format #746's derivation objects would eventually want to emit against.

5. Tier D — curriculum sources. These give the taxonomy, not the items.

This is where "primary education all the way to tertiary" actually gets pinned down, and where the
copyright line matters most.

OpenStax (Prealgebra through Calculus vol. 1–3) — openly licensed, but check per book: editions
differ between CC BY 4.0 and CC BY-NC-SA 4.0, and OpenStax states the books may not be ingested
into LLMs without permission. Use for structure and topic ordering.
Exam-board specifications and released papers — the AP Calculus AB/BC Course and Exam
Description plus the three most recent years of released free-response questions are published free
by the College Board (https://apcentral.collegeboard.org/courses/ap-calculus-bc/exam); Cambridge,
Edexcel, AQA, IB and the Dutch examenblad.nl syllabi are likewise public documents. The papers
are copyrighted.
Competition archives — IMO problems and shortlists at https://www.imo-official.org/problems.aspx
(copyright IMO). The AoPS wiki has not settled on a documentation licence
(https://artofproblemsolving.com/wiki/index.php/AoPS_Wiki:Copyright) and its own policy warns
contributors off importing text — so it is a reading source only.

The line, concretely. "A B-level paper expects you to solve |ax + b| < c for x, and to
integrate x·e^(kx) by parts" is a fact about a curriculum and we can write it down. The forty
worked exercises that teach it are an authored selection and we cannot copy them. So the workflow
below mines shapes and generates instances. That is not a compromise forced by licensing; it is
the approach that has already outperformed corpora in this repo.

6. Proposed workflow

Six stages, each leaving a re-runnable artifact rather than a claim.

1. work/syllabus.md — the grid. One row per cell of level × topic × shape, from primary
arithmetic to tertiary analysis. A cell is (level, topic, shape template, expected answer kind, source that says it belongs at this level). Built from Tier D specs plus the Rubi and Wester
category trees, which between them cover far more of the integral and algebra space than our current
41 coverage.md categories. The grid is the definition of "full coverage" — without it, "full" is
unfalsifiable.

2. work/examgen — generators per cell. Same pattern as rootcheck and simpsweep: build the
instance from a construction whose answer is known before the library is called, so the harness can
detect a missing or incomplete answer and not merely a wrong one. Where an answer cannot be known
by construction, fall back to numeric verification: sample at random points including negative ones,
relative tolerance, and treat NaN as a failure only where the original had a value.

3. Verification ladder, strongest first. Known-by-construction → invertible check (substitute
roots back, differentiate antiderivatives back) → numeric sampling → external agreement (12000.org's
published per-problem results, work/libcompare for Math.NET Symbolics and Symbolism, SymPy for
#717) → hand-checked, explicitly labelled as such and kept small.

4. Nuance capture — the part this issue asks for by name. Each cell records what the school-level
answer hides: domain restrictions, principal branches, ±, real-vs-complex root counts, division by a
possibly-zero factor. We already emit this: coverage.md has
(x^2 - 1)/(x - 1) → x + 1 provided not x - 1 = 0. Turning those into a checked column means this
goal simultaneously produces the test set for #746's claim that answers should carry their conditions.

5. Reporting — extend coverage.md with a level axis. A level × topic table with the
solved / unevaluated / wrong split kept separate, because they are different kinds of news: a
wrong answer stops the line, an unevaluated one is a feature request. Only then does "we can do a
tertiary calculus paper" become a number someone can re-derive.

6. Monitoring — a scheduled re-run, not a resolution. The issue asks for tracking, so the loop
needs a trigger list: MathArena's competition feed, the AIME/HMMT/Putnam/IMO calendar, new exam-board
specification versions, new Rubi releases and each summer's 12000.org rebuild. Quarterly re-run;
every new miss enters triage like any other defect.

Rules to keep it honest, in the spirit of AGENTS.md:

No claim that a paper is passable from spot checks. The number comes from the harness or it is not
made.
unevaluated and NaN stay distinct; conflating them is a wrong answer, not a rounding of one.
Nothing copyrighted is vendored. Every corpus row carries a source and a licence column, and
anything non-permissive is referenced by URL with a generated equivalent standing in for it.
Growth of the grid is allowed to reduce the coverage percentage. A goal issue that can only
improve its own metric is measuring the metric, not the library.

7. How this sits against the other goals

Goal: Math OS — a ten-year vision for AngouriMath as an open mathematical reasoning platform #746 — this is that vision's evidence base. Worth noting that at Rubi scale the corpus also
makes the "split common-use from edge cases" suggestion in that thread measurable: we would be
able to say which rules only ever fire for a fraction of a percent of real problems, instead of
guessing. The same corpus, timed, is the speed-and-memory regression net asked for there.
Goal: feature parity with sympy #717 — Wester's review and the 12000.org tables are already cross-CAS, so the same run that
scores us produces the SymPy parity column for free.
we need more integral solvers #233 — coverage.md's remaining misses are all integrals, and two of the four are
non-elementary. Tier A is exactly the material that says how far the integrator can go.

First concrete step

I would build stage 1 and 2 for a single vertical slice — secondary-level integration, where Rubi's
category tree gives the shapes and the 12000.org results give an external ceiling — and only widen
the grid once that slice reports a number that survives a re-run.

First slice measured: textbook integration, 536/1774

Stage 1 and 2 of the workflow above, built for one vertical slice. The result is worth
posting on its own, because it disagrees sharply with the number this project currently
quotes itself.

What was run

Rubi's integration test-suite has a directory called Independent test suites: problems
lifted from textbooks and papers rather than generated from a rule taxonomy — Stewart's
Calculus, Apostol, Timofeev's integration textbook, Hearn, Moses, Charlwood, Welz and
others. That is the education-level slice this issue is about, and unlike a corpus we
write ourselves, we did not choose it.

Every problem carries an antiderivative, so the harness can tell a gap from an
impossible integral without anyone judging by hand. Answers are still graded our own
way — differentiate ours, compare to the integrand numerically at real sample points.
Rubi's answer decides only whether the problem is fair, never whether we are right.

1,892 problems read
2 excluded: the integrand uses a function AngouriMath has no node for
116 excluded: the recorded antiderivative is non-elementary, so the problem is unfair
1,774 run, 5s budget each

Result

536 answered — 30.2%. 1,194 unevaluated, 7 wrong, 8 unverifiable, 29 timeouts.

Source	Run	Answered
Apostol	159	52%
Moses	107	50%
Stewart	376	45%
Hearn	259	40%
Hebisch	4	25%
Timofeev	666	19%
Bronstein	10	10%
Welz	104	3%
Charlwood	50	2%
Bondarenko / Jeffrey / Wester	39	0%

By Rubi's own step count — an outside difficulty measure, so it says where our ceiling is
rather than where we put it:

Rubi steps	Run	Answered
1 (table lookup)	268	57%
2–3	851	36%
4–6	432	16%
7–12	127	6%
13+	48	6%

The ceiling is not somewhere exotic. It is already visible at two rule applications.

Why this matters more than the number itself

work/casbench reports 113/117 — 96.6% — on a corpus of 117 problems we wrote. Both
figures are correct measurements of different things, and only one of them is a
measurement of the library. A first-year calculus textbook is at 45%.

This is the argument for the taxonomy-and-generator loop rather than more hand-written
cases: a corpus we choose cannot surprise us, and the first corpus we did not choose
moved the number by two thirds.

What it found

Seven wrong answers, and all seven are one defect: a symbolic coefficient on x^2 makes
the quadratic integration rules answer NaN — filed as
#771. A second, cosmetic one came out
of the same run and is
#772.

The other 1,194 misses are unevaluated rather than wrong, which is the honest failure
mode, and every one of them has an elementary antiderivative somebody else already found.
That is the real size of #233.

Three things the harness got wrong first

Recording these because each one invented defects before it was fixed, and any
future stage of this workflow will meet them again:

d/dx ln|u| carries a provided condition, and a provided node has no numeric
value — so every answer containing abs() failed verification regardless of whether
it was right.
A real sample point can lie outside the integrand's real domain. 1/sqrt(x^2-1)
at x = 0.37 is purely imaginary, and abs() in a correct antiderivative is not
holomorphic there, so the comparison is meaningless rather than failing. The
comparison is three-valued now — match, differs, unverifiable — and the unverifiable
cases are listed individually rather than counted, because a silent bucket reads as a
clean one.
Some entries list two acceptable antiderivatives, so they have five fields rather
than four; 91 were being dropped as unparseable.

The first two are the reason a self-verifying harness has to be verified itself. Four
"wrong answers" in the first run were artifacts of (1) and (2), not defects, and I had
written them up as bugs before checking.

On not vendoring the corpus

The suite states no licence, so it is downloaded rather than committed, and the harness
takes a path to it. Same rule as in the survey above: mine the shapes, keep the
provenance, do not copy the problems in.

Measured status for this goal, on master at a45a7256, since it has never carried a number.

On problems taken from an outside source, integration answers 30%.

work/intbench runs Albert Rich's Rubi test-suite — 1892 problems, of which 1774 are fair (116 excluded because Rubi's own antiderivative is non-elementary, 2 because the integrand uses a node AngouriMath lacks). Answers are graded by differentiating them back and comparing numerically to the integrand, never against Rubi's answer:

Answered 536/1774 (30.2%) — 1194 unevaluated, 7 wrong (6 of them NaN), 8 unverifiable on the reals, 0 error, 29 timeout.

On the corpus we wrote ourselves, it answers 97.5% — casbench reports 116/119 with 0 wrong, 0 error, 0 timeout.

Both numbers are correct measurements of different things, and the gap between them is the most useful fact for this goal: a corpus you assemble yourself measures the corpus, not the library. Competition and past-paper problems are outside-source problems, so 30% is the figure to plan against, not 97.5%.

Three things follow that seem worth recording here rather than discovering later:

The 7 wrong answers matter more than the 1194 unevaluated ones. An unevaluated integral is a refusal a caller can see; a wrong antiderivative is not. Six of the seven are NaN, which this library treats as "does not exist" — and I filed two separate defects this week where NaN was being used to mean "I could not do this" (#958, and the same confusion in the solver at #964). Those seven are worth reading individually before any of the 1194.

29 timeouts at a 5-second budget are a different failure again — the answer may exist and simply cost too much, which is #896's territory.

The remaining unticked boxes on #233 are the same two problems the corpus leaves unsolved, so that issue and this goal agree about where the frontier is.

The cosecant, the secant, and a linear denominator in the cosine, by …

A radical over something linear is a substitution, and 35 more Rubi i…

The tangent, cotangent, secant and cosecant reach the half-angle subs…

An integrand rational in e^(k x) is a rational function under u = e^(…

A product of trigonometric functions of different arguments is a sum,…

The tangent, cotangent, secant and cosecant reach the half-angle subs…

A symbolic parameter no longer stops a rational integrand being integ…

One integrand written two ways got one answer and one refusal, becaus…

A sum is split before anything searches it, and a polynomial stops ta…

One integrand written two ways got one answer and one refusal, becaus…

A sum is split before anything searches it, and a polynomial stops ta…

An odd power of a quadratic's derivative over a power of it, in the q…

A power of x past the twelfth beside a block is split off (#1840)

A symbolic cubic's linear factors are found before the partial fracti…

An odd rational function with symbols in it is integrated in the squa…

Every linear beside a block is split off, a quadratic with rational r…

What a reduction leaves over its block is integrated in named coeffic…

A trinomial the numerator holds is divided out of a power of it (#1845)

An imaginary sum of a cosine and a sine beside powers of the secant, …

A power of a quadratic with symbols in its outer coefficients is inte…

A sine's binomial cube below the bar is split over the cube's two fac…

A power of the cosine beside a secant binomial is integrated in the c…

Clear a sum with the root of a quadratic below the bar by its conjuga…

Take a whole power out of a power of a product, in front of a constant (

Split a root of a product beside a root of one of its factors (#1876)

Integrate a trigonometric function of a + b/x beside a rational funct…

Navigation Menu

Goal: be able to evaluate math competition/textbook/past papers #718

Source survey, and a workflow that can actually reach full coverage

1. The distinction that decides which sources are useful

2. Tier A — CAS-native test suites. Run these.

3. Tier B — permissively licensed problem sets. Mine these.

4. Tier C — the AI benchmarks proper. Mostly not for us.

5. Tier D — curriculum sources. These give the taxonomy, not the items.

6. Proposed workflow

7. How this sits against the other goals

First concrete step

First slice measured: textbook integration, 536/1774

What was run

Result

Why this matters more than the number itself

What it found

Three things the harness got wrong first

On not vendoring the corpus

285 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Goal: be able to evaluate math competition/textbook/past papers #718

Description

Activity

Rafael-SOWNet commented on Aug 6, 2026

Source survey, and a workflow that can actually reach full coverage

1. The distinction that decides which sources are useful

2. Tier A — CAS-native test suites. Run these.

3. Tier B — permissively licensed problem sets. Mine these.

4. Tier C — the AI benchmarks proper. Mostly not for us.

5. Tier D — curriculum sources. These give the taxonomy, not the items.

6. Proposed workflow

7. How this sits against the other goals

First concrete step

Rafael-SOWNet commented on Aug 6, 2026

First slice measured: textbook integration, 536/1774

What was run

Result

Why this matters more than the number itself

What it found

Three things the harness got wrong first

On not vendoring the corpus

Rafael-SOWNet commented on Aug 16, 2026

285 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions