Skip to content
← Back

Harder maths problems, checked by code

Can a language model write harder maths problems, and can you check automatically that they're right?

Approach

For my MSc thesis (UvA, 2025), a teacher model rewrote maths problems into harder variants with a Python solution.

Result

A verifier ran that code and kept only variants that ran, gave the right answer and were new to the model: 6,213 problems from 7,460 originals, harder than the hand-made GSM-Symbolic benchmark. In a manual sample, 81 of 100 were fully correct. Honest result: fine-tuning on the new data gave no significant improvement.

Highlights

  • 6,213 verified problems from 7,460 originals; in a manual sample, 81 of 100 were fully correct
  • Harder than the hand-made GSM-Symbolic benchmark
  • Honest result: fine-tuning on the new data gave no significant improvement

Technology

PythonQwen2.5LLM verificationSynthetic data
© 2026 Storm IT SolutionsAmsterdam
NL