Harder maths problems, checked by code
Can a language model write harder maths problems, and can you check automatically that they're right?
Approach
For my MSc thesis (UvA, 2025), a teacher model rewrote maths problems into harder variants with a Python solution.
Result
A verifier ran that code and kept only variants that ran, gave the right answer and were new to the model: 6,213 problems from 7,460 originals, harder than the hand-made GSM-Symbolic benchmark. In a manual sample, 81 of 100 were fully correct. Honest result: fine-tuning on the new data gave no significant improvement.
Highlights
- 6,213 verified problems from 7,460 originals; in a manual sample, 81 of 100 were fully correct
- Harder than the hand-made GSM-Symbolic benchmark
- Honest result: fine-tuning on the new data gave no significant improvement
Technology
PythonQwen2.5LLM verificationSynthetic data
Let's connect
mannes@mokkenstorm.com