Mutmut: A Python Mutation Testing System
The author, building Python libraries, realized that even 100% code coverage can miss boundary conditions and error handling. Mutation testing works by making tiny changes to source code (e.g., turning < into <=) and then running the test suite; if the tests still pass, the mutation is considered unkilled, revealing weaknesses. After evaluating Mutpy and Cosmic Ray, the author decided to write Mutmut from scratch. A key design choice was using the baron library (later replaced by parso) to enable lossless AST round-tripping, so mutations can be written back to disk without disturbing formatting. The author attempted to intercept module loading via import hooks to mutate in memory and parallelize, but abandoned this after discovering that the Python import system forces reimplementing every loader. The disk-based serial approach was kept for its simplicity and flexibility across test runners. Applied to tri.declarative and tri.struct, Mutmut found untested edge cases and dead code even with 100% coverage, genuinely improving the test suites. This is a first-hand, honest account for engineers interested in Python testing tooling.