Mutmut：用变异测试揪出 100% 覆盖率背后的盲区
作者在维护 Python 库时发现，即便测试覆盖率 100%，仍可能漏掉边界条件和异常行为。变异测试通过在源码上做微小改动（如把 < 改成 <=）并运行测试套件，可以暴露出这些盲区。作者调研了 Mutpy 和 Cosmic Ray 后，决定自己实现一个变异测试工具 Mutmut。核心设计是使用 baron（后替换为 parso）实现 AST 无损往返，让变异后的代码可以以原格式落盘，便于审查具体变异。作者尝试用 import hook 在内存中变异模块以支持并行，但受限于 Python 导入系统必须重写全部 loader，最终放弃并退化为基于磁盘的串行变异，换来简单性和对不同测试运行器的兼容性。在 tri.declarative 和 tri.struct 上运行时，虽未发现 bug，却找到了未充分测试的边界和死代码，实际改善了测试套件。适合对 Python 测试工具链和变异测试感兴趣的工程师阅读。
Skip to “How hard can it be?” if you don’t care about the background.
What is mutation testing?
Mutation testing is a way to be reasonably certain your code actually tests the full behavior of your code. Not just touches all lines like a coverage report will tell you, but actually tests all behavior, and all weird edge cases. It does this by changing the code in one place at a time, as subtly as possible, and running the test suite. If the test suite *succeeds *it counts as a failure, because it could change the code and your tests are blissfully unaware that anything is amiss.
Examples of mutations are changing “<” to “<=”. If you haven’t checked the exact boundary condition in your tests, you might have 100% code coverage but you won’t survive mutation testing.
如果你不在乎背景，可以直接跳到“这能有多难？”那部分。
什么是变异测试？
变异测试用合理的确定性来验证代码是否真正测试到了全部行为，而不只是像覆盖率报告那样告诉你所有行都被执行过。它会一次只在代码中做一处尽可能细微的改动，然后运行测试套件。如果测试套件通过了，反而算失败——因为代码已经被改掉了，你的测试却毫无察觉。
变异的一个例子是把“<”改成“<=”。如果你的测试没有检查精确的边界条件，也许覆盖率是 100%，却过不了变异测试。
Background
I wanted to try out mutation testing for libraries I build in Python so I looked at what was available. I had some ideas for ways of radically speeding up mutation testing based on what pytest-testmon does and adding some ideas of my own on top.
Googling showed me two alternatives:
Mutpy: a simple system developed as a thesis. Not maintained anymore.
Cosmic Ray: actively developed. Both these are Python 3 only which is a bit sad because I’m still on Python 2 for work at least for a year more. But I could live with that since the libs are Python 2 and 3.
背景
我想为自己用 Python 写的库试试变异测试，于是看了看现有工具。我基于 pytest-testmon 的做法，再加上一些自己的想法，对如何大幅加速变异测试有些思路。
用 Google 一搜，找到两个选择：
Mutpy：一个作为毕业论文开发的简单系统，已不再维护。
Cosmic Ray：仍在积极开发。 这两者都只支持 Python 3，这有点遗憾，因为工作中我至少还得再用一年的 Python 2。不过我也能接受，毕竟这些库本身要同时兼容 Python 2 和 3。
Cosmic Ray seemed the more promising so I tried installing it but after struggling just to get the dependencies installed I decided that if it’s this hard just to install it probably won’t be practical. I looked into the code a bit to see if I could use just the mutation parts as a library but it looked to me like a big monolithic system so I gave up on that.
Cosmic Ray 看起来更有前景，所以我试着安装它。可光是装依赖就折腾得够呛，于是我心想：装起来都这么费劲，用起来大概也不现实。我又翻了翻它的源码，看能不能只把变异部分当库来用，但在我看来它是个庞大的单块系统，于是放弃了。
Next I looked at mutpy. This code is radically smaller and simpler but after struggling to refactor it in some ways to make it even simpler I thought to myself:
How hard can it be?
Turns out, not that bad! Mostly the building blocks are already available.
I decided that I absolutely wanted a feature both Cosmic Ray and mutpy lacked: being able to apply a mutation on a source file and not screw up the entire file. Cosmic Ray and mutpy use Pythons built in AST library but it has the unfortunate property of not representing formatting in the AST, making it impossible to just dump the AST back out and get the original file. So if I can’t use Pythons own AST, what then? Enter baron, an independently developed Python->AST library specifically created to be able to round trip without changing your source files. Baron doesn’t support all of Python 3 syntax yet unfortunately, but it looks like people are working on it.
[EDIT: Since this article I’ve replaced Baron with Parso and now I fully support Python 3!]
接着我看了 mutpy。它的代码体量小得多，也简单得多，但我在尝试用它做进一步简化时，怎么改都别扭，于是心里想：
这能有多难？
结果发现，还真没那么难！大部分积木本来就已经现成。
我特别想要一个 Cosmic Ray 和 mutpy 都没有的功能：对源码文件做变异时，不要搞得整个文件面目全非。Cosmic Ray 和 mutpy 用的是 Python 内置的 AST 库，但那个库有个要命的缺点：AST 里不保存格式信息，所以你不可能把 AST 原样倒回去得到原始文件。不能用 Python 自己的 AST，那用什么呢？答案是 baron，一个独立开发的 Python→AST 库，它的设计目标就是能往返转换而不改动你的源文件。可惜 baron 还不支持全部 Python 3 语法，不过看起来有人在推进。
[编者按：这篇文章之后，我已经用 Parso 替换了 Baron，现在完整支持 Python 3！]
My battle plan was this:
Make a mutate function that receives source code and can mutate everything relevant (so you can get a count of available mutations) or a specific mutation specified by an index.
Make an import hook so that the file you want to mutate is mutated in memory on the way from disk. This will enable parallelization.
Pytest plugin that sets up the import hook and enables you to specify what mutation you want.
Make a small command line program that runs through the mutations and checks the output from tests. It should also be able to apply a specific mutation on disk, so when you find an interesting one you can see very easily what the mutant was.
我的作战计划是这样的：
写一个 mutate 函数，接收源码，既能变异所有相关位置（这样能统计有多少种可用的变异），也能按索引指定某一处变异。
做一个 import hook，让想要变异的文件在从磁盘读入时，于内存里完成变异。这样才能并行化。
写一个 pytest 插件，负责设置 import hook，并让你指定要跑哪个变异。
再做一个小命令行程序，遍历各种变异并检查测试输出。它还得能把特定变异直接应用到磁盘文件上，这样当你发现一个有意思的变异时，能很直观地看到变异体长什么样。
Point 1 was fairly easy: basically I needed to make sure all AST node types were either not mutated (because it doesn’t make sense) or mutated in the most nasty way I could think of. In this step I ran through the code of lots of big open source projects (e.g. django and numpy). I found some parse bugs in baron at this step, but nothing that impacted the code I wanted to run mutation testing on. I just reported the bugs and moved on.
第 1 步相当轻松：基本上，我要确保所有 AST 节点类型要么不变异（因为变异没有意义），要么以我能想到的最刁钻方式变异。这一步里，我翻了很多大型开源项目的代码（比如 django 和 numpy）。过程中发现 baron 有一些解析 bug，但没有一个会影响我想做变异测试的代码。我只管上报 bug，然后继续往前。
Point 2 was nasty. It turns out the import hook system in python is pretty shit. The default behavior to load from the filesystem isn’t in the list of hooks because it’s in C code somewhere, so you can’t base an importer on it. That would be ok if the design was ok, but unfortunately the import hook system works like this: Python asks one import hook at a time to import the module. That sounds simple and simple is often good, but importing actually contains these steps:
Find the source file
Read the source file
Convert the source file from text to a runnable module
And all importers must do ALL of the steps. So the zip file importer must do the steps that are the same as the default and it can’t just call into the default loader because it doesn’t exist as python code. And it also means that if I want to intercept between step 2 and 3 on ALL importers I have to reimplement all importers.
This obviously sucks (and might not even be possible, for systems with their own custom importers), but even worse is that implementing an import hook correctly at all is a lot of nontrivial code that is a bloody beast to get right. Supposedly this is somewhat better in python 3 with importlib, so I found a backport of it to python 2 but it was broken. I managed to hack around the crashes but in the end I didn’t get my import hook working with that either. I asked for help on reddit too, to no avail.
After several hours of fighting this fight I gave up (for now) and just went with disk based mutation. It’s not great because it can’t be run in parallel but at least it works and it’s super simple. It’s also very flexible with regards to which test runner you use, since you don’t need any plugins that would have to be made for pytest, nose, unittest, etc, one by one. Giving up on this means Point 3 becomes moot, so that’s great.
Basically I should have made this thing first anyway, because it’s very good to have :P
第 2 步很恶心。事实证明，Python 的 import hook 系统相当烂。默认的从文件系统加载行为并不在 hook 列表里，因为它在某个 C 代码里，所以你没法在它基础上写 importer。如果设计合理倒还好，可惜 import hook 系统的工作方式是这样的：Python 一个接一个地问 import hook 要不要导入这个模块。听起来很简单，简单通常是好事，但一次导入其实包含这几步：
找到源文件
读取源文件
把源文件从文本转换成一个可运行的模块
而且所有 importer 都必须完成全部步骤。于是 zip 文件 importer 也得做和默认加载相同的步骤，却又没法直接调用默认 loader，因为默认 loader 并不是以 Python 代码形式存在的。这也意味着，如果我想在所有 importer 的第 2 步和第 3 步之间做拦截，就必须把每个 importer 都重新实现一遍。
这显然很糟糕（对使用自定义 importer 的系统来说甚至可能根本做不到），更糟的是，正确实现一个 import hook 本身就需要一大堆不简单的代码，真的是个极难搞对的庞然怪物。据说 Python 3 的 importlib 会好一些，于是我找了一个移植回 Python 2 的版本，但它有问题。我勉强修掉了崩溃，可最后 import hook 还是没能跑通。我也去 Reddit 求助过，没有结果。
跟它搏斗几个小时后，我（暂时）放弃了，改回基于磁盘的变异。这种方案不好，因为没法并行，但至少能用，而且超级简单。对测试运行器也非常灵活——你不必为 pytest、nose、unittest 等逐个写插件。放弃 import hook 也让第 3 步失去了意义，这倒省事。
其实我本来就该先做这个，因为它非常有用 :P
Point 4 was pretty easy. The hardest was finding out how to nicely output continual progress updates on the console :P
So where do I stand now?
We’ve run mutmut on tri.declarative and tri.struct at work and it found several things we didn’t test as thoroughly as we thought, even though we had 100% coverage on our tests. For tri.declarative it also found a piece of dead code and an error in correctly creating a plural in an error message. It clearly improved our test suite, even though it didn’t find any bugs.
第 4 步也相当简单。最难的只是琢磨怎么在控制台上友好地持续输出进度 :P
那我现在进展如何？
我们在工作中用 mutmut 跑了 tri.declarative 和 tri.struct，它发现了几个我们以为自己测得很充分、其实并没有测到的地方，尽管测试覆盖率是 100%。在 tri.declarative 上，它还发现了一段死代码，以及一条错误消息中复数形式生成不正确的问题。虽然没发现 bug，但明显改善了我们的测试套件。
You can probably just run mutmut right now. It’s a pretty simple piece of code and for it to work with your test runner it just requires that it has an exit code of zero for success and anything else for failure. It’s pretty slow obviously since it’s not parallelized at all and it has to run the entirety of whatever test suite you specify for every mutation (and there are many!)
你现在大概就能直接跑 mutmut。它是一段相当简单的代码，要让 mutmut 配合你的测试运行器，只需满足一个条件：成功时退出码为 0，失败时是其他值。显然它也挺慢，因为完全没有并行化，每跑一个变异都要把整个指定测试套件完整执行一遍（而变异数量非常多！）
What’s next?
I still have some work to do. Using pytest-testmon is still on my list, as is solving the import hook system to enable parallelization. Those things alone should be able to give me orders of magnitude faster tests. The goal is to keep the super simple system I have now so that there’s always a simple to debug and adapt system that you can use for weird scenarios or debugging.
I also have some ideas for pytest-testmon like being able to keep a central database shared between developers, which could also be used for mutmut so if someone has already tried running a specific mutation for a specific version of a file, you don’t have to.
接下来呢？
我还有些事要做。接入 pytest-testmon 仍在计划里，解决 import hook 系统以实现并行化也是。光凭这两项，测试速度应该就能快上几个数量级。我的目标是保留现在这个超级简单的系统，这样始终有一个容易调试、容易改动的实现，供你在奇怪场景或调试时使用。
另外我对 pytest-testmon 还有一些想法，比如维护一个开发者之间共享的中央数据库。它也可以用于 mutmut：如果已经有人对某个版本的文件跑过某个特定变异，你就不必再跑一遍。