「短绳」编程法：人类把关，AI编码质量胜过Fable
本文总结了一年多来在安全关键系统中使用 AI 代理编码的研究成果。作者提出“短绳法”：只有资深开发者才能使用，全程禁止 YOLO 模式，开发者必须逐 diff 审查 AI 提议的每次改动，及时打断跑偏倾向，每个子任务结束后提交，从而杜绝“氛围编程”带来的代码鬼祟。文章还给出了 AI 辅助代码评审的最佳实践——人类与 AI 共同审查 PR，AI 负责快速扫描常见错误，人类把关方向性决策；并且要求作者在提交 AI 辅助 PR 前必须自我逐行审查并加上 AI 使用声明。该方法即便使用非前沿模型，也能产出胜过 Fable 的高质量代码。适合希望用 AI 提升效能但不牺牲代码质量的资深工程师阅读。
This post is the culmination of over a year of research into how to properly use AI agents to write high-quality software in security-critical systems.
I will be writing this post primarily from my perspective as a software developer, protocol developer, and maintainer of security-critical software.
Over the past year I dove deep into AI agents. I have explored their limits, what they can and cannot be relied upon to do. I’ve created our own AI review tools that perform just as well as multi-billion dollar AI-review systems. I’ve maintained my own custom fork of an AI coding agent called Crush. And this post is my distillation of what I’ve learned to be the best approach if you want to create high-quality software using AI tools.
There are some people who hate AI. Indeed, many developers should hate AI, because it is an enemy to their own learning of software development. This post is not for them. This post is for the few expert developers whose skills have reached the point where they outclass any and all “frontier AI models” in their area of expertise. It is for these expert developers, who want to use AI as a method of increasing their performance without sacrificing any quality that I write this post.
这篇文章是我一年来对如何正确使用AI代理编写高质量、安全关键型软件研究的结晶。
我主要从软件开发者、协议开发者以及安全关键型软件维护者的视角来撰写本文。
过去一年里，我深入探索了AI代理的极限，研究了它们可以信赖做什么、又不能指望做什么。我创建了自己的AI审查工具，其表现不亚于价值数十亿美元的AI审查系统。我维护着一个名为Crush的AI编码代理的自定义分支。这篇文章是我所学到的、以AI工具打造高质量软件的最佳方法的提炼。
有些人厌恶AI。事实上，许多开发者确实应该厌恶AI，因为它会阻碍他们自身对软件开发的学习。本文不是为他们写的。本文是为少数专家级开发者而写的——他们的技能已臻化境，在自己的专业领域内超越了所有“前沿AI模型”。我写下这篇文章，正是为了这些希望利用AI提高效率又不牺牲任何质量的专家级开发者。
Problems With Current Approaches
If you’ve used AI agents much, you know that during the course of a session the following can happen:
You can discover that your initial idea was dumb and a better one exists
Your agent might go “off the rails” and start doing something you don’t want it to do
I’ve watched videos with hundreds of thousands of views where YouTubers explain how they invented complicated systems of 12 parallel agents managed by an orchestrator, doing a billion things simultaneously. How they no longer have to involve themselves in the coding process. It’s just slop writing and reviewing slop while the YouTuber sits on a beach, goes to the bathroom, or sips coffee for no reason.
It is humanly impossible to build your own understanding of a codebase if you use such a “Vibe” approach. The AI will have gone off the rails multiple times and you will only notice it later when you actually try to use the software. This method may be perfectly OK in situations where you do not care about quality, but if you do care, a different approach is needed.
The problem is that even code written and/or reviewed by Fable 5, will stink:
The code works, but it is horribly inefficient and ugly. And this will definitely happen more often if you are working in some kind of a niche area that doesn’t have much training data for the model to fall back on. Contrary to marketing statements made by certain CEOs, these models are not able to think beyond their training data.
当前方法的问题
如果你大量使用过AI代理，你会知道在会话过程中可能发生以下情况：
你会发现自己最初的想法很蠢，存在更好的方案
你的代理可能“脱轨”，开始做你不希望它做的事
我看过一些观看量达数十万的视频，YouTuber们解释他们如何发明了由编排器管理的12个并行代理的复杂系统，同时处理海量任务。他们声称自己不再需要参与编码过程。结果就是一堆垃圾代码在写垃圾、审垃圾，而YouTuber坐在海滩上、上厕所或毫无理由地啜饮咖啡。
如果采用这种“氛围式”方法，人类根本无法构建自己对代码库的理解。AI会多次脱轨，而你只有在真正尝试使用软件时才会发现。这种方法在你不在乎质量的情况下可能完全没问题，但如果你在乎质量，就需要不同的方法。
问题是，即使是由Fable 5编写和/或审查的代码，也会很糟糕：
代码能运行，但效率极其低下且丑陋。如果你在某个小众领域工作，模型缺乏足够训练数据可以依赖，这种情况会更常发生。与某些CEO的市场宣传相反，这些模型无法超越它们的训练数据进行思考。
AI Code Generation — The “Short Leash” Method
That brings us to the “short leash method” for using AI coding agents.
This method cannot be employed by just anyone. Only professional software developers can use this method. But what’s great about it is that it will lead to Fable-beating results even if you aren’t using a frontier model.
In the Short Leash method:
You use a planning phase to research the task and formulate a plan, along with something like my tasks skill to track progress and break large tasks into steps (this is one point of commonality with many “vibe engineering” methods; the approach diverges in the following bullet points.)
You never use “YOLO” mode (aka “dangerously skip permissions”)
The AI never works “while you play video games”
You use a coding agent that displays a diff of the changes that are about to be made via the permissions prompt
You sit there like some crazed person from the 20th century, and actually analyze the changes the AI is proposing to make
You keep yourself in the loop at all times instead of removing yourself (the trend promoted by YouTubers)
You use the diffs in the permissions prompts as a way to keep your understanding of the codebase up-to-date and the AI on a “short leash”
You DENY permissions any time you see that the AI is about to do something you don’t want it to do
You intervene frequently and as needed to prevent the AI from “going off the rails”
At all times, the AI is “kept on a short leash”
Commits are made at the end of every subtask to protect you from the AI screwing up and deleting previously done work (this can happen, I’ve seen Opus do it)
Finally, we do a review
AI代码生成 —— “短链”方法
这引出了使用AI编码代理的“短链方法”。
这种方法并非人人都能使用。只有专业软件开发者才能运用。但它的好处是，即使你使用的不是前沿模型，也能取得超越Fable的结果。
在短链方法中：
你使用规划阶段来研究任务并制定计划，配合类似“my tasks skill”的功能来跟踪进度并将大任务拆解为步骤（这是与许多“氛围工程”方法的一个共同点；该方法在以下要点中分道扬镳。）
你从不使用“YOLO”模式（又称“危险地跳过权限”）
AI从不“在你打游戏时”工作
你使用的编码代理会通过权限提示显示即将进行的变更差异
你像个20世纪的疯狂人士一样坐在那里，实际分析AI提议的更改
你始终让自己保持参与，而不是把自己排除在外（这是YouTuber们推崇的趋势）
你利用权限提示中的差异来保持自己对代码库的更新理解，并将AI“拴在短链上”
每当看到AI要做你不希望它做的事时，你就拒绝权限
你根据需要频繁干预，防止AI“脱轨”
在任何时候，AI都被“拴在短链上”
在每个子任务结束时进行提交，以保护你免受AI搞砸并删除已完成工作的影响（这确实会发生，我曾见过Opus干过这事）
最后，我们进行审查
How to do AI Reviews
A PR reviewed by just a human or just an AI will have more mistakes in it than a PR that’s reviewed by both a human and an AI.
The AI can be treated as a linter. It will quickly catch common mistakes, while the human will catch higher-level issues and directional changes that need to be made.
So when it comes to reviews:
You should be using AI to review every single PR.
The AI must have access to sufficient context (the issue, the PR description, the codebase, and the changes).
You should use the latest and greatest models available to review.
The PR description must disclose the precise models used (if any) in assisting with the creation of the PR under an “AI Disclosure” heading. This serves multiple purposes:
It informs the maintainer that AI was used.
It lets them suggest better models if weak ones were used.
It signals that you’re a “good guy” developer and aren’t trying to “sneak AI in”.
Finally, and most importantly, the PR must be reviewed by the PR ‘author’ if it used AI.
AI-assisted PRs are really PRs from an AI with human assistance. Therefore, the human submitting the PR is expected to understand what they are submitting, and they cannot do that if they haven’t reviewed the code the AI wrote.
So they must treat their own PR as if they’re reviewing someone else’s PR, and review it themselves, line-by-line. Once finished, they can confirm their own approval of the PR, and request attention from the maintainer. This builds and demonstrates their understanding of the codebase.
如何做AI审查
仅由人类或仅由AI审查的PR，其错误数量会比由人类和AI共同审查的PR更多。
AI可以被当作一个linter来对待。它能快速捕捉常见错误，而人类则能发现更高级别的问题和需要做出的方向性调整。
因此，在审查方面：
你应该使用AI来审查每一个PR。
AI必须拥有足够的上下文信息（问题、PR描述、代码库以及变更内容）。
你应该使用市面上最新最好的模型来进行审查。
PR描述必须在“AI披露”标题下披露协助创建PR所使用的确切模型（如果有的话）。这有几个目的：
它告知维护者使用了AI。
如果使用了弱模型，它能让维护者建议更好的模型。
它表明你是一个“好”开发者，而不是试图“偷偷塞入AI”。
最后，也是最重要的，如果PR使用了AI，那么PR的“作者”必须审查它。
AI辅助的PR本质上是AI生成的PR，附带人类辅助。因此，提交PR的人类应该理解他们提交的内容，而如果他们不审查AI写的代码，就无法做到这一点。
所以，他们必须像审查他人的PR那样对待自己的PR，逐行地自我审查。完成后，他们可以确认自己批准该PR，并请求维护者关注。这建立并展示了他们对代码库的理解。
Fin
And that’s how we use AI at okTurtles. You can read our official AI Usage Policy.
We hope this post has been helpful.
AI Disclosure: this post was entirely written by human fingers connected to a human brain. A final AI-style “spell check” was performed before publishing.
Donating = Loving!
Without our supporters, we can't do what we do.
Please take this moment to support our work.
总结
这就是我们在okTurtles使用AI的方式。你可以阅读我们的官方AI使用政策。
希望本文对你有所帮助。
AI披露：本文完全由人类的手指连接人类的大脑所撰写。发布前进行了最后的AI式“拼写检查”。
捐赠就是爱！
没有我们的支持者，我们无法实现所做的一切。
请花点时间支持我们的工作。