Code Factories Without Quality: The AI Development Blind Spot

Zapier. Nubank. Goldman Sachs. Three companies with great engineering and risk management teams – and all three are now handing coding tasks to AI agents with minimal human intervention.

The workflow of code factories is deceptively simple: a developer hands off a task and the agent plans, codes, reviews, tests, debugs, and ships it end-to-end.

Zapier deployed 800+ AI agents internally, with 89% AI adoption across the entire organization. Nubank pointed autonomous AI agents at one of the most critical projects in the company, migrating its core ETL. Goldman Sachs, meanwhile, is piloting autonomous software engineers on its own codebase.

The appeal of code factories is obvious: velocity. Code generation at scale. 10x speed or more. Entire projects shipped without a human hand touching the keyboard.

But every code factory is quietly skipping or underinvesting in the same step: testing.

Agents are generating production code faster than anyone can verify it. Generation capacity increased 10x. Verification capacity didn’t. So, the code ships untested or undertested making code factories not just about velocity, but about velocity at all costs, quality be damned.

And when it breaks in production (it will), there's no feedback loop to make the next generation any better. The whole system is designed to ship more slop, faster.

Every code factory is built on an assumption that nobody says out loud: generated code is production-ready code.

It's not. But the assumption persists because QA is rarely prominently in the equation.

Instead, QA is being treated as a downstream checkbox or something you can solve with minimal coverage. In a code factory, where velocity is the core value, QA is the friction—the thing preventing you from shipping even faster.

The reasons are complex but because it’s seen as slow, it gets deprioritized or automated away with shallow test coverage that passes but doesn't truly verify much.

And teams often know this but still ship code they know is likely to break because they feel they have no other choice. Fast slop is now seen as better than slightly slower quality.

The real problem is that verification velocity hasn't yet matched generation velocity. AI code review tools are helping with one side of verification. They're catching some bugs, suggesting improvements, flagging obvious issues. But they're still letting bugs through.

What's needed now is an increase in testing velocity.

A code factory might say they ship with 80% coverage and call that success. But coverage like that typically measures lines executed, not whether user flows actually work. You can have 100% line coverage and still ship code that fails under load, handles errors wrong, or leaves data in an inconsistent state.

The code factory speeds up the rate at which bad code hits production.

And the results are brutal. Incident rates are climbing as code generation scales. Teams shipping AI-generated code are seeing a 30% increase in change failure rates and a 23.5% increase in incidents per pull request.

More code shipped faster without proper verification means more bugs in production, more customer-facing failures, more firefighting.

A bug caught at the PR stage cost minutes to fix. A bug caught in staging costs hours. A bug caught in production can cost days, angry customers, incident responses, and cascading failures across dependent systems. The cost of fixes post-deployment dwarfs the cost of catching them before they ship.

The real problem is that code factories are optimized for generation speed, not verification speed. QA is positioned as sometimes a blocker to velocity, when it should be seen as the enabler of it. Because if you have good automated test coverage in place, code factories become something you can actually trust to ship fast.

The problem is that you can't hire your way to 10x verification since human test creation is linear, whereas agent generation isn't. You also can't let agents grade their own homework because then tests that inherit the code's blind spots just automate your false confidence.

The tests that actually catch these failures – end-to-end coverage of real user flows – are historically the slowest to build and most painful to maintain. That's the trap: the verification your code factory actually needs is the exact verification that's hardest to build, slowest to run, and most expensive to keep alive, at the precise moment your generation velocity makes all three problems worse.

Verification has to become an autonomous pipeline, too. It needs to have the same velocity and run in parallel, not bolted on downstream as the thing that slows everyone down.

But to do that well requires a few non-negotiables.

Ability to scale

It has to scale like generation, not like headcount. Test creation can't depend on how many QA engineers you can hire. It has to expand at the rate code ships because that's the only rate that keeps pace.

Tests real user flows

It has to test the way users actually use the product. End-to-end coverage across real flows, not line counts. You need to make sure the customer flows work properly, not just that the code executes.

Independent from the coding agent

It has to be independent from the coding agent that generated it. Verification built separately from the generation agent doesn't inherit the code's blind spots. It's an actual second opinion instead of an echo. But more than just that, a coding agent isn’t built to understand how to create good tests. Don’t trust us though. Trust the dozens of threads on Reddit of people struggling to automate test coverage using a coding agent in the last few months.

Self-maintaining

It has to maintain itself. When a flow changes, the tests update automatically, so flake stays near zero and the team keeps trusting the signal. Self-maintaining coverage is the only kind that survives contact with 10x velocity.

Put those together and QA stops being the friction in the code factory. It becomes the thing that lets you ship fast without shipping slop.

Verification has to finally move at the speed of generation.

This is the part of the article where we do a bit of a plug but bear with us because it’s actually a solution to the problems most teams are facing.

Our platform creates an autonomous QA pipeline that runs at the speed of generation, independent from the code it's checking. It maps your app, writes the tests, runs them on every deploy, and keeps them alive as the code churns. It’s an end-to-end platform for the entire testing lifecycle.

Here's how it works:

It maps your app autonomously

The Mapping Agent independently navigates your application and builds a detailed map of its features and real user workflows, instead of the coding agent’s assumptions about it. It switches user roles and toggles between web, iOS, and Android to catch flows that span multiple users and multiple platforms—the exact multi-step, cross-state behavior where AI-generated code quietly breaks. The agent is 32x faster than a human doing it alone.

It writes real, deterministic tests

The Automation Agent turns those mapped workflows into deterministic Playwright (web) and Appium (mobile) code. These are tests that call APIs, seed databases, mock external dependencies, and flip feature flags to exercise the app under real conditions. Because the tests are deterministic code rather than an LLM re-interpreting a script each run, a failure means something actually broke, and the failure is reproducible. And because the suite is generated independently of whatever produced your production code, it's a genuine second opinion instead of an echo. The Automation Agent increases a QA engineer's capacity 10-20x, which is how test creation finally scales at the rate code ships instead of at the rate you can hire.

It runs on every deploy, in full parallel

Coverage that takes hours to run doesn't help a code factory. QA Wolf runs the full suite with 100% parallel execution, kicked off instantly on deploy. It can be wired directly into your CI pipeline—so verification keeps pace with a pipeline that's merging constantly.

It maintains itself so the suite survives 10x velocity

This is the piece that kills most testing efforts. Autonomous agents don't generate code once; they refactor, optimize, and rewrite it, and every one of those changes threatens to break the suite. QA Wolf's Automation AI addresses close to 100% of flakes including timing issues, runtime errors, un-rendered components, versus the roughly 20% that selector-repair tools catch. When a test does break, it reproduces the failure, diagnoses the cause, rewrites the code, and validates the fix. Meanwhile, the Mapping Agent watches for new features and folds them into existing flows rather than dumping in isolated one-off tests or duplicating tests you already have.

Put the QA Wolf lifecycle together and you get the thing the code factory was missing: verification that maps, writes, runs, and maintains itself at generation speed, so your developers keep shipping and QA stops being the bottleneck.

That's how you ship fast without shipping slop.

Building factories that actually work

QA can’t be a bottleneck in code factories. It has to be foundational. Code factories that work treat testing as a first-class layer in the pipeline and focus on matching testing velocity to generation velocity.

Real-time verification catches failures before production. Code generated by autonomous agents gets tested immediately, comprehensively, and continuously. Bad code never ships. The factory has gates and those gates work.

Autonomous code generation only works at scale if you can verify at scale. QA Wolf's automation means tests scale with generation. You can ship 10x the code without 10x the incident rate.