Benchmarks

Agentic test processes: from chip design to AI workflows

Drawing from his experience at chip company Centaur, the author compares test processes that scale well with LLM agents: no code review by default, heavy reliance on fuzzing, and a dedicated test team. He argues that while LLMs are poor at writing tests directly, directed fuzzing with LLMs can find real bugs in minutes. The article highlights the high variance of LLM outputs—benchmark rankings often flip with minor task changes—and cautions against over-reliance on aggregated metrics. Through examples like building a superhuman board game AI, he advocates systematic data-driven iteration over prompt tricks. Targeted at engineers interested in AI-assisted development, testing, and agent workflows.