07-11
Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island
Dan Luu shares his extensive experience with AI coding agents over the past year, focusing on testing, benchmarking, and agentic loops. He compares fuzzing vs. LLM-driven bug finding, finding fuzzing faster with lower false-positives; evaluates 'caveman mode' with 50 runs showing inconsistent savings; highlights high variance in LLM benchmarks, making public evals nearly useless for individual users. He also discusses automated PR generation from support tickets, multi-persona false-positive reduction, and challenges in data analysis and autonomous loops. For engineers interested in real-world effectiveness of AI coding tools.