08-09
Your App Is Only As Good As Its Evals
In LLM apps, inputs no longer map to deterministic outputs—any small change can remake the entire behavior. This post argues that evals are the AI engineer's unit tests, the primary way to impose predictability on a probabilistic system. It walks through three evaluation types: deterministic assertions, human review, and LLM-as-a-judge, noting the cost trade-offs and recommending split cadences (fast local set vs. daily full set). It then explains the data flywheel: converting user downvotes into new eval cases and feeding them back into the system. Tooling compares Braintrust, a cloud eval platform, with Evalite, the author's Vitest-based local runner. Useful for engineers moving LLM prototypes to production.