Research · 1 min read
Write the evaluation before the prompt
The evaluation suite is the spec. We write it first, and it's the single biggest reason our AI features survive contact with users.
Radicand Labs, Research
When we start an AI feature, the first thing we build isn’t a prompt. It’s a set of examples with the answers we’d accept.
Examples are the spec
A requirement like “find where the code disagrees with the ticket” is impossible to test. Fifty real tickets and commits, with the disagreements a senior engineer has signed off on, is a spec.
Grade what matters
We grade outputs on the properties our users care about: did it find the missing requirement, did it avoid inventing one, did it point to the right file. Some checks are code, some use a model as a judge, and a few always go to a person.
Run it on every change
The suite runs in CI on every prompt change, model change and retrieval change. It’s the only way to know whether “this feels better” is true, and the only way to switch models without fear.