A demo that works once tells me where to start testing. It doesn’t tell me whether I’d trust the agent to repeat the job tomorrow.
I’d start an evaluation set with a handful of real tasks. Include one with missing information, one with a failed tool call, and one where the user changes their mind. Write down what a satisfactory result looks like before running the agent.
For a coding task, that might mean a passing test, a scoped diff and no changes to unrelated files. For research, it might mean sources that actually support the claims. A fluent answer is insufficient in both cases.
Keep the trace too. An agent that gets the right answer after six retries and an accidental file overwrite needs different attention from one that follows a stable path. Record tool calls, errors, time and cost alongside the result.
Then run the same tasks after changing a prompt, tool or model. Small, repeatable comparisons give “this feels worse” somewhere useful to go. I’d rather have ten cases I understand than a large score I can’t explain.