How to evaluate an LLM application before you launch it
A practical method for testing LLM apps, RAG systems and agents: build a test set from real tasks, score with calibrated graders, separate retrieval from generation, and run it on every change.
Most LLM applications are launched on the strength of a few good demos. That works until a prompt tweak quietly breaks a case nobody thought to retest, or a model upgrade changes the tone, or a user asks something the demo never covered. An evaluation suite replaces impressions with evidence. This is the method we use.
1. Define what 'good' means for this system
Write down the task, the user, and the ways the system can fail that would matter. For a support assistant, a wrong refund policy is serious and an awkward sentence is not. Turn each important failure into something you can check, such as 'cites a source for every policy claim' or 'declines to give legal advice'.
2. Build a test set from real inputs
Start with 50 to 200 realistic inputs. Real user questions, support tickets and logs are better than invented ones. Include the common cases, the awkward ones, and the cases where the right behaviour is to refuse or hand off. For each, record what a good answer must contain or must avoid. Version the set like code.
3. Use several kinds of graders
- Code checks for anything deterministic: valid JSON, required fields, a citation present, a forbidden phrase absent.
- Model-based graders for qualities that need judgement, such as faithfulness to the source or whether a response answers the question.
- Human review for a sample, to catch what the automated graders miss and to calibrate them.
A model grader is a measuring instrument, so check it. Have people label a sample, compare their labels to the grader's, and fix the rubric until agreement is good enough for the decision you will make with the score.
4. Measure retrieval and generation separately
For RAG systems, a bad answer has two possible causes: the right passages never came back, or the model misused them. Score retrieval on whether the needed passage is in the results, and score generation on whether the answer is supported by what was retrieved. Without the split, teams tune the wrong half.
5. For agents, score the path as well as the outcome
Check that the agent chose the right tool, passed correct arguments, stayed within its permissions and stopped when it should. Two runs can reach the same result while one of them took a dangerous step on the way.
6. Test the refusals and the attacks
Add cases where the right answer is 'I cannot help with that' or 'I do not know'. Add prompt-injection attempts, requests for data the user should not see, and instructions hidden in retrieved documents. These tests tend to find the problems that matter most.
7. Run it on every change, and feed it from production
Wire the suite into your release process so a prompt, model or retrieval change that lowers the score is caught before users see it. After launch, sample real traffic, and turn each real failure into a new test case. The suite then grows with the product.
Where to start
If you have nothing today, build 50 test cases, write three code checks, and run them before your next release. A small suite that runs every time beats a large one that nobody runs.