AI Evaluation & Hallucination Reduction
"It seemed to work when I tried it" is not a testing strategy — this chapter is about what actually is.
6.8.1Definition
Evaluation is the systematic measurement of an AI system's output quality against a defined set of test cases and criteria, as distinct from informal spot-checking. A hallucination is a confident, plausible-sounding output that is factually wrong or unsupported by the given context — one of the primary failure modes evaluation exists to catch before it reaches production.
6.8.2Why It Exists
Language models produce fluent, confident-sounding text regardless of whether the underlying content is accurate, which means quality problems are easy to miss on casual inspection. Evaluation exists because "it looked right in my test" is not a reliable signal at scale — a proper evaluation set, run consistently, is the only way to know whether a prompt or system change actually improved output rather than just looking better on the examples someone happened to try.
6.8.3Core Evaluation Approaches
| Approach | How it works | Best fit |
|---|---|---|
| Golden test set | Fixed set of inputs with known-correct expected outputs | Tasks with a clear right answer (classification, extraction) |
| Model-graded evaluation | A second model scores output against a rubric | Open-ended tasks with no single correct answer |
| Human review | People manually assess a sample of outputs | High-stakes or subjective quality checks; validating automated evals |
6.8.4Common Mistakes
- No fixed evaluation set at all, relying entirely on ad hoc manual testing that can't be run consistently as the system changes.
- Evaluating only on easy, typical cases, missing the edge cases most likely to trigger hallucination in production.
- Treating a passing eval score as proof of correctness rather than a directional signal, when the eval set itself may not cover the failure mode that later appears.
- No RAG grounding check (6.3) — never verifying whether the model's claims are actually traceable to retrieved source material.
6.8.5Best Practices
- Build a fixed, versioned evaluation set early, including deliberately difficult and edge-case inputs.
- Re-run the same evaluation set after every meaningful prompt or system change, tracking results over time.
- For RAG systems, explicitly check whether generated claims are grounded in retrieved content, not just plausible.