VI EN
Reading

Your AI Product Needs Evals: Hamel Husain's three-level framework

Binh 3 min read Đọc bài này bằng tiếng Việt

TL;DR

Hamel Husain argues most AI products fail for lack of an evaluation system. His three levels: cheap assertions on every change, human evaluation and a calibrated LLM judge, then A/B tests; above all, read real traces regularly.

Many AI products get stuck in the same place: an impressive demo, but every prompt fix breaks something else, and the team can't tell whether it's moving forward or backward. Hamel Husain, who has advised many companies building LLM products, wrote "Your AI Product Needs Evals" with a blunt thesis: the root cause of most failed AI products is the lack of an evaluation system. This post summarises his three-level eval framework and its practical lessons.

The story: Lucy, a real-estate assistant

The post uses the example of Lucy, an AI assistant for the real-estate software Rechat. Early progress came fast through prompt engineering, then stalled: fixing one feature broke another, and nobody had a grip on overall quality. Building evals systematically is what unlocked progress again.

Three levels of evals

LevelWhat it isWhen it runs
1. Unit tests (assertions)Cheap, fast automated checks: correct format, no internal IDs leaked, required fields present…On every code or prompt change
2. Human and model evaluationLog traces, look at real data, use an LLM judge calibrated against human gradersRegularly, on a development cadence
3. A/B testingMeasure real impact on user behaviourOnce the product is mature enough

The most memorable lessons

  • Look at your data: reading real traces is the most valuable activity many teams skip. Real failures are often very different from the ones you imagine.
  • Remove friction from looking at data: build a genuinely easy trace viewer; if viewing data is hard, nobody will do it.
  • Domain-specific unit tests: write assertions from failures you've seen, like regression tests.
  • Calibrate the LLM judge: measure agreement between the LLM judge and human graders, and iterate until it's trustworthy.
  • Evals unlock everything else: once you have them, debugging, comparing models and even fine-tuning become feasible.

Where to start

  1. Log every interaction fully (input, output, tool calls).
  2. Read 50–100 traces and note the failure types.
  3. Write assertions for recurring failures and put them in CI.
  4. For failures code can't check, add an LLM judge calibrated on a few dozen human-graded samples.

Risks and caveats

  • Generic benchmarks: scores on public test sets say little about quality on your task.
  • Buying tools before understanding failures: eval tooling doesn't replace reading the data yourself.
  • "Dead" evals: a test set that isn't updated with new failures gives a false sense of safety.

How to apply it

  • Dev: For every AI bug you find, write an assertion before fixing the prompt.
  • Tech Lead: Set aside fixed weekly time for the whole team to read traces; build a simple internal trace viewer.
  • PM: Define "good" with concrete examples; they're the raw material for both assertions and the LLM judge.

Sources & references

  1. Your AI Product Needs Evals — Hamel Husain
More on Reading