TOPIC AI
Evals testing something non-deterministic
IN 10 SECONDS
Evaluating LLM outputs by comparing model responses against test benchmarks (accuracy, toxicity, safety) using deterministic assertions or evaluator models.
GOTCHA Relying on simple exact-string assertions fails because LLMs express the same correct answer in many different valid phrasing combinations.
HOW AN EVAL RUNS
01 Prompt test set defines 100 benchmark prompts alongside target ground-truth answers.
02 Model execution runs prompts through the target candidate LLM to collect outputs.
03 Scoring evaluator model or script scores answers: semantic similarity, keyword checking, etc.
04 Performance report outputs accuracy metrics: 94% passed, 6% failed.
POKE IT YOURSELF
promptfoo eval — run a prompt engineering evaluation matrix test locally
Drill this topic →
~65 sec read