Free tool · no signup · runs in your browser

LLM-as-a-Judge eval builder

Pick a failure mode and get a complete evaluator prompt — single criterion, pre-specified procedure steps, bias defenses, and the research behind each choice. Built on the same frame as the generate-eval skill; nothing you enter leaves this page.

1 · What failure mode should the judge catch?

2 · Is there a ground truth to check against?

A reference is a ground truth the judge can check against — retrieved chunks for RAG, gold SQL, a reference summary.

3 · One output, or comparing two?

4 · Bias defenses

Length controlAdds a line so the judge doesn't reward verbosity (Saito 2023, Dubois 2024). Cross-family judgeScore with a different model family than the one under test (Panickssery 2024).

Evaluator prompt

Run settings

Why this config

When you run it

Research

One judge from a description is a start — production needs more

Ground it in real traces

The generate-eval skill builds this same judge from your actual production behavior — real few-shot examples pulled over MCP.

Find the judges you're missing

coverage-gaps diffs what runs in production against what your evals measure, and ranks what to build next.

Install the toolkit

Five skills for Claude Code and GitHub Copilot over the Progress Observability Platform — trace triage, cost, coverage, evals.