How big does the eval set need to be?

Start with 50 to 200 hand-checked cases covering every slice; grow it with every production failure you find rather than chasing size up front

Can an LLM grade the outputs?

Yes for scale, but calibrate the grader against human scores on a sample first and re-check that agreement whenever the grader model changes

When do the regression gates run?

On every prompt, model, retrieval or parameter change, before it ships, the same way unit tests run on code

How you will measure an AI feature before and after launch: the task, a quality rubric, the eval dataset, metrics, baseline vs target, human review, regression gates and a cost and latency budget

Opens on reph with the template ready to start. No account needed; sign in to keep your document.

Filled example

The template filled in for a made-up company, so you can see what good looks like.

Eval plan: Ticket summary for support agents

Author: Rosa Lindqvist · Date: 2026-09-18 · Feature: Helpdesk ticket summaries · Prompt spec: ticket-summary v3

Fernhill Helpdesk will show a three-line summary at the top of every support ticket longer than 10 messages. Good means an agent can act without scrolling, and the summary never states something the customer did not say. This plan gates the move from Model A to the cheaper Model B.

The task

Input: the full ticket thread (customer and agent messages, internal notes excluded), up to 40k tokens.
Output: three bullets: the customer's problem, what has been tried, and the open ask.
Consumer: a support agent, reading in the ticket sidebar.

Quality criteria

Criterion	Pass looks like	Fail looks like	Weight
Correctness	Problem and ask match the thread	Wrong product or wrong ask	40%
Grounding	Every claim traceable to a message	Invented order numbers or promises	30%
Format	Exactly three bullets, under 60 words	Paragraphs or extra bullets	15%
Tone and safety	Neutral, no customer PII beyond first name	Repeats card or address details	15%

Eval dataset

Slice	Source	Cases	Why it matters
Typical	Random closed tickets, Q2	120	The bulk of traffic
Long or messy input	Tickets over 30 messages	40	Where summaries get lost
Edge cases	Multi-language, forwarded email chains	30	Known weak spots
Adversarial	Customer text containing instructions	20	Prompt injection

Dataset location and version: evals/ticket-summary/v4 (210 cases, PII scrubbed).

Metrics

Rubric pass rate

91%

▼ -2%

Hallucination rate

1.9%

▼ -0.5%

p95 latency

2.1s

▼ -38%

Cost per 1k requests

$1.40

▼ -61%

Baseline vs target

Metric	Baseline (Model A)	Target	Current (Model B)
Rubric pass rate	93%	at least 90%	91%
Hallucination rate	2.4%	at most 2%	1.9%
p95 latency	3.4s	under 2.5s	2.1s
Cost per 1k requests	$3.60	under $2.00	$1.40

Pass rate by slice

baseline
current

View data

slice	baseline	current
Typical	95	94
Long input	90	86
Edge cases	87	88
Adversarial	95	95

Human review protocol

Reviewers: two senior support agents, blind to which model produced each summary.
Sample: 50 random cases plus every case the grader scored between 0.4 and 0.6.
Calibration: human and model grader agree on 89% of cases; re-checked whenever the grader model changes.

Regression gates

Mermaid diagram (drawn as a picture in reph)

flowchart LR
  A[Prompt or model change] --> B[Run eval suite]
  B --> C{Gates pass?}
  C -- yes --> D[Human spot check]
  D --> E[Ship behind flag]
  C -- no --> F[Fix or revert]
  F --> B

Rubric pass rate does not drop more than 3 points on any slice (long input dropped 4)
Hallucination rate stays at or below 2%
p95 latency and cost stay inside budget
Every previously fixed production failure still passes (2 of 14 regressed: both long forwarded chains)

Cost and latency budget

Budget	Limit	Current	Notes
p95 latency	2.5s	2.1s	Measured at peak hour
Cost per request	$0.002	$0.0014
Monthly spend at forecast volume	$900	$620	440k tickets per month

Open questions

Fix the two regressed forwarded-chain cases before ship, or ship and track?
Should the long-input slice gate be relaxed to 5 points, given it is 8% of traffic?

The template

AI feature eval plan

Author: [your name] · Date: [date] · Feature: [link] · Prompt spec: [link]

One paragraph: what the feature does, what "good" means for its users, and the decision this eval plan supports (launch, model swap, prompt change).

The task

Inputs the model receives, the output it must produce, and who consumes it. One concrete input and ideal output beats a paragraph of description.

Input:
Output:
Consumer: a person, another system, or both

Quality criteria

The rubric graders (human or model) score against. Each criterion is checkable and has a clear fail case.

Criterion	Pass looks like	Fail looks like
Correctness
Grounding	Every claim traceable to the input	Invented facts
Format	Matches the output contract
Tone and safety

Eval dataset

Where cases come from, how many, and the slices that matter. Real inputs beat synthetic ones; strip personal data.

Slice	Source	Cases	Why it matters
Typical		0
Long or messy input		0
Edge cases		0
Adversarial		0

Dataset location and version:

Metrics

Few, defined, and tied to the rubric. Update after each eval run.

Rubric pass rate

▲ +0%

Hallucination rate

▲ 0%

p95 latency

Cost per 1k requests

Baseline vs target

Baseline is today's system (or a simple non-AI fallback). Targets are the bar for shipping.

Metric	Baseline	Target	Current
Rubric pass rate
Hallucination rate
p95 latency
Cost per 1k requests

Pass rate by slice

baseline
current

View data

slice	baseline	current
Typical	0	0
Long input	0	0
Edge cases	0	0
Adversarial	0	0

Human review protocol

Who reviews, how many cases, blind or not, and how disagreements are resolved.

Reviewers:
Sample: e.g. 50 random cases plus every case the grader marks borderline
Calibration: agreement between human and model grader, re-checked when the grader changes

Regression gates

Checks that must pass before any prompt, model, retrieval or parameter change ships.

Mermaid diagram (drawn as a picture in reph)

flowchart LR
  A[Prompt or model change] --> B[Run eval suite]
  B --> C{Gates pass?}
  C -- yes --> D[Human spot check]
  D --> E[Ship behind flag]
  C -- no --> F[Fix or revert]
  F --> B

Rubric pass rate does not drop more than [x] points on any slice
Hallucination rate stays at or below [x]%
p95 latency and cost stay inside budget
Every previously fixed production failure still passes

Cost and latency budget

Budget	Limit	Current	Notes
p95 latency
Cost per request
Monthly spend at forecast volume

Open questions

Questions

Fill it with your agent

Connect Claude to reph over MCP, then paste this. The draft arrives as suggestions you accept or reject section by section. How to connect Claude.

Find my reph doc "AI feature eval plan" with search_docs, then read it with read_doc. Draft the task definition and rubric from the linked prompt spec and PRD, propose dataset slices from real user inputs and support tickets (no personal data), and fill baselines from the latest eval run results. File it with suggest_edit, one replace_section op per section (heading line kept) so each can be accepted alone. Ask each open question (targets and gate thresholds you guessed) with add_comment quoting that section's heading line, then call request_review with a one-line summary.

AI feature eval plan template

Use it when

Who it's for

Part of a kit

Filled example

Eval plan: Ticket summary for support agents

The task

Quality criteria

Eval dataset

Metrics

Baseline vs target

Human review protocol

Regression gates

Cost and latency budget

Open questions

The template

AI feature eval plan

The task

Quality criteria

Eval dataset

Metrics

Baseline vs target

Human review protocol

Regression gates

Cost and latency budget

Open questions

Questions

Fill it with your agent

Related templates

Prompt spec

Agent instructions

Experiment (A/B) plan

ADR

API spec

Bug report