Opens on reph with the template ready to start. No account needed; sign in to keep your document.
Use it when
You are about to put a change in front of a slice of users and want the decision rule agreed before anyone sees results
Who it's for
Product managerResearcherEngineer
Part of a kit
Product launch, step 3 of 6. From brief and PRD through experiment, launch checklist, release notes and the customer newsletter.
Shipping AI features, step 4 of 4. Prompt spec, eval plan, agent instructions and the experiment that proves it works.
Filled example
The template filled in for a made-up company, so you can see what good looks like.
Smart reply suggestions in the inbox
Author: Priya Raman · Date: 2026-09-14
Owner
Priya Raman
Start date
2026-09-21
Planned end date
2026-10-05
Status
Approved
Hypothesis
"We believe showing three AI-drafted reply suggestions for Juniper Helpdesk agents on Pro plans will result in faster first responses, measured by the share of tickets answered within one hour."
Variants
Variant
Description
Allocation
Control
Current composer, no suggestions
50%
Treatment A
Three suggestion chips above the composer, one click to insert and edit
50%
Primary metric
Share of new tickets that get a first agent reply within 60 minutes.
Replied within 1h (baseline)
41%
▲ +0%
Minimum detectable effect
+3 pts
Guardrail metrics
Reopen rate
8.2%
▲ 0%
CSAT
4.5
▲ 0%
Guardrail
Baseline
Stop threshold
Ticket reopen rate
8.2%
Above 9.5% for two days running
CSAT on resolved tickets
4.5 / 5
Below 4.3
AI cost per agent per day
$0.18
Above $0.40
Audience & allocation
Agents on Pro workspaces with English as the ticket language. Bucketing is by workspace, not by agent, so teammates never see different composers. Enterprise workspaces are excluded until legal signs off on the data processing addendum.
Sample size & duration
Powered at 80% to detect a 3 point lift on a 41% baseline at p<0.05.
Users per variant
4,300 tickets
Daily eligible traffic
about 650 new tickets
Runtime
14 days, covering two full weekly cycles
Decision rule
Ship to all Pro workspaces if the primary metric improves by at least 3 points with p<0.05 and no guardrail crosses its threshold. If the lift is positive but under 3 points, iterate on suggestion quality and rerun. If any guardrail breaches, stop and kill.
Results
Metric
Control
Treatment A
Change
Significant?
Replied within 1h
41.3%
46.0%
+4.7 pts
Yes (p=0.008)
Reopen rate
8.1%
8.4%
+0.3 pts
No
CSAT
4.52
4.55
+0.03
No
AI cost per agent per day
n/a
$0.21
n/a
Under threshold
Decision
Ship. Roll out to all Pro workspaces on 2026-10-12 behind the existing flag, then start a follow-up test on Enterprise once the addendum is signed. Daniel Okafor owns the rollout; Priya reviews reopen rate weekly for a month.
The template
Show the blank template
Experiment (A/B) plan
Author: [your name] · Date: [date]
Owner
[your name]
Start date
Planned end date
Status
Draft
Hypothesis
"We believe [change] for [audience] will result in [outcome], measured by [metric]."
Variants
Variant
Description
Allocation
Control
Treatment A
Primary metric
The one metric that decides win/lose. Pick one. A bag of metrics makes every result look like a win somewhere.
Primary metric (baseline)
0%
▲ +0%
Minimum detectable effect
+0%
Guardrail metrics
Metrics that must not regress even if the primary metric improves, each with the threshold that stops the test early.
Guardrail 1
0
▲ 0%
Guardrail 2
0
▲ 0%
Guardrail
Baseline
Stop threshold
Audience & allocation
Who's eligible, how they're bucketed, and what % of traffic is in the test.
Sample size & duration
The minimum detectable effect you're powered for, and the runtime that gives you it. Commit to this before peeking at results.
Users per variant
Daily eligible traffic
Runtime
Decision rule
The rule you'll follow regardless of how the numbers look mid-flight, e.g. "ship if primary metric +2% with p<0.05 and no guardrail regression."
Results
Filled in once the test reaches its planned end date, not before.
Metric
Control
Treatment A
Change
Significant?
Decision
Ship / iterate / kill, and what happens next.
Questions
Can I have two primary metrics?
No. Pick one that decides the result and make the rest guardrails
When can I look at results?
Only at the planned end date, unless a guardrail breaches its stop threshold
What if the result is flat?
Follow the decision rule you wrote before launch, which should say what flat means
Fill it with your agent
Connect Claude to reph over MCP, then paste this. The draft arrives as suggestions you accept or reject section by section. How to connect Claude.
Find my reph doc "Experiment (A/B) plan" with search_docs, then read it with read_doc. Draft hypothesis, variants, primary and guardrail metrics from the linked PRD or feature brief and last quarter's dashboard baselines, and estimate sample size and runtime from current daily traffic. File it with suggest_edit, one replace_section op per section (heading line kept) so each can be accepted alone. Ask each open question (the minimum detectable effect, metrics with no baseline) with add_comment quoting that section's heading line, then call request_review with a one-line summary.