Email experiment
Experiment: onboarding email sequence test
Onboarding email tests are high-leverage because activation is the leading indicator of retention. The framework below isolates the email sequence as the variable, names activation as the primary metric, and provides the sample-size honesty most founders skip.
Min sample size: 300+ trial signups per variant for a 25%+ lift detection on activation.
Duration: 30-60 days from test start. Activation rates at day 7 require waiting 7 days per cohort plus measurement window.
Verified · editorial policy
Hypothesis structure
Changing the onboarding email sequence from [CURRENT] to [VARIANT] will increase activation rate at day 7 by at least [EXPECTED LIFT] because [SPECIFIC REASON].
If you cannot complete this template, you do not have an experiment — you have a guess.
Variant design
Only the email sequence changes. Number of emails, timing, content, or all three — but document exactly what changes. Same in-product UI, same nudges, same dashboard.
Primary metric
Activation rate at day 7 (or whatever day matches your product's natural activation point). Activation = the specific action that correlates with paid conversion.
Secondary metrics (watch but do not decide on)
- Email open rate (proxy for engagement).
- Trial-to-paid conversion (downstream effect).
- Time-to-first-value (does the new sequence accelerate?).
Procedure
Step 1
Define activation specifically
What specific in-product action correlates with paid conversion? '3+ feature uses in week 1', 'first integration connected', 'first invitation sent'. This is the leading indicator.
Step 2
Draft the variant sequence with a specific hypothesis
What is different in the variant — fewer emails, more emails, different content, different timing? Document the hypothesis behind each change.
Step 3
Split new signups at the queue level
Server-side split keyed to user ID. Email-service-side splits (Mailchimp, Customer.io) work too if they reliably persist the assignment.
Step 4
Measure activation at day 7
Cohort by signup date, not by experiment date. Day-7 activation for the variant cohort.
Step 5
Watch downstream conversion at day 60
Activation that does not produce paid conversion is misleading. Check that the variant's activation lift actually flows through.
Self-deceptions to avoid
- Using open rates as the success metric. Opens are vanity; activation is the leading indicator.
- Stopping at day 7 conversion. Day-60 paid conversion is the real downstream test.
- Testing multiple email changes at once. Hard to attribute later.
What success looks like
Variant produces 25%+ higher activation at day 7 AND maintains or improves day-60 paid conversion.
Related benchmark
See the directional range for email open rate to calibrate the expected lift in your hypothesis.
Frequently asked
- Should I test number-of-emails or content-of-emails first?
- Content first. Most under-converting sequences have content problems, not volume problems. Test volume after content is right.
Other experiments
Test on a page that is already pointed in the right direction
A/B tests on a misaligned page produce two losing variants. The diagnostic labels the alignment problem first; the test optimizes within the right alignment.