An A/B test sends two versions of an email to similar groups of recipients and compares the results. The idea is simple, and the mistakes are common: testing too many things at once, declaring a winner after an hour, using a list that is too small to show a difference, or judging by a metric that no longer measures what it did. This guide explains how to design email tests that give trustworthy answers. It covers what to test, how to pick the metric, how to estimate the sample size you need, how to run the test and how to avoid the errors that produce false winners. It also covers the impact of mail privacy features on open-rate based tests and how to test things that affect deliverability without harming it. The numbers shown are illustrations; the right values depend on your baseline rates and audience.
What to test
| Element | Example variations | Best judged by |
|---|---|---|
| Subject line | Question vs statement; with or without a number | Clicks (opens are noisy) |
| Preheader | Supporting detail vs offer | Clicks |
| Sender name | Brand vs person and brand | Clicks, replies, complaints |
| Send time | Morning vs evening; weekday vs weekend | Clicks and conversions |
| Content layout | Single column vs two column; image vs text | Clicks, conversions |
| Call to action | Button text, position, colour | Click-through and conversion |
| Offer | Percentage vs fixed amount; deadline vs no deadline | Revenue per recipient |
| Personalisation | Name or product recommendations vs none | Clicks, conversions |
See subject lines, preheaders and best time to send for what to try in each.
Choose the metric first
The metric is the answer to "how will we know which is better?" and must be fixed before the test. Match it to the goal. If the goal is revenue, measure revenue per recipient or conversion rate, not clicks. If the goal is engagement, use unique click rate. If you test the sender name or subject for a transactional notification, measure the action that follows, such as logins. Open rate needs particular care: mail privacy features load images automatically for many Apple Mail users, which counts as an open whether or not a person read the message. A test of two subject lines judged by opens can still be useful for roughly equal audiences, but its absolute numbers are inflated and the signal is weaker. Prefer clicks and downstream actions. Read Apple Mail Privacy Protection and open rates and email open rate.
How big must the sample be?
Small differences need large samples. If your baseline click rate is 3 per cent and you want to detect an improvement to 3.6 per cent (a relative lift of 20 per cent) with the usual standard of 95 per cent confidence and 80 per cent power, you need on the order of 12,000 to 14,000 recipients in each group. If you can only detect large changes, say a lift from 3 to 4.5 per cent, around 3,000 to 3,500 per group is enough. The smaller your list, the larger the effect has to be before you can believe it. You can use a sample size calculator for two proportions; enter the baseline rate, the smallest lift worth detecting, the confidence level and the power. If the required sample is more than your list, either test a bolder change, pool results over several sends, or accept that the test cannot give a firm answer.
Run the test properly
- Write the hypothesis. "A subject line with the discount amount will raise clicks by 15 per cent for our newsletter audience."
- Change one variable. If the subject, the layout and the offer all differ, you will not know which made the difference. Use multivariate tests only with very large lists.
- Split at random. Let the system assign recipients to versions by chance. Do not split by alphabet, signup date or region, because these correlate with behaviour.
- Send at the same time. Unless time is the variable, both versions should go out together.
- Keep groups equal. Equal-sized groups are the simplest to analyse.
- Wait long enough. Most clicks arrive in the first day or two, but some audiences respond over several days. Decide the duration in advance, for example 24 or 48 hours, and do not stop early because one version leads.
- Read the result with a significance test. A difference of one or two per cent relative may be noise. Many platforms report a confidence value; if yours does not, use a two-proportion test.
- Record and learn. Keep a log of tests, hypotheses and results so that you do not repeat tests and so that learning accumulates.
The winner-sends-to-rest pattern
Many platforms offer to send each version to a small share of the list, for example 10 per cent each, then send the better version to the remaining 80 per cent. This is convenient for one-off promotions, and it has limits. The test groups must be big enough to show a real difference, otherwise the "winner" is chosen by chance. A short waiting period before sending to the rest may favour versions that are opened fast, not those that do best over the day. And the lesson only applies to that send. If you want learning that you can reuse, run a cleaner test on equal halves of the whole list and analyse it properly.
Common mistakes
| Mistake | Why it misleads | Better practice |
|---|---|---|
| Stopping at the first lead | Early differences often vanish | Fix duration and sample in advance |
| Testing on a tiny list | Result is mostly noise | Test bolder changes or pool several sends |
| Changing several things | Cannot attribute the effect | One variable per test |
| Judging by opens only | Privacy features inflate opens | Judge by clicks and conversions |
| Testing many variants and picking the best | Multiple comparisons raise false positives | Limit variants or adjust for it |
| Ignoring complaints and unsubscribes | A version can win clicks and hurt reputation | Check complaint and unsubscribe rates for each version |
| Never retesting | Audience and context change | Revisit important results every year |
Testing and deliverability
Treat deliverability as a guardrail. A subject line that raises opens by using urgency or all capitals may also raise complaints, which affect reputation for every later campaign. Always compare complaint rate, unsubscribe rate and bounce rate between versions, and discard a winner that hurts them. Do not test authentication, list sources or unsubscribe placement by A/B methods on live subscribers; fix those on principle. For content changes that could trigger filters, run the spam checker first. Read spam complaint rates for what to watch.
A simple testing programme
- Pick one metric tied to revenue or engagement.
- Plan a test each month for the campaign with the largest audience.
- Start with high-impact elements: offer, subject, call to action.
- Keep a log with hypothesis, sample, duration, result and decision.
- Roll out the winner and test the next idea against it.
Done with discipline, A/B testing turns opinions about email into evidence. Done carelessly, it produces confident answers that are wrong. Fix the plan before you press send, respect the sample size and judge by what matters to the business.
Run campaigns on solid delivery Authenticated sending and clean event data make test results more reliable.
References and further reading
Frequently asked questions
How many recipients do I need for an email A/B test?
It depends on your baseline rate and the lift you want to detect. Detecting a 20 per cent relative lift on a 3 per cent click rate needs roughly 12,000 or more per version.
Should I judge a test by open rate?
Prefer clicks and conversions. Mail privacy features inflate opens and weaken them as a signal.
How long should an email test run?
Decide in advance. A day or two covers most clicks for typical campaigns, but longer for slow audiences.
Can I test more than one change at once?
Only with very large lists and a multivariate design. Otherwise test one variable at a time.
Does a winning subject line always help?
Not if it raises complaints or unsubscribes. Check those metrics for each version.