A/B test

Analytics

A test that compares two versions of a message, offer, or creative to see which performs better.

What is A/B test?

An A/B test is a controlled experiment that compares two versions of a single variable to determine which performs better on a defined metric. In B2B marketing, A/B tests are used to compare subject lines, email body copy variants, ad creative, landing page headlines, offer framings, and CTA wording. One variable is changed between version A and version B, all other elements remain constant, and the results are compared once enough data is collected.

The discipline that makes A/B testing valuable is changing only one thing at a time. Testing a new subject line alongside a new email body produces a result you cannot interpret because you do not know which change drove the outcome. This confusion is called confounding. Rigorous A/B tests isolate a single variable and control everything else.

Statistical significance is the standard for determining whether a result is real or a product of random variation. A subject line that achieves 35% open rate versus 30% in a test of 50 sends each may just be random. The same result across 300 sends each is statistically meaningful. Always validate that your test sample is large enough before drawing conclusions and making permanent changes to your sequences or campaigns.

Analytics terms are useful only when they change a decision. A metric can look sophisticated and still be low value if nobody knows how it is calculated, which segment matters, or what action should follow when it moves. It usually becomes more useful when it is defined alongside Hypothesis, Conversion rate, and Creative fatigue.

A/B test — example

A B2B agency tests two subject line approaches for a campaign targeting HR leaders: Version A uses a curiosity-based subject line ("Have you heard about this hiring trend?"). Version B references a specific operational challenge ("Reducing time-to-hire in a tight labour market"). After 400 sends per variant, Version B achieves 41% open rate versus 27% for Version A. The agency updates all their HR-targeted sequences to use the specific-challenge approach and establishes it as a messaging principle for the segment.

A B2B team uses A/B test to compare sources that look similar at the lead level but perform very differently once quality and pipeline impact are included. The metric becomes more useful once it is reviewed by segment instead of in aggregate. They also make sure it connects cleanly to Hypothesis and Conversion rate so the definition is not trapped inside one team.

Frequently asked questions

How do I run an A/B test properly in a cold email sequence?

Randomise your list so both groups are statistically similar. Change only one element between A and B. Run both versions simultaneously rather than sequentially to control for time effects. Collect enough data to reach statistical significance before declaring a winner. Do not check results daily and change the test early.

What is statistical significance and do I really need to worry about it?

Statistical significance measures whether your result is likely due to the change you made or just random variation. In practice, for cold email tests, use a simple rule: minimum 200 sends per variant for reply rate tests. If both variants have not reached 200 sends, do not declare a winner. Free A/B test significance calculators confirm whether your specific result is reliable at your sample size.

How long should an A/B test run before I review results?

Until both variants have reached your minimum sample size, not by calendar time. A test with 50 sends per week takes at least four weeks to reach 200 sends per variant for a reply rate test. Reviewing results at week two when you have 100 sends per variant and declaring a winner produces unreliable conclusions.

What should I do after an A/B test confirms a winner?

Update your default to the winning variant. Document the result with context in your experiment log: what was tested, the sample sizes, the result, and the conclusion. Identify the next variable to test based on what the winning result suggests about your audience. Keep the losing variant for potential retesting in different contexts.

Can I A/B test personalisation approaches?

Yes, and this is one of the highest-value tests for outbound. Compare a genuinely personalised approach, using a specific signal about the prospect, against a well-written segment-level approach. If personalisation does not produce a meaningfully higher positive reply rate, you are spending enrichment cost without the return. The result tells you whether deeper personalisation is worth the investment for your specific ICP.