Rethinking your A/B Tests

Let’s say you ran an A/B test. The email with the higher open rate wins, right?


Unfortunately, most Email Service Providers (ESPs) that provide an automated A/B test in their tools will automatically pick the campaign that has the best open rate, but you may be missing more critical metrics (revenue, conversion rates, clicks, etc)

To illustrate, here are real A/B test results I reviewed for a client where they tested two subject lines with the same email content:


When I saw these results, I noticed that Subject Line B had a higher click through rate (CTR) and a lower unsubscribe rate. Even if the CTR was higher, this doesn’t guarantee that Subject Line B would have performed better since the OR was lower. Without having access to conversion or revenue data, I re-analyzed the A/B test results with the objective of finding the campaign that would have resulted in higher clicks through to the website.

To do this, I took their sending volumes for both campaigns and “reset” them to reflect results for sending to exactly 100,000 subscribers for each test group:

Client Supplied Data:


“Reset” Data:


Based on these results, it becomes very clear that Subject Line B should have been the test winner. Even though Subject Line A had more opens than Subject Line B, these results show that Subject Line B was better at preparing the subscriber for the email content and resulted in a higher number of click-throughs and lower unsubscribes.

When running A/B tests, make sure you measure the results against your email goals. Open rates are important, but should only determine the results of an A/B test when the goal of your email program is brand awareness or re-engagement.

For more information on A/B tests, here are some helpful links:

  • All About A/B Testing” by Return Path. In this ebook, we’ll give you the tools to become an A/B testing expert, including: How to set up and run your A/B test, analyzing your results, A/B testing best practices, and 50 testing ideas to get you started.
  • How to Set Up and Run an A/B Test” by Lauren McCombs.
  • Our free Sample Size Calculator to help determine what percentage of your list should receive the “test” version of your email marketing experiments.

minute read

Popular stories



BriteVerify email verification ensures that an email address actually exists in real-time


The #1 global data quality tool used by thousands of Salesforce admins


Insights and deliverability guidance from the only all-in-one email marketing solution

GridBuddy Cloud

Transform how you interact with your data through the versatility of grids.

Return Path

World-class deliverability applications to optimize email marketing programs

Trust Assessments

A revolutionary new solution for assessing Salesforce data quality


Validity for Email

Increase inbox placement and maximize subscriber reach with clean and actionable data

Validity for Data Management

Simplify data management with solutions that improve data quality and increase CRM adoption

Validity for Sales Productivity

Give your sales team back hours per day with tools designed to increase productivity and mitigate pipeline risks in real-time