Don't Roll Out the Loser: How To Analyze Subject Lines

Testing is incredibly important.  If this comes as a surprise, you have some serious homework to do.  Testing should be the caveat for just about every recommended change for an email program.   As a marketer, you’ve likely heard, read and said “test, test, test” more times than you can count.  What can often slip through the cracks are tips on how to analyze email tests in a way that have the most positive impact for your program.

While many of the recommendations in this post can apply to email content and program strategy testing, we’ll focus specifically on subject line tests.   In combination with the friendly handle or from address, the subject line is what inspires the open with subscribers.  You can have the best content in the world, but if your subject line bombs, that fantastic email creative won’t get its time in the sun.  Subject line testing is one of the easiest to run and can provide a nice boost to campaign performance if it’s done well.

Give Tests Time to Mature

It’s all too easy to jump the gun when analyzing subject line results, especially when a smaller test segment is used prior to deploying to the full list.  Campaign results at 6 hours can look drastically different than those at 24 hours.  Understanding the engagement patterns of your subscribers and establishing timeframes accordingly can help ensure that you select the subject line that is the true winner.  Run a subject line test or two where results can be interpreted over the course of several days.  Keep a close eye on opens, clicks and conversions, and track when engagement rates slow substantially.  Repeat the process several times until a consistent pattern of engagement can be identified.  How long you need to wait to determine the winner of a subject line test should be dependent on the response cycles of recipients.  

Defining the “Win”

Many marketers make the mistake of focusing only on open rates for subject line tests.  While this metric is fitting for some campaigns, establishing this as the go-to may end up hurting performance in the long run.  For example, a vague but interesting subject line that drives high opens may not be channeling qualified traffic into the email and website, which can damage conversion rates and/or order values for the campaign.  By making roll out decisions based on the metric that is most appropriate and closely tied with the overall objective of the message, subject line tests can help get you closer to your goals. 

Based on the needs of your program, the following metrics can be leveraged to define a winner:

  • Open Rate – Focus on open rate when the goal of the campaign is to maximize subscriber impressions.  This metric is ideal for major launches, to raise awareness or drive offline activity.  Unique opens allow insight into the overall reach, where total opens can provide perspective on whether subscribers are opening the email more than once.
  • Click Rate – Review this metric for campaigns that are intended to drive traffic to destination pages in order to contribute to branding, add value, maintain or confirm engagement, etc.  Testing for unique clicks maximizes exposure to the website and landing pages, where total clicks can be a good indication of how engaged subscribers are with the email content. To drive as many people as possible to a landing page, review unique clicks.  For emails with multiple important calls to action, consider reviewing total clicks.
  • Click to Open Rate – The difference in how this metric is calculated allows you to further determine whether the traffic coming in to the email was qualified and engaged.  Low rates indicate that the subject line may be missing the mark with well-qualified subscribers or there may be a disconnect between what the subscriber expects and what the email content contains.
  • Conversion Rate – A high conversion rate for a specific subject line indicates that it is driving high quality, motivated traffic to the website.  While these subject lines may not win in terms of overall opens, testing for conversion rates ensure that the right subscribers are engaging with the email.

Beware of Negative Metrics

Though the goal of most subject line tests is to drive engagement, neglecting to review metrics that indicate subscriber dissatisfaction can have long term implications for a brand’s deliverability and reach.  Does one subject line drive more opens but also contribute to higher spam complaints and unsubscribes?   Over the long run, these types of campaigns can strain relationships with both subscribers and ISPs.

During testing, carefully review complaints and unsubscribes across individual campaigns and emails series to ensure that increased conversions or email activity does not also translate into problematic subscriber behavior.    While they may provide a quick lift, campaigns that provoke negative behavior can create deliverability issues and distaste for the brand.  These problems can be long lasting and detrimental for the email program.

Interpreting Results by Segment

Different subscriber groups respond to messages in different ways.  If you fail to review tests by different key segments, you may be missing important variances in response rates.   While it may not be efficient for each test, broaden your understanding of what works or falls flat by testing subject lines with key segments in your program.  Additionally, segmenting by recency of email or conversion behavior can provide meaningful results with important implications for your program.  Understanding the differences between what drives higher response rates for new subscribers versus those nearing inactivity thresholds can have a long-term positive impact on engagement rates.  Make this step easy on yourself by setting up tests with segmentation in place prior to deployment.

Results Not Meaningful?

Minor variations in verbiage and punctuation may not lead to the strong results that you’re looking for.   Additionally, some campaigns may require several rounds of testing before the true winner reveals itself in the data.  Be bold and experiment with different approaches but be careful to only test one at a time.   As we’ve often seen in Inbox Insight, subject lines that break the mold and employ content that is attention grabbing and fresh can lead to a lift in read and forward rates.  Consider including the following approaches to your tests:

·         Vague/Mysterious Subject Lines

·         Subject Lines that pose a question

·         Inclusion of special characters and symbols

·         Personalization/Dynamic Content

·         Subject Lines with very limited character counts

·         Humorous approaches

·         Social media focus (most shared, highest rated, etc.)

Don’t Bury Your Results

Can you easily access and review the results of the tests you’ve run over the last year?  Are you using the results of prior tests to provide direction for upcoming campaigns or new tests?  If not, the creation of a centralized repository of test results is an important next step.  Consistency in how results are reported and summarized makes this information easier to process and build upon.  Leveraging past results can enhance the effectiveness and efficiency of future efforts.

Keep It Up

There should never be a time where your brand has finished testing subject lines (or any other email program element).  This goes for each and every type of email in your program.   Not only will this help keep subject lines fresh, it will ensure that they resonate with subscribers. 

minute read

Popular stories



BriteVerify email verification ensures that an email address actually exists in real-time


The #1 global data quality tool used by thousands of Salesforce admins


Insights and deliverability guidance from the only all-in-one email marketing solution

GridBuddy Cloud

Transform how you interact with your data through the versatility of grids.

Return Path

World-class deliverability applications to optimize email marketing programs

Trust Assessments

A revolutionary new solution for assessing Salesforce data quality


Validity for Email

Increase inbox placement and maximize subscriber reach with clean and actionable data

Validity for Data Management

Simplify data management with solutions that improve data quality and increase CRM adoption

Validity for Sales Productivity

Give your sales team back hours per day with tools designed to increase productivity and mitigate pipeline risks in real-time