Customer Blog Takeover: You’re Testing That Wrong if…

Working as an account manager at a major email service provider was a wonderful opportunity. I spent 5 years learning the ins and outs of a variety of programs – retail, ecommerce, B2B and even a video store. Yes, I’m dating myself here. Kids, that used to be a thing! You’d walk around and look at new releases and then….oh never mind.  

Of course, being an account manager wasn’t all sunshine and puppy dogs. Anyone who’s been in a similar role will tell you that some clients are better than others.

I’ve had clients who loved to wait until 5 pm on a Friday before they decided to change the creative on an offer going out Saturday morning. Certain clients liked to use very colorful magic words in the rare case of a QA miss. I didn’t really mind a late night or inventive use of English language – it’s all par for the course. What really ground my gears was this: a complete lack of strategy around testing.

Whether you’re brand new to the email marketing world or a seasoned wiz, the directive coming from leadership is usually Test, Test, Test. Sure, it’s easy to slap together a subject line A/B or pit one template against another. But, it can be really time-consuming. And, what I found too often was that clients didn’t leverage any of the findings.

Having a defined testing strategy is just as important as impactful creative, targeting users with the right message at the right time, and maintaining data quality. By the way, if you need help with the latter, I know this really cool company with a great Certification program.

Companies large and small are guilty of willy-nilly testing strategies. Let’s take for example a very well-known e-commerce retailer who shall remain unnamed. Three or 4 times a week, we’d run some kind of 50/50 subject line test. One day, it would be a discount offer vs free shipping. The next day, it would be a green shirt vs a blue shirt. Going into the weekend, let’s go with header navigation vs no navigation. It was testing for testing’s sake. It’s testing so you can tell your CMO you’re testing.

To be fair, maybe I didn’t have full visibility into my ex-client’s overall strategy. Maybe they were stashing the results away in some kind of master marketing tomb to create the ultimate customer profile. It sure didn’t feel that way.

In any case, there are some very basic rules for differentiating a good test from a bad test, whether you’re about to send your first email or your 35,000,000th.

Here are three tips to keep in mind:

  • Set specific KPIs – What is the goal of your test? Are you trying to increase your average order value? Grow your list? Understand how to make better product recommendations? If you don’t know what your key performance indicators are, don’t bother testing. The idea of testing is that you use the results to make your program better. Figure out what needs to change first. 
  • Make results actionable – If your creative department is 6 months behind and you know there’s no way you can change your email template…don’t test your email template.  Partner with other teams in your organization, like IT and web production, to understand what’s an easy win and what’s not going to happen this quarter…or year. 
  • Go for quick wins – Unless you’re using the results quickly, what works this week may not work the next. That’s why I’m a big fan of the 10/10/80 test. Ten percent of the audience gets one experience, 10% gets another experience, and then after a set amount of time (commonly 24 hours), the remaining 80% gets the winning experience. Boom. You’re capitalizing on how your audience is feeling at the moment. Salesforce Marketing Cloud is one email service provider with an automated A/B testing tool that makes the setup on this really easy.

I know it can be hard to pull out of the day-to-day grind to look at strategy from a higher level, but it’s well worth the exercise. Better tests mean better results. Better results mean satisfied CMOs. Satisfied CMOs mean happier email marketers.

minute read

Popular stories



BriteVerify email verification ensures that an email address actually exists in real-time


The #1 global data quality tool used by thousands of Salesforce admins


Insights and deliverability guidance from the only all-in-one email marketing solution

GridBuddy Cloud

Transform how you interact with your data through the versatility of grids.

Return Path

World-class deliverability applications to optimize email marketing programs

Trust Assessments

A revolutionary new solution for assessing Salesforce data quality


Validity for Email

Increase inbox placement and maximize subscriber reach with clean and actionable data

Validity for Data Management

Simplify data management with solutions that improve data quality and increase CRM adoption

Validity for Sales Productivity

Give your sales team back hours per day with tools designed to increase productivity and mitigate pipeline risks in real-time