Email Testing

Why use email testing tools?

As students, we were taught to never walk into class unprepared. Email testing takes that same philosophy and applies it to campaign quality control. Hours spent on campaign planning, design, and coding are wasted if customers don’t engage with your emails.

Testing helps catch problems before deploying a campaign. A lack of testing can lead to a variety of unintended consequences, such as unreadable subject lines, strange rendering in various email clients, broken links, or emails that arrive in spam folders.

Types of Email Tests

Preview/Design Tests

They say image is everything, and when it comes to email marketing, images that display correctly can make the difference between failure and successful campaigns. Design testing tools can give you confidence that your emails look great for subscribers, regardless of their viewing environment.

Ideally, every mailbox provider—including Google, Outlook, and Yahoo—would render messages the same way. In the real world, different email clients have their own display quirks. What looks great on Outlook’s desktop app can look dramatically different on iPhones and other mobile devices.

Good design testing tools show previews of what your email template will look like across a wide range of inboxes and help you identify which ones matter most to you. Most will display email previews of desktop, mobile, and webmail inbox rendering environments. Some can show what an email looks like to colorblind readers, which can be helpful if you’re layering colored text on top of images. Many tools will also show how an email looks in Dark Mode, which can completely change the way some templates display.

Predictive eye-tracking technology can be used for more advanced design tests. These tests will help you measure the visual impact of your email and predict which content stands out most and in what order elements are read. This allows you to optimize the rendering of your message and what content to emphasize.

Design testing is important to ensure your emails aren’t truncating, your image load times aren’t excessive, and your most important content is shown above the fold. Ultimately, design testing provides actionable results that can be shared with your team to improve the look and subscriber experience for every campaign.

HTML Tests

First, a good tool should identify that all image paths are valid and that all images include alt text. It can also show you the file size for each image, which is important to know since email clients like Gmail will truncate messages that exceed their limit. 

Next, the tool should identify broken or questionable links. If your email includes links to multiple websites, the test should be able to verify whether any of them are pointing to blocklisted domains that would raise spam alarms. However, no tool can verify a link is landing where you intend, so if you expect your hero image to lead to your homepage, it’s still a good idea to check that it’s not pointing to your “About Us” page. 

Finally, HTML testing shows if your coding follows best practices, such as including closing tags for every element. Choose a tool that shows useful information, like which code is incompatible with specific clients, along with details about why it is incorrect, so you can resolve any issues. A good tool will provide visibility into your subscriber’s distribution across different devices and OSs, so you can focus your energy on the platforms that have the most impact.

Subject Line Tests

It’s important to check that subject lines and pre-header text are readable, since subject lines can determine whether or not subscribers open your messages. Subject line testing provides insight into how the text looks on different devices, giving you the opportunity to tweak the copy before hitting send. 

Subject line testing tools show how much of the subject is visible as a preview. The best practice is to write subject lines that contain around 45 characters, but even then, some inbox displays will cut off the end of the line. Testing shows where the preview text ends so you can lead with the most important messaging. 

For example, let’s say a retailer knows they have higher open rates with subject lines that include a discount. 

Something like “See What’s New for Fall + Summer Clearance 70% Off” might only show “See What’s New” in the preview within some viewing environments.

After testing, the wording order could change to “70% Off Summer Clearance + See What’s New for Fall” so customers don’t miss the portion of the subject line that’s most likely to trigger an open.  Because there are length restrictions, it’s important to take advantage of the pre-header text to act as a separate subject line.

Emojis are another way to capture attention, but not every inbox displays them the same way. “Turn that ☹ into a 😊” becomes “Turn that into a” when the emojis don’t appear. Email marketers using a subject line testing tool would catch this early, giving them time to optimize the subject line to something like “☹ Turn that frown upside down 😊” which is readable in every inbox.

Spam Tests

Spam testing goes beyond validating creativity to check for spam-related warning signs. These tools test more technical details, such as the reputation of the IP addresses you send from. Any good testing program should include checks for authentication, filtering, blocklisting, and mailbox placement.

An authentication check verifies that your IP address has valid SPF and DKIM records. These are basic anti-spam protocols that prove your domain belongs to you.

Filter checks show if your email is flagged by popular spam-blocking services. These spam filters are used by businesses and mailbox providers to help stop spam before it gets into the inbox.

A blocklist check validates that your sending IP addresses don’t appear on blocklists. These lists are used by mailbox providers to determine if a sender has a good reputation or if they’re a potential spammer. If your IP address appears on a blocklist, you can sometimes request to have it removed. Other times, you’ll need to improve list hygiene processes to be taken off blocklists.

A mailbox placement check works by sending test emails to various email accounts. The tool shows whether messages arrived in the inbox, spam folder, or were blocked entirely. For emails sent to tabbed inboxes, you’ll also see which tab your message was placed in.

And of course, it never hurts to check your Sender Score to see if the quality of your sender reputation has been affected by any spam complaints or blocklists.

Top spam-testing tools also validate that your template follows best practices like including a valid-unsubscribe header and abuse reporting email address. These elements help you quickly remove unsubscribe requests and spam complaints from your sends and indicate that you are a responsible sender.

A/B Tests

A/B testing is a fast, easy way to find opportunities for improvement over time. A/B testing is done by sending two versions of an email (an “A” version and a “B” version), and then comparing the results.

It is helpful to start by defining the metric you want to affect. For example, if improving open rates is your goal, A/B testing subject lines can help you discover what gets your audience to open. 

If you hypothesize that customers are more likely to open messages when you offer promotions, create an “A” subject line that includes the promotion, such as “Get Free Shipping on the Coolest Toys for Toddlers,” and a ”B” version without the promotion, like “The Coolest Toys for Toddlers.”  You would then split your audience for the campaign into two groups, sending one group the “A” version and the other the “B” version. The version that generates a higher open rate is the winner.

Keep in mind, that the introduction of Apple’s Mail Privacy Protection (MPP) impacts A/B tests that use open rate to determine the winner.  With MPP, the open rate is no longer a reliable signal of engagement for Apple Mail users. Senders should create a reliable open audience that excludes Apple Mail users or consider other metrics to evaluate the effectiveness of their tests. Senders should also experiment with which metric best aligns with their desired outcomes. For example, a winning open rate doesn’t necessarily predict higher campaign revenue.

Obviously, one test will not provide definitive answers, making it important to conduct A/B testing over multiple deployments. Once you’ve consistently seen a lift to your test metric, you can confidently say subject lines with promotions see higher open rates, or creative that includes a clear CTA generates a higher click-through rate. Every campaign is an opportunity to learn and continually improve the performance of your email marketing. 

Unlike A/B tests, which focus on just one variable within your email, multivariate tests can be used to test multiple variables at the same time. This type of test provides insight into how numerous variables interact with each other so you can find the ideal combination. Both types of testing have their pros and cons. The type of test you use should be determined based on the overall goals of your test.

Frequently Asked Questions About Email Testing

How can I test my email?

How do you use email testing?

Is there a way to test why my emails are being placed in the SPAM folder?

What is a good way to test my email in different web applications?

Design testing tools, like those included in Everest, show how your email will display in a range of inboxes. This helps you update your templates so they look great no matter where your audience is reading them.

What is the right sample size for an A/B test?

Your sample size should be dependent on your total list size. We recommend using a sample size calculator, like this one, to determine the correct size for your list.

Conclusion