Hotmail Reputation Panel Data: What It Is, How It Influences Inbox Placement Rates (Part 1 of 2)

As discussed in my previous post, email providers have long used engagement metrics to determine email reputation and deliverability. These metrics include “This is Spam” (aka TiS) data, “This is Not Spam” (aka TiNS) data, panel data, Trusted Reporter data, and Inactive Account data. In this post, we’re going to focus on the largely unknown, yet highly effective use of panel data in Microsoft’s anti-spam filtering technologies.

Panelists, or Windows Live Sender Reputation data as Microsoft calls them, are a small group of Windows Live Hotmail users that come from every corner of the world, and classify up to 300,000 messages per day as “junk” or “not junk.” These users are selected at random (sorry, invite only!) and can come from any type of email service that Microsoft offers, like Windows Live or MSN. These panelists are actual subscribers on your lists and are asked to evaluate messages that either have landed in the spam folder or in the inbox and determine if it’s something they expected to receive or not.

Microsoft’s use of panelist data helps with sender reputation scoring in its own anti-spam filtering product, SmartScreen, as well as Return Path’s Certified product. Microsoft factors panelists’ votes to determine a sender’s reputation. Their decisions can negatively or positively affect your email deliverability. Let’s say, like in this example for Coach, that most of your email gets delivered to the spam folder. Randomly selected panelists, who are also subscribers in your program, are asked if your message is something they expected to receive in their inbox, or something that they didn’t expect to receive. If a majority of panelists vote “Not junk e-mail,” your chances of reaching the inbox are greatly increased to prevent a false positive. As a result, panelist data had a direct positive impact for your email delivery.

The same holds true for the inverse. If it lands in the inbox and most people respond by saying it’s a message that they didn’t expect to receive, like this message which is clearly spam, future messages will more likely be delivered to the spam folder. False negative averted!

Microsoft uses panelists for training SmartScreen as it’s more reliable than Bayesian content filters or even “This is Spam” and “This is Not Spam” votes. What’s more, it’s a system that can’t be gamed or abused. The message from Microsoft is clear. You can jump IPs, hop domains, try to manipulate complaint data, but you still can’t hide from panelists. Since these users are not known, are selected randomly from Microsoft’s most active users from around the world, and are given messages at random to vote on (without consideration of the sending domain or IP), the chances of changing the outcome of this filtering process are highly improbable. Additionally, panelists can only vote on a message when it was received within 24 hours, so the timing of the send and complaint can’t negatively impact the sender’s rating, something that frequently happens with other feedback loops.

Marketers can access this data in two ways. The first is through Return Path Certified as it is part of the compliance checks to remain in the program. The second way is through Microsoft’s Smart Network Data Services (SNDS) through its filter results. While they don’t display the actual panelist votes, they do have a color coded result (green, yellow, and red) which is based off of their SmartScreen filtering product. SmartScreen is widely used in most of Microsoft’s email products ranging from their free webmail services, Exchange, Hosted Exchange, as well as Outlook. SmartScreen looks at things like a properly configured mailing infrastructure, authentication, content, spam traps, unknown users, complaints, as well as panelist data. If the data you have access to is showing that you are fully within compliance at all of these metrics, and your filter result is red, you most likely have a problem with panelist votes.

What can you do to make sure that panelists recognize your email as wanted and vote you into the inbox? In Part 2 of this post my colleague Bonnie Malone will discuss strategies to help.

minute read

Popular stories



BriteVerify email verification ensures that an email address actually exists in real-time


The #1 global data quality tool used by thousands of Salesforce admins


Insights and deliverability guidance from the only all-in-one email marketing solution

GridBuddy Cloud

Transform how you interact with your data through the versatility of grids.

Return Path

World-class deliverability applications to optimize email marketing programs

Trust Assessments

A revolutionary new solution for assessing Salesforce data quality


Validity for Email

Increase inbox placement and maximize subscriber reach with clean and actionable data

Validity for Data Management

Simplify data management with solutions that improve data quality and increase CRM adoption

Validity for Sales Productivity

Give your sales team back hours per day with tools designed to increase productivity and mitigate pipeline risks in real-time