Pre-testing: A Better Way to Beat the Control

Pre-testing: A Better Way to Beat the Control

Issue: 

2014, February

The A/B split test is as fundamentally sound, as it is slow, expensive and inefficient. 

Direct mail testing—even for high-volume mailers—means putting a finite number of possible tests in market.  These tests are either incremental changes or wholesale redesigns.  The former usually assures incremental gain/loss, and the latter requires a lot of risk and reputational capital.    

Truth is, the vast majority of tests fail.  For two reasons.   
 
1)   The control is hard to beat.  It benefits immensely from something your test package can never have—exposure.  Your target audience, however large, is still finite.  You mail the same people over and over.  Even non-buyers are exposed to the control. 
 

2)   Coming up with a test, or 10 tests or 100, for a single campaign is like searching for a needle in the haystack.  The number of possible test packages is infinite, and you are forced to choose an imperceptibly small percentage of them to mail.    

In split testing, the odds are against you.  With a complete redesign, the odds are even more onerous.  The A/B test will tell you if the new package test wins or not.  But it cannot tell you why.  Perhaps there are components within the complete redesign that are clear winners, but are getting drowned out by the weaker elements.  

So, what is a better alternative?  Look no further than the consumer package goods industry for an idea.  CPG marketer do lots of in-market testing, also lots of product development work in advance.  By comparison, the direct marketing world does very little. 

The methodology now available to direct marketers is sophisticated, but simple and intuitive.  In short, It is an online testing program through which your target audience evaluates thousands of different direct mail package ideas in mere minutes.  

The secret to its success, and why this pre-testing matches up with live test results so well, is the ability to replicate real-world choice and decision making by:

1-  Showing the target audience concepts, packages or offers holistically, just as they view them home. 

2-  Asking the target audience answer one question—overall preference—within a few seconds, the same amount of time you get before your package is thrown in the trash.  

Behind the scenes is very sophisticated statistical modeling to answer the question we really want to know, which is Why?  You end up with scores for every single test element, which may encompass 30 or 40 different component parts of a direct mail package (OE, letter, buck slip, reply form) and, in turn, thousands of package combinations.  

The business upside is four fold:  
- Test exponentially more ideas in a radically shorter period of time.
-   Put fewer tests in the mail, and at higher volume, to get to rollout faster.
-    Find out exactly what impact each component has on preference and response.  
-  Test big ideas, those that would never make it in the mail unless you have empirical proof they can work, in a low cost, low risk environment. 

A/B split testing method has been around for decades, and yet direct marketing has changed dramatically, with more channels, better targeting, better production methods and now, a better way to beat the control.


Author: 

Chris Locker
Chris Locker's picture

Chris Locker is EVP of marketing and strategy at The Consumer Voice, in the Minneapolis area.  Reach him at clocker@theconsumervoice.net.