A/B Testing and Wine Tasting: Maximizing Your Return on Effort

I’m not a wine snob – I can’t always tell the difference between an expensive bottle and a reasonably priced one. But I do know that a reasonably priced bottle of wine often behaves like an expensive one when it’s been opened, aerated and decanted. To get this result, I have to be patient and wait for the wine to breathe. Some days, that’s easier said than done.

I feel the same way about A/B testing. Taking the time to get the best results is a best-in-class recommendation I’ve given to hundreds of demand creation professionals, but it requires patience. A lack of time often holds marketers back from developing a comprehensive and consistent testing strategy. Many believe they can’t take the time to put together tests, wait for and analyze results, and then revise programs based on performance.

But how much time is wasted on things that are not working because we’re not testing? Testing helps determine what works and what doesn’t and allows us to spend more time on things that help us get improved results to achieve our goals. Can we put in moderate effort and still get maximum results?

Using the wine analogy, here are four approaches to A/B testing, from the least amount of effort to the most effort:

• Pour a glass of wine directly from the bottle (minimal effort, minimal return). If you don’t have to time to wait for the results on your A/B testing, split your list 50/50. Send variable A to 50 percent of your list and variable B to the remaining 50 percent of your list. Make sure the list splits are at random and all other elements are identical. Analysis of the results can determine which variable performed better. These results can be used for similar programs in the future.
• Use a wine aerator (minimal to moderate effort, moderate return). Many marketing automation platforms include out-of-the-box features for conducting A/B tests from within the software. For example, if you have 1,000 contacts in a list and two offers to test, you might use 10 percent of the list (100 contacts) as testing group. Half the contacts on the list would receive offer A, while the other half would receive offer B. After one week, analyze the results and send the offer that performed better to the remaining 90 percent of the original list.
• Aerate and let the wine breathe (minimal to moderate effort, moderate to maximum return). Not every test needs to be A/B or multivariate. You can also incorporate alternative subject lines into your perpetual nurture flows. For example, create one email (body copy and offer) and two subject lines. Send the email with the first subject line option to the entire list, and then send the same email with the second subject line option to contacts who did not open the first email. Reverse the order and see if the results over the same period of time yield the same response. If one subject line performs better, use it with future emails. If the subject lines perform at the same rate, it’s most likely the email frequency – not the content – that is impacting the results.
• Aerate, decant and wait (moderate to maximum effort, maximum return). Incorporate a test into every program you execute and make it part of the overall plan, rather than a separate activity. Every program is already an experiment. Why not formalize the approach? Create a hypothesis to prove or disprove.

Testing is a proven strategy to increase overall marketing effectiveness and improve overall program results. However, many marketing professionals do not consistently leverage testing in their demand creation programs. Just like with wine tasting, if you are patient and give testing the time it requires, minimal to moderate efforts can yield maximum return.