As web marketers seek new ways to boost conversion rates and improve their visitors' site experience, interest in multivariate testing is on a feverish rise. But those unfamiliar with the techniques are often unclear about where to start, or how to ensure success.
In this article, I'll discuss the following:
- A clear definition of what multivariate testing is and how it differs from another common type of testing: A/B testing.
- How testing provides a foundation for continuous improvement of your Web marketing initiatives.
- An overview of five common mistakes to avoid when planning and running tests.
What Is Multivariate Testing?
Common methods for running controlled experiments on Web sites range from simple A/B testing to sophisticated multivariate testing, also known as multivariable testing.
In A/B testing, one or more new versions of a page or single site element competes against an existing control version. For example, two versions of a headline might compete against an existing headline.
Multivariate testing, on the other hand, is like running many A/B tests concurrently, where there are multiple elements being tested at the same time. For example, two alternate product images, plus two alternate headlines, plus two alternate product copy text, for a total of 27 possible combinations (including the original control versions).
What's important to understand about multivariate testing is that it not only shows you which combination of elements generate more sales or pull more leads but also reveals which individual elements influence visitor behavioral vs. those that do not. For example, did variations in product image influence visitor behavior more, less, or the same as the copy?
Understanding how each site element causes visitors to interact with your site is the essence of a test-learn-repeat process that marketers can use to synthesize new ideas and continually improve their site's ability to achieve—and exceed—their marketing goals.
Testing as a Platform for Continuous Improvement
The process of testing reveals not only what works and should be implemented but also what doesn't work and should be avoided.
Every web site idea, whether related to content, functionality, or campaign, should be put to the test to determine whether it helps or hurts the visitor experience. While some new ideas lift conversions, others fail—sometimes significantly. But even in such failure there is definable knowledge gained about what to avoid the next time.
The ability to test a new idea and "look before you leap" is an unmistakable advantage that breaks the constraints on marketing innovation. Only once a solid testing capability is in place, and the impact of any site change quantified, can marketers truly optimize their site's effectiveness.
What Are Common Errors to Avoid?
There are five types of mistakes that are easy to make when running multivariate tests. In subsequent articles, we'll review these in more depth; but, in summary, they are as follows:
- Improper factoring caused by poor or no isolation of individual test changes; for example, changing a headline's text, font color, and font size, all at the same time as an A/B test instead of a multivariate test. Why is this problematic? Because it's difficult or impossible to isolate the impact of each change—e.g., was it the font color or the text that caused the visitor to behave differently?
- Running a test too short/long. Stopping a test early because you think you have a winner increases the risk for statistically invalid data and may increase time bias from events and/or conversion cycles. Running a test too long increases the risk of wasting time waiting for marginal results and consumes test sample that could be applied toward another test.
- Tracking or analyzing wrong key performance indicators (KPIs). For example, measuring a KPI that is too far upstream (in a conversion funnel) from the ultimate goal, or measuring only one KPI when there are multiple indicators or goals that matter. There's also the risk that a measured KPI improves, but at the expense of another (untracked) KPI, or that the measured KPI is actually a bad predictor of the ultimate goal.
- Not targeting or segmenting visitors. This means optimizing your site or campaign for anyone and everyone by not targeting tests to include good visitors (and exclude bad visitors) and not segmenting the results. Why is this problematic? Because not all visitors are the same—they're at different stages of the buying/customer cycle, and some may be mistakenly on the wrong site altogether.
- Not taking action on results! This could range from not making the winning changes to your site or not taking what you've learned and running another test (iterative test-learn-repeat). The risk here is that there is no momentum gained, no ongoing strategy applied, no realization of test results, and worst of all—underwhelming ROI.
How Can Multivariate Testing Optimize My Web Marketing?
Multivariate testing can yield some spectacular results in enhancing online effectiveness.
For example, we worked with a well-known online auction house to perform a series of multivariate test campaigns to understand which elements were most influential in bidding conversion. The team tested variations in elements such as catalog page layout and messaging, individual item landing pages, and calls-to-action.
The firm made test variations in critical elements on the site, such as creative elements, event promotions, image sizes, copy, navigation, and page layouts, without the need to make a single change to the underlying catalog auction system. These changes resulted in...
- A 429% increase in bidding activity
- An 83% increase in catalog browsing activity
- A 166% increase in individual item views
- A 590% increase in opt-in registrations
If you are looking optimize your Web marketing, multivariate testing should be part of your arsenal of analytics and optimization tools.
Next, I'll take a look at how to define your success goals and what to measure in multivariate testing. In the meantime, if you have any questions, feel free to email me at ehansen@sitespect.com.
