Allen Tyler
Allen Tyler

Posted on: August 24, 2012

In: Design


A/B Testing

A/B Testing (sometimes referred to as a Split Test) is the type of test that many people are more familiar with. It is a simple concept. In its most basic form, this test involves comparing the effectiveness of the default (or Control) variation of a single web page to the effectiveness of a single variation of that control page. In many cases, only a single element on the page will be different between the two variations. We have seen these tests used for all sorts of changes to a page: images, calls to action, button text, button colors, layout, background color, etc.

ABT chat widget

In one A/B test we saw big results from simply moving a chat widget up the page.

But, what if you have more than one alternate variation? Don’t worry! You’re in luck. A/B testing doesn’t mean that you can only have an “A” and a “B” variation of a test. It is sometimes also referred to as A/B/n testing, because this type of test can be run with as many variations of a single element as desired. However, it is important to keep in mind that more test variations requires higher traffic volume to determine a statistically significant winning experience.

ABT KU button

One test had four possible experiences for a visitor to see (three variations and the default). The only difference between each experience was the text actually seen on one of the CTA buttons.

When do I use an A/B test?

Use an A/B test when you are trying to optimize a single element at a time. Layout, copy, colors, position, back-end system, and CTA are all good elements to test. When you want to see the impact of one single change on your website goals, use an A/B test.

Testing multiple page elements

While A/B testing is used to test multiple variations of a single element on a page, multivariate testing (sometimes shortened to MVT or MV Testing) is used to test multiple variations of multiple elements on a single page. This would allow us to test combinations of variations of multiple elements. Rather than sequentially running A/B tests on each of these elements, utilizing multivariate testing allows us to test variations of all of those elements at the same time.

There are two big ideas behind the results found from multivariate testing. One of these is that the winning experience will be a combination of the winning element variations that have the highest success rate (or conversion rate) for a given website. This combination can then be used as the sole experience for the page, with the knowledge that it was the best performing combination out of the available variations.

The other important thing that multivariate testing provides insight to is element contribution. This type of analysis tells the analyst specifically which elements had the biggest impact on increasing the conversion rate, and then more specifically which variations of those elements were more effective.

For different types of sites, the most important element is likely to be very different. Some sites may find that headline copy is the most important factor in boosting conversion rates, while other sites may learn that imagery is most important to its users.

ABT homepage

This test allowed us to determine the best copy and imagery for a key call to action.

When do I use a multivariate test?

Multivariate tests are best used to more quickly determine the most effective combination of possible elements on a page. In a sense, this type of test can be thought of as multiple A/B tests being run simultaneously to determine the best possible combination of winners.

Use a multivariate test if you have a hypothesis that there are multiple elements on your page or site that, with improvement, could increase your conversion rate. It’s usually ideal to do MV Testing on a high traffic page.

The important takeaway is that what works for one company may not work for another – it’s important to test, test, test to figure out what works best for your own site.

What are your pain points with testing? Have you seen dramatic or confusing results from tests you’ve performed?

Recent Posts

Upcoming Events

CHIFOO Meetup: March

March 4 @ 5:30 pm - 8:00 pm PST


Ready to make your next project the first step in delighting your customers and growing your business? Let’s talk!