Ultimate Guide to A/B Split Testing – Articles, Tips, Case Studies, Tools and Resources

Posted in Tips, Tools, Usability, Web Design • Posted on 9 Comments

First of all, what is A/B split testing? In basic terms A/B split testing is a variety of marketing testing where customers randomly receive almost identical, yet slightly different, test samples. This method can efficiently be used in various niches such as brochures, email campaigns and landing pages. A/B split testing can help you to improve conversion rates, better understand visitor behavior, test out new products and much more. Email newsletter services like Mailchimp have already implemented A/B split testing in their feature range.

More and more people are becoming aware of A/B split testing and not without reason. Case studies and examples show how greatly testing can improve a site’s performance, usability and conversions. What’s more, it costs you comparatively nothing. There are plenty of useful tools already made like Google Website Optimizer which cost you nothing. Continue reading to find comprehensive guides on A/B split testing, exciting case studies, useful tools and resources.

A Few Quick Tips To Get You Started

Know what to test

The goal of A/B split testing is to improve your website’s performance, not to build a whole website. Find out what exactly you want to test before testing it. Help out with case studies and have a closer look at your own website. If you’re not sure start with a bit. Some of the most popular things to test are call to action buttons, headlines, forms, text length and layout.

Give it time

Don’t jump to any conclusions after you’ve run the test for a couple of days. A solid A/B split test might even take a month. Giving up too early on a test will only cost you time and the results will be marginal. Once you start seeing a solid path where the results are going you can close the test or start a new one. There are also calculators available to test results confidence if you’re unsure.

Analyze results

You’ve run your first A/B test, great, now what? Don’t take it for granted that the test version is always a better solution. Examine your results carefully and see where the flaws were and what you could improve even more. One test often won’t be enough. You’ll have to do several or even a multivariate one. Once you’ve done a decent amount of tests you’ll see where the actual results are going.

Mix it up

Like you just found out – one test won’t be enough. Be creative with your tests. Don’t stick to the ordinary A/B option, 50/50 audience. Create various templates and vary the proportions. Change something else in each test. At the end you’ll have rich and valuable material to analyze. However don’t get too fancy. Don’t torture your regular visitors with too many testing. Address tests to your new visitors then see how they adapt. To sum up, follow the guidelines but don’t take them as a must. Be creative with some limitations though.

Case study

Case study from Visual Website Optimizer blog. Read the full article here.


In the screenshot above you can see Basekit’s original pricing page (on the left) and the redesign (on the right) which increased conversations by 25%. The goal was to increase the number of people who visit the Buy Now page after visiting their Plans and Pricing page.

As you can see the redesigned page is more organized, smoother, cleaner, brighter and more user friendly. This design showed a consistent 25% improvement in conversations for the entire duration of the test.

Even though they accomplished their goal with this redesign, they’re already working on a new redesign to do another split test. As we already discussed – the key to success lots of tests.

This A/B test showed the importance of pricing table design. Remember that pricing table article we did recently? It’s a great example of efficient A/B testing in action.

Articles & Tutorials

1. The Ultimate Guide To A/B Testing

Ultimate-guide-ab-split-testing-resources-tutorials

This article by Paras Chopra, the founder of Visual Website Optimizer, is meant to be the best guide you will ever need for A/B testing.

2. Split Testing Landing Pages to Boost Conversions

Landing-pages-ab-split-testing-resources-tutorials

Running split tests on campaign landing pages is an important way to test and optimise conversions. Learn how to use split tests and optimize landing pages.

3. Using Google Analytics to get insights from A/B testing

Analytics-ab-split-testing-resources-tutorials

In this article Felipe Wesbonk from Traffic Builders explains the integration from VWO (Visual Website Optimizer) in Google Analytics and the way you can use this plugin in making custom reports.

4. How to Do A/B Split Testing on Your Mailing List

Mailing-ab-split-testing-resources-tutorials

Learn how to perform split testing in your mailing campaigns, check out case study and improve your results.

5. A Crash Course on A-B Split Testing

Crash-course-ab-split-testing-resources-tutorials

Another article giving you an insight in AB testing advantages and disadvantages.

6. The Pitfalls Of A/B Ad Split Testing, Part 2

Pitfalls-ab-split-testing-resources-tutorials

Article digging into ad testing and examining some pitfalls of AB ad split testing

7. A/B Split Testing vs. Multivariate: Pros & Cons.

Multivariate-ab-split-testing-resources-tutorials

Which of the two primary testing methods should you employ? Find out pros and cons of each test, when to use them together and why.

8. Gathering Variables for A/B Split Testing

Variables-ab-split-testing-resources-tutorials

This article will take a look at elements of a website that should affect its users, teach you what to test and where to gain ideas.

9. Split Testing

Zeald-ab-split-testing-resources-tutorials

Very detailed and comprehensive article about split testing basics, showing how these tests work, case study and tools to do the job.

10. Fundamentals of A/B and Multivariate Testing

Fundamentals-ab-split-testing-resources-tutorials

Look at the fundamentals of both A/B and multivariate testing and learn how to create a split test step by step.

Case studies

1. Ecommerce website? Bring price and add-to-cart button closer to increase sales

Cart-button-ab-split-testing-resources-tutorials

Insight into Taylor Gifts A/B test they recently did to improve add-to-cart clickthrough.

2. You should follow me on twitter

Twitter-ab-split-testing-resources-tutorials

Dustin Curtis outlines his experiment with Twitter links and shows the power language has on clickthrough rates.

3. Split testing works. A very short case study from AgileTask.

Agiletask-ab-split-testing-resources-tutorials

Short case study showing how call to action buttons can significally improve user reactions.

4. Writing Decisions: Headline tests on the Highrise signup page

Headlines-ab-split-testing-resources-tutorials

37signals was experimenting and rotated some headlines and subheads on the Highrise signup page to see if headlines have an effect on signups. Want to know the answer?

5. Split Testing: How To Increase Your Adsense Earnings 94% Overnight

Adsense-ab-split-testing-resources-tutorials

In this post Brian Armstrong from StartBreakingFree.com shares some tips on using Split Testing to increase his AdSense earnings.

6. Skype Boosts Homepage Conversion Rate with Radical Simplicity

Skype-ab-split-testing-resources-tutorials

Skype designed two new homepages to draw user focus to the call to action. See which was the most succesful option.

7. Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement

Google-optimizer-ab-split-testing-resources-tutorials

This post will show exactly how one start-up improved their homepage conversion rate (visitor to sign-up flow) more than 20%, then 16% again, with a few simple changes and Google Website Optimizer.

8. A/B Test Case Study: Can Split Test Results Be Trusted?

Trusted-ab-split-testing-resources-tutorials

Little A/B experiment done for the Official Vancouver 2010 Olympic Store to optimize conversion.

9. How to Double Your Subscriber Growth With 10 Minutes Work

Double-growth-ab-split-testing-resources-tutorials

See some split testing examples, learn how to do one yourself and find out which web elements to test and why.

10. A/B testing browser extensions: do annoying notifications cause uninstallations?

Browser-extensions-ab-split-testing-resources-tutorials

Most people refer to web pages when they talk about A/B testing. However, A/B testing can be useful in other domains too. At InvisibleHand, Visual Website Optimiser was used to test the performance of different versions of a browser extension.

Tools & Resources

1. Google Website Optimizer

Website-optimizer-ab-split-testing-resources-tutorials

Google Website Optimizer is the easiest and cheapest tool to do A/B and multivariate tests.

2. Visual Website Optimizer

Visual-optimizer-ab-split-testing-resources-tutorials

Visual Website Optimizer is an easy to use A/B testing tool featuring point-and-click test designer and WYSIWYG editor for creating variations. If you aren’t sure if Google Website Optimizer will work for you this is definitely the second best choice.

3. VWO Free Testing Tools

Tools-ab-split-testing-resources-tutorials

Visual Website Optimizer also offers four great free tools split testing. These include landing page analyzer, test significance calculator, test duration calculator and A/B Ideafox where you can gain ideas for various A/B tests.

4. genitify

Genetify-ab-split-testing-resources-tutorials

genitfy enables webpage authors to accelerate their development cycles. It is like A/B testing but it goes beyond it.

5. A/B Theme Testing Plugin (premium)

Theme-plugin-ab-split-testing-resources-tutorials

This WordPress plugin rotates themes to assign them evenly between visitors.

6. Vanity

Vanity-ab-split-testing-resources-tutorials

Vanity is an experiment driven development framework for Rails. It will help you to create and run A/B tests though it requires some heavy knowledge.

7. phpScenario

Phpscenario-ab-split-testing-resources-tutorials

phpScenario is an object-oriented split testing library written in PHP. It’s designed to do simple A/B testing with as little code as possible, but should also be as powerful as you need it to be.

8. ShrimpTest

Shrimp-ab-split-testing-resources-tutorials

ShrimpTest is an A/B testing solution for WordPress under development at Automattic.

9. WhichTestWon

Whichtest-ab-split-testing-resources-tutorials

A/B split testing showcase with nearly 150 case studies. Yet you can only view one per week for free.

10. ABTests

Abtests-ab-split-testing-resources-tutorials

Showcase of various split tests with screenshots, results and comments.

111 Written ArticlesWebsite

Works at 1stwebdesigner, studies photography at ISSPand seeks to enjoy life. You can check out his photography blog and follow him on twitter.

9 Comments Best Comments First
  • John

    Tuesday, August 16th, 2011 03:41

    1

    Great collection of useful information.
    I just started with Google Website Optimizer because it is the easiest tool to use.

    In your post you said: “A solid A/B split test might even take a month.”
    What is the best amount of visitors to test an A/B split testing before you know when A is better then B (and it is not just luck) ?

    0
  • Værktøj

    Thursday, September 1st, 2011 00:17

    4

    One of the best blog post in a long time, thanks

    Many regards here from Denmark

    0
  • Allan Helleskov Kleiner

    Tuesday, August 30th, 2011 08:25

    3

    Thanks for the brief overview and all the great links. Hopefully, I’ll get some devine inspiration and improve my testing.

    0
  • aditya menon

    Tuesday, August 16th, 2011 18:52

    2

    Thank you for taking time to clear up a jargon bomb! Have a nice day :)

    0
  • Lars Bo Thomsen

    Friday, September 9th, 2011 09:41

    5

    #John

    Just look in GWO, when it has declared a winner, you have enough data.

    0
  • Christian

    Thursday, December 29th, 2011 03:28

    6

    This is very good information, I have bookmarket this site now.

    I will come back another day to read :-)

    0
  • Benjamin

    Tuesday, March 13th, 2012 10:03

    9

    This is such a usefull blog! Can’t stop reading :-) Thanks

    0
  • Split

    Tuesday, February 28th, 2012 16:13

    8

    your website is bookmarked!
    I like the website whichtestwon to get new ideas for AB-Testing

    0
  • Leasy

    Monday, February 13th, 2012 14:20

    7

    Hi Daniels

    I used your guide for my A/B testing for 2 landing pages, it really helped me improve my site, found new ways, ways I didn’t even realize until now. Thanks :)

    0
  • Benjamin

    Tuesday, March 13th, 2012 10:03

    9

    This is such a usefull blog! Can’t stop reading :-) Thanks

    0
  • Split

    Tuesday, February 28th, 2012 16:13

    8

    your website is bookmarked!
    I like the website whichtestwon to get new ideas for AB-Testing

    0
  • Leasy

    Monday, February 13th, 2012 14:20

    7

    Hi Daniels

    I used your guide for my A/B testing for 2 landing pages, it really helped me improve my site, found new ways, ways I didn’t even realize until now. Thanks :)

    0
  • Christian

    Thursday, December 29th, 2011 03:28

    6

    This is very good information, I have bookmarket this site now.

    I will come back another day to read :-)

    0
  • Lars Bo Thomsen

    Friday, September 9th, 2011 09:41

    5

    #John

    Just look in GWO, when it has declared a winner, you have enough data.

    0
  • Værktøj

    Thursday, September 1st, 2011 00:17

    4

    One of the best blog post in a long time, thanks

    Many regards here from Denmark

    0
  • Allan Helleskov Kleiner

    Tuesday, August 30th, 2011 08:25

    3

    Thanks for the brief overview and all the great links. Hopefully, I’ll get some devine inspiration and improve my testing.

    0
  • aditya menon

    Tuesday, August 16th, 2011 18:52

    2

    Thank you for taking time to clear up a jargon bomb! Have a nice day :)

    0
  • John

    Tuesday, August 16th, 2011 03:41

    1

    Great collection of useful information.
    I just started with Google Website Optimizer because it is the easiest tool to use.

    In your post you said: “A solid A/B split test might even take a month.”
    What is the best amount of visitors to test an A/B split testing before you know when A is better then B (and it is not just luck) ?

    0

Comments are closed.

54.226.213.228 - unknown - unknown - US