Ultimate Guide to A/B Split Testing – Articles, Tips, Case Studies, Tools and Resources


First of all, what is A/B split testing? In basic terms A/B split testing is a variety of marketing testing where customers randomly receive almost identical, yet slightly different, test samples. This method can efficiently be used in various niches such as brochures, email campaigns and landing pages. A/B split testing can help you to improve conversion rates, better understand visitor behavior, test out new products and much more. Email newsletter services like Mailchimp have already implemented A/B split testing in their feature range.

More and more people are becoming aware of A/B split testing and not without reason. Case studies and examples show how greatly testing can improve a site’s performance, usability and conversions. What’s more, it costs you comparatively nothing. There are plenty of useful tools already made like Google Website Optimizer which cost you nothing. Continue reading to find comprehensive guides on A/B split testing, exciting case studies, useful tools and resources.

A Few Quick Tips To Get You Started

Know what to test

The goal of A/B split testing is to improve your website’s performance, not to build a whole website. Find out what exactly you want to test before testing it. Help out with case studies and have a closer look at your own website. If you’re not sure start with a bit. Some of the most popular things to test are call to action buttons, headlines, forms, text length and layout.

Give it time

Don’t jump to any conclusions after you’ve run the test for a couple of days. A solid A/B split test might even take a month. Giving up too early on a test will only cost you time and the results will be marginal. Once you start seeing a solid path where the results are going you can close the test or start a new one. There are also calculators available to test results confidence if you’re unsure.

Analyze results

You’ve run your first A/B test, great, now what? Don’t take it for granted that the test version is always a better solution. Examine your results carefully and see where the flaws were and what you could improve even more. One test often won’t be enough. You’ll have to do several or even a multivariate one. Once you’ve done a decent amount of tests you’ll see where the actual results are going.

Mix it up

Like you just found out – one test won’t be enough. Be creative with your tests. Don’t stick to the ordinary A/B option, 50/50 audience. Create various templates and vary the proportions. Change something else in each test. At the end you’ll have rich and valuable material to analyze. However don’t get too fancy. Don’t torture your regular visitors with too many testing. Address tests to your new visitors then see how they adapt. To sum up, follow the guidelines but don’t take them as a must. Be creative with some limitations though.

Case study

Case study from Visual Website Optimizer blog. Read the full article here.

In the screenshot above you can see Basekit’s original pricing page (on the left) and the redesign (on the right) which increased conversations by 25%. The goal was to increase the number of people who visit the Buy Now page after visiting their Plans and Pricing page.

As you can see the redesigned page is more organized, smoother, cleaner, brighter and more user friendly. This design showed a consistent 25% improvement in conversations for the entire duration of the test.

Even though they accomplished their goal with this redesign, they’re already working on a new redesign to do another split test. As we already discussed – the key to success lots of tests.

This A/B test showed the importance of pricing table design. Remember that pricing table article we did recently? It’s a great example of efficient A/B testing in action.

Articles & Tutorials

1. The Ultimate Guide To A/B Testing


This article by Paras Chopra, the founder of Visual Website Optimizer, is meant to be the best guide you will ever need for A/B testing.

2. Split Testing Landing Pages to Boost Conversions


Running split tests on campaign landing pages is an important way to test and optimise conversions. Learn how to use split tests and optimize landing pages.

3. Using Google Analytics to get insights from A/B testing


In this article Felipe Wesbonk from Traffic Builders explains the integration from VWO (Visual Website Optimizer) in Google Analytics and the way you can use this plugin in making custom reports.

4. How to Do A/B Split Testing on Your Mailing List


Learn how to perform split testing in your mailing campaigns, check out case study and improve your results.

5. A Crash Course on A-B Split Testing


Another article giving you an insight in AB testing advantages and disadvantages.

6. The Pitfalls Of A/B Ad Split Testing, Part 2


Article digging into ad testing and examining some pitfalls of AB ad split testing

7. A/B Split Testing vs. Multivariate: Pros & Cons.


Which of the two primary testing methods should you employ? Find out pros and cons of each test, when to use them together and why.

8. Gathering Variables for A/B Split Testing


This article will take a look at elements of a website that should affect its users, teach you what to test and where to gain ideas.

9. Split Testing


Very detailed and comprehensive article about split testing basics, showing how these tests work, case study and tools to do the job.

10. Fundamentals of A/B and Multivariate Testing


Look at the fundamentals of both A/B and multivariate testing and learn how to create a split test step by step.

Case studies

1. Ecommerce website? Bring price and add-to-cart button closer to increase sales


Insight into Taylor Gifts A/B test they recently did to improve add-to-cart clickthrough.

2. You should follow me on twitter


Dustin Curtis outlines his experiment with Twitter links and shows the power language has on clickthrough rates.

3. Split testing works. A very short case study from AgileTask.


Short case study showing how call to action buttons can significally improve user reactions.

4. Writing Decisions: Headline tests on the Highrise signup page


37signals was experimenting and rotated some headlines and subheads on the Highrise signup page to see if headlines have an effect on signups. Want to know the answer?

5. Split Testing: How To Increase Your Adsense Earnings 94% Overnight


In this post Brian Armstrong from StartBreakingFree.com shares some tips on using Split Testing to increase his AdSense earnings.

6. Skype Boosts Homepage Conversion Rate with Radical Simplicity


Skype designed two new homepages to draw user focus to the call to action. See which was the most succesful option.

7. Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement


This post will show exactly how one start-up improved their homepage conversion rate (visitor to sign-up flow) more than 20%, then 16% again, with a few simple changes and Google Website Optimizer.

8. A/B Test Case Study: Can Split Test Results Be Trusted?


Little A/B experiment done for the Official Vancouver 2010 Olympic Store to optimize conversion.

9. How to Double Your Subscriber Growth With 10 Minutes Work


See some split testing examples, learn how to do one yourself and find out which web elements to test and why.

10. A/B testing browser extensions: do annoying notifications cause uninstallations?


Most people refer to web pages when they talk about A/B testing. However, A/B testing can be useful in other domains too. At InvisibleHand, Visual Website Optimiser was used to test the performance of different versions of a browser extension.

Tools & Resources

1. Google Website Optimizer


Google Website Optimizer is the easiest and cheapest tool to do A/B and multivariate tests.

2. Visual Website Optimizer


Visual Website Optimizer is an easy to use A/B testing tool featuring point-and-click test designer and WYSIWYG editor for creating variations. If you aren’t sure if Google Website Optimizer will work for you this is definitely the second best choice.

3. VWO Free Testing Tools


Visual Website Optimizer also offers four great free tools split testing. These include landing page analyzer, test significance calculator, test duration calculator and A/B Ideafox where you can gain ideas for various A/B tests.

4. genitify


genitfy enables webpage authors to accelerate their development cycles. It is like A/B testing but it goes beyond it.

5. A/B Theme Testing Plugin (premium)


This WordPress plugin rotates themes to assign them evenly between visitors.

6. Vanity


Vanity is an experiment driven development framework for Rails. It will help you to create and run A/B tests though it requires some heavy knowledge.

7. phpScenario


phpScenario is an object-oriented split testing library written in PHP. It’s designed to do simple A/B testing with as little code as possible, but should also be as powerful as you need it to be.

8. ShrimpTest


ShrimpTest is an A/B testing solution for WordPress under development at Automattic.

9. WhichTestWon


A/B split testing showcase with nearly 150 case studies. Yet you can only view one per week for free.

10. ABTests


Showcase of various split tests with screenshots, results and comments.



  1. Leasy

    Hi Daniels

    I used your guide for my A/B testing for 2 landing pages, it really helped me improve my site, found new ways, ways I didn’t even realize until now. Thanks :)

  2. Christian

    This is very good information, I have bookmarket this site now.

    I will come back another day to read :-)

  3. John

    Great collection of useful information.
    I just started with Google Website Optimizer because it is the easiest tool to use.

    In your post you said: “A solid A/B split test might even take a month.”
    What is the best amount of visitors to test an A/B split testing before you know when A is better then B (and it is not just luck) ?