In the online shopping world, customer experience is everything.
In fact, Statista found that 44.5% of businesses worldwide perceive customer experience to be a primary competitive differentiator. But creating a quality user experience isn’t just about using your intuition to predict what will attract and convert potential customers — you need to have tangible insights into what your target audience actually wants.
Enter A/B testing.
A/B testing is a method of determining which website design, content or functionality is more successful with your site visitors. It allows you to test a variation or element of your page that may affect your consumers’ behavior.
You’ve probably heard about basic examples of A/B testing like changing button colors and call to action copy — but many ecommerce businesses don’t know how to do A/B testing effectively.
However, when done well, A/B testing is a powerful way to improve the most important metrics in your business. If this is your first time using A/B testing, here is your personal roadmap to help you get started.
When executed well, A/B testing (aka split testing) allows you to significantly improve your customers’ shopping experience, leading to better click-through rates, more conversions, greater loyalty and much more.
Not convinced? Take a look at Amazon, which evolved from an ecommerce store that only sold books to the largest online retailer in the world — and much of this success was due to CEO Jeff Bezos’s insistence on constantly testing, experimenting and innovating.
Consider all the ways Amazon has revolutionized the online shopping experience:
But these innovations don’t occur by coincidence — they are the result of rigorous multivariate testing and taking action based on reliable results.
Inevitably, you’re going to make changes to your ecommerce site at some point in the future, whether it be to your messaging, SEO tactics or checkout process. But if you want to make the right changes and convert more of your existing traffic, A/B testing is the best way to uncover insights and guide you in the right direction.
There are two primary types of A/B testing: client-side and server-side, with client-side being the more common.
For example, you may send 50% of your traffic to variation A and 50% of your traffic to variation B.
Then, analyze the performance of each page variation to determine whether variation A or variation B is more effective according to a set objective (conversion), whether that’s sales, email opt-ins, click-through rates, etc.
However, in order to be confident that test results are accurate and not the result of an anomaly, a certain number of visitors — the sample size — must be reached. If this threshold isn’t met, multivariate tests can be inaccurate, leading you to implement changes that can actually hurt your site.
If your site doesn’t generate enough traffic, you may want to consider first investing in paid advertising or SEO to increase your overall traffic numbers. Once you’ve reached the necessary level of traffic, you can implement A/B testing at any point and see a significant return on your investment.
A/B testing is all about removing consumer pain points. Whenever you can remove friction from the shopping process, it significantly increases the chances of conversions.
In the context of your own ecommerce website, here are some common A/B testing ideas:
The beauty of A/B testing is that you can systematically work through each of these areas, incrementally and consistently improving each one to increase conversions.
A/B tests can come in many shapes and sizes. Here are four common A/B test examples you might see as part of an optimization program.
In this test example, the goal is to see whether consumers are able to better navigate a site and reach checkout by exposing the menu at the top level rather than through a drop-down menu. Or, you may test whether it’s easier for customers to find the right product using a “tag & filter” (like the filtering feature on Amazon) or traditional navigation system.
Using this test, you can determine whether placing more educational information about the product higher on the page will affect the add-to-cart rate. Or, you may test various product titles, product images or stock status messages.
This test aims to find whether consumers are more inclined to complete a form with fewer fields and simpler placeholder text within the fields. Or, you may test whether customers are more willing to fill out the form earlier or later in the buying process.
If you want your customer to complete a purchase, your checkout process needs to flow as smoothly as possible — which is why A/B testing is so crucial.
A checkout process test will help determine whether a simple multipage checkout or a single-page checkout will result in better conversions. You may also test merging several small steps within the checkout process or using breadcrumbs to tell your customers users about the next steps of the process.
Before you begin testing, it’s a good idea to conduct thorough research on your target audience and create a roadmap that will guide you specifically through what to test and in what order.
Some research tools include:
After doing a thorough analysis of your customer’s behavior, you can begin to develop hypotheses about your current online store and how you might improve the overall customer experience. These will form the basis of your A/B testing roadmap.
As mentioned, A/B testing is a powerful tool for crafting the ideal customer experience, but that’s not the only perk you’ll come across. Let’s take a look at some other great reasons for implementing A/B testing:
But just like anything, A/B testing also comes with its setbacks. Let’s look at a few of the challenges you may encounter when launching your first A/B test:
Given the challenges and struggles listed above, it’s understandable if you have some reservations. But when executed properly, A/B testing can have a huge impact on some of your most important KPIs:
The more friction you remove from your website, the more paid traffic turns into sales, effectively increasing your return on ad spend.
Similar to how testing helps improve ROAS, the money you spend on other methods of customer acquisition will benefit from the reduced friction and shrink your customer acquisition cost (CAC).
Testing makes your website more engaging, which helps drive more return business. In other words, the easier you make the shopping process, the more customers will want to come back again and again, thus increasing your customer lifetime value.
Your email list can be a huge revenue driver. Testing different ways of inviting new subscribers will help you maximize your email list sign-ups, which increases the number of people you can promote your products and services to.
Increased testing can help you find out what product bundles or upsell opportunities could work best, increasing your average order value. At scale, this directly impacts your bottom line revenue.
Every time a person abandons their shopping cart, that’s money walking out your digital door. Using A/B testing to optimize your checkout process can remove friction, confusion and mistrust and reduce your cart abandonment rate.
Smoothing out any barriers on the path to purchase is the most effective way to increase your conversion rate.
When it comes to A/B testing, experience and skill are just as, if not more important than the tools themselves.
First, you need to determine whether you want to do your testing in-house or hire an outside agency. Here are a few best practices to help you find the best solution for your business:
Regardless of whether you choose an agency or in-house, here are some of the key skills to look for:
Each of these skills contributes uniquely to the CRO process, and the absence of any of them can lead to ineffective A/B testing.
There are a number of specific tools required to do A/B testing, the three most common being Optimizely, VWO and Google Optimize. Each offers a full suite of A/B testing tools, including the ability to easily deploy tests, analyze the results, evaluate website heatmaps and much more.
Optimizely is a feature-rich, enterprise-level A/B testing tool, and is priced accordingly. It is widely considered an industry leader for larger sites.
With a Stats Engine that helps analyze A/B test results, Optimizely lets users run tests with ease and get real-time results so they can take proactive steps to enhance their ecommerce stores.
However, for many small to medium-sized businesses Optimizely can be like using a semi-truck when a pickup will do. We recommend considering whether you will realistically utilize its full suite of features before selecting this tool.
Visual Website Optimizer (or VWO) is an effective A/B testing tool, providing a solid mix of features that will do all of your core A/B testing tasks. It offers a SmartStats analysis tool, a WYSIWYG editor for beginners as well as on-page surveys, form analytics, heatmaps, etc.
Google Optimize, formerly called Google Website Optimizer, is a free A/B testing, website testing and personalization tool for small businesses. It comes with Google Analytics and allows users to see how multiple page variations perform against a specified objective.
Whichever tool you choose, spend time researching the capabilities, pricing and overall level of support for each. Also, if you plan on doing your testing in-house, be sure you feel some level of comfort with the software you choose.
A number of service-based businesses provide A/B testing and CRO (conversion rate optimization) as a “done-for-you” service — but be warned: many marketing agencies might add CRO or A/B testing to their laundry list of services yet have little more than beginner knowledge.
There is a simple way to do A/B testing and a more methodical, professional way to do it. Make sure you’re getting the latter.
Here are some things to look for in A/B testing services:
Bottom line: Do your research and ask the right questions before committing to a CRO agency — your research will pay off.
Needless to say, A/B testing is one of the most powerful and effective ways to drive ecommerce growth. Rather than relying solely on intuition to predict what will convert customers, A/B testing provides tangible insights into what your audience actually wants — so you never have to play the guessing game.
Luckily, with A/B testing and CRO apps like VWO and Shogun Page Builder, BigCommerce merchants have access to a variety of tools to help you increase conversions and make your existing traffic that much more valuable.
CRO is an entire process designed to remove friction from the buying journey, thereby creating a better customer experience, while A/B testing is a method of determining which elements of your website will be more successful with your site visitors.
While some people often mistake A/B testing and CRO as interchangeable, CRO is an entire process that needs to be repeated, while A/B testing is only a component of that process.
Choosing what metrics to measure in your A/B test will largely depend on what your business goals are and how you want to affect customer behavior. However, here are some of the most common metrics used in ecommerce A/B testing:
To help you outline your goals and objectives, check out this A/B test tracking template from Hubspot, which outlines what variables you should test and provides a simple significance calculator to track your results.
Before deciding about how you should run your A/B test, you first need to understand statistical significance.
According to Optimizely, “statistical significance is the likelihood that the difference in conversion rates between a given variation and the baseline is not due to random chance.”
Therefore, in order to get significant results from your A/B test, don’t stop testing until you reach a statistical significance of 95%–99%. This simply means that you are 95%–99% confident that you achieved accurate results. This may take days, weeks or months depending on your website’s traffic levels.
Unless you expect extreme overlaps between tests, it is okay to run multiple A/B tests at the same time.
However, if you do suspect significant overlap, it may be best to either run the tests in parallel, but on different audiences (the audience for the A test will not participate in the B test, and vice versa), or run the tests at different times, not beginning variation B until variation A is complete.