How to A/B Test Shopify Themes

Do you know what I love about e-commerce the most? You typically have one audience (your buyer) and you are aiming for one result (purchasing). This means that changes to your website should be able to be tracked from the buyer to the result. I like data and I like knowing A+B (could possibly, maybe) = C. Of course, there are always outside factors, like change in audience, holidays and buying cycles that will affect the results of the test, but in general information on the results of changes to your e-commerce site are helpful and can help you improve your site for the better.

In Shopify, the theme powers the look and feel of your site. But it also controls the content for your site through “sections” on each page. If you change a section it remains with that theme. This can be both helpful and frustrating.

There are several use cases in which you’d want to A/B test a different theme in Shopify:

  1. You changed all of the button colors for your whole site and want to test the conversion rate of the new color vs the old color
  2. You changed the order of more than two sections on your front page, testing two themes might be a faster way to test which order works better.
  3. You are testing a new sitewide color scheme
  4. You are testing two sales pages against each other to see which converts better
  5. You are making a minor change across all product pages

Warnings as to why you might not want to test a different theme in Shopify:

  • If you make many changes to create a new theme, a test of new vs old theme might not be the best choice. Why? Because in the end you will know which theme converts better but you will not know WHY. If you make too many changes you will still be in the dark as the why for the increase or decrease in conversion.

Google Optimize

For the purposes of this tutorial, I am using Google Optimize. Google optimize is a great tool for building and running A/B tests. It may not be the best, but it is one of the best options out there. It is free, reliable and works in conjunction with Google’s other analytics products (analytics, search console, and tag manager).
Google Optimize allows users to build all kinds of tests, such as A/B, multivariate, split URL, funnel, ad, and more kinds of tests. For the purpose of this tutorial, we are running a redirect test and sending 50% of the audience to the new theme and 50% of the audience to the old theme.

**This same method could also be duplicated to A/B test 2 different landing pages if you would like. Just skip the final step about hiding the preview bar at the bottom of the page.

How to A/B test two Shopify Themes

Step 1: Duplicate your theme (if you haven’t already)

  1. You can select two different Shopify themes or duplicate the existing theme, then make changes as needed in the duplicate theme. You should add the following CSS:

.shopify-preview-bar { display: none }

The line of code removes the preview text at the bottom insert by Shopify, so users in the variation do not see it.

Step 2: Install Google Optimize.

  1. The good news is that you’ll only have to install this once on your website then you can run all of the experiments that you wish.
    Make sure you install this under your Shopify admin area under Preferences>Google analytics> additional JS so that the code is injected into both of the themes that you are going to test. You’ll also need to adapt your Google Tag Manager code to add in the optimize ID. A full article on how to install Google Optimize is located here.

Step 3: Find the web address of the theme you want to test:

  1. Find your Preview Theme ID for the theme that is not currently live.
  2. Preview the new theme from within the Shopify admin, you’ll see the theme id in the URL of the theme.
  3. Then, what we need to do is append ?preview_theme_id=65568381 as a variation for a URL test so that the new design loads for 50 percent of people who visit the store.
  4. The resulting URL will be something like this

Step 4:Set up your Google Optimize Experiment:

  1. Creating your first A/B test with Google Optimize is really easy. Click the Create Experiment button in the top-right corner.

    Google Optimize for Shopify
    Google Optimize for Shopify
  2. Next, select the Redirect test option from the right-hand bar to get started. Create the variation for your A/B test using your current URL and the preview URL ( of your other theme as the pages.
  3. Link your experiment to your analytics.
  4. Define the Google Analytics “Goals” that will be the criteria for evaluation. A “Goal” can be users purchasing an item, bounce rate, or length of time on site. You can set up to three goals for your experiment.
  5. Now, technically, we could go live with this experiment by clicking the Start Experiment button, you can set a start and end date.  In general, I set experiments to run for two weeks to get a good sample of visitors.

Step 5: Define your audience targeting:

  1. I would recommend that you customize your targeting first, to ensure that your experiment is reaching the desired audience. You can define the percentage of visitors who are subjected to this experiment, and choose which segment of visitors to target. (For example, you could target only returning users or those from a specific country. This will depend on the problem users are facing on your website — where, for example, maybe only a certain demographic of users are experiencing said problem.)
  2. Customizing the targeting can be extremely useful when testing a highly experimental change that may have a negative effect on your website. Rather than risk losing traffic and/or revenue, you could show the variation to only 25% of your visitors, and the remaining 75% will see the usual variation. To do this, select the total percentage of visitors you wish to be included in your experiment in the Who slider on the left, and then split what percentage of those users go to which variation by clicking the Edit button on the right.

Step 5: Test your theme redirect:

If you find that the simple CSS addition does not hide the preview bar at the bottom of your alternate theme, you can hide this by adding the following code to your .js file.