> ## Knowledge Base Index
> Fetch the complete knowledge base index at: https://supr-bundles-and-subscriptions.crisp.help/sitemap.xml
> Use this file to discover available pages before exploring further.
> Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash).

# A/B tests

An A/B test shows your shoppers two to four versions of one offer and measures which one earns more. Try a bigger discount against a smaller one, a different set of bundle options, or new titles and badges, then keep the version that wins.

| To find A/B tests, go to [**Supr Bundles & Subscriptions > Grow**](https://admin.shopify.com/apps/super-subscriptions/app/grow) and open [**A/B tests**](https://admin.shopify.com/apps/super-subscriptions/app/grow/ab-tests).

### Table of contents

* [How an A/B test works](#3-how-an-ab-test-works)
* [Creating a test](#3-creating-a-test)
* [Editing the variants](#3-editing-the-variants)
* [Starting the test and previewing each variant](#3-starting-the-test-and-previewing-each-variant)
* [Reading the results](#3-reading-the-results)
* [Ending the test and keeping the winner](#3-ending-the-test-and-keeping-the-winner)
* [What is not measured yet](#3-what-is-not-measured-yet)
* [Common questions](#3-common-questions)
* [Related guides](#3-related-guides)

### How an A/B test works

A test starts from one of your offers. That offer, as it is now, is **variant A**. Each other variant (**B**, **C** and **D**) is a copy of it that you change in **Offers**, so a variant can have different discounts, subscription options, bundle options, gifts, add-ons, texts and badges.

Once the test runs, each shopper is shown one variant and keeps seeing that same variant on every visit from the same browser. Each variant prices its own options at checkout, exactly like any other offer. The app counts how many shoppers saw each variant, how many added it to their cart, and what they bought, and tells you which variant earns the most per visitor.

These stay the same in every variant, because they decide who sees the offer rather than what it offers:

* the products the offer is on,
* **Markets**, **Customers** and **Schedule**,
* the design the offer draws with.

They follow variant A: change them on variant A and save, and every other variant follows.

### Creating a test

1. Go to **Grow > A/B tests** and click **Create A/B test**.
2. Under **Offer to test**, choose the offer. Only published offers that are not already in a test are listed.
3. Give the test a name, or keep the one suggested.
4. Under **Variants**, choose **2 (A and B)**, **3 (A to C)** or **4 (A to D)**.
5. Under **Traffic split**, choose **Even** to give every variant the same share of shoppers, or **Custom** to choose the share that sees the other variants (from 5% to 95%). With **Custom**, variant A gets the rest, and the other variants share their part equally. The line under it shows the split, for example A 70%, B 15%, C 15%.
6. Click **Create test**.

The test is saved as a draft. Its variants B and on are created in **Offers** as drafts with the same name as the offer, because the name is the title shoppers see. In **Offers**, each variant has an **A/B test** badge with its letter, for example **A/B test: B**. Nothing changes on your store until you start the test.

### Editing the variants

On the test's page, click **Edit** next to a variant to open it in **Offers**. A line at the top says which test and variant it belongs to.

While a test is a draft or running, the **Preview** in each variant's editor has buttons for every variant: **A**, **B** and so on. Click one to open that variant, with its own preview, so you can move between the variants and compare them as you edit. Save your changes first: with unsaved changes, the editor stays where it is until you save or discard them. **A/B test** next to them opens the test's page.

Change what you want to test, then save. For a clear result, change one thing per variant, for example only the discount, so you know what made the difference.

On variants B and on, the products are greyed out, **Markets**, **Customers** and **Schedule** are not shown, and the design reads **Design follows variant A**. Change them on variant A.

### Starting the test and previewing each variant

When every variant is ready, click **Start test** on the test's page. Every variant is published and shoppers start being split between them. While the test runs, a variant cannot be set to draft or deleted in **Offers**.

Once it runs, click **Preview** next to a variant to open the product page showing that variant, in a new tab. The preview applies to your own browser only, so you can check each variant before shoppers buy it. Place a test order in each variant if you want to check checkout too.

You can rename the test or change the traffic split at any time while it runs. Under **Settings**, make the change and click **Save**. Changing the split moves some shoppers who already saw the offer to another variant, so settle on it before you start if you can.

### Reading the results

The **Results** on the test's page cover the whole test by default. Use the date button to look at part of it.

The line at the top says how the test stands:

* When a variant clearly earns more per visitor, it names that variant, how much more it earns than the next best, and says you can be 95% confident the difference comes from your change and not from chance. That variant gets a **Significant winner** badge.
* Until then, it says to keep the test running, and roughly how many more visitors each variant needs before the result is reliable.

**Revenue per visitor decides the winner.** It weighs both how many shoppers buy and how much they spend, so a bigger discount that sells more but earns less does not win by accident.

The cards, the chart (one line per variant, pick the metric above it) and the variants table use these numbers:

* **Visitors**: shoppers who saw the variant, each counted once a day.
* **Views**: times a page showed the variant, reloads included.
* **Add to cart** and **Add to cart rate**: times shoppers added the variant to their cart, and that out of its visitors.
* **Orders** and **Conversion rate**: orders with a product bought through the variant, and that out of its visitors.
* **Revenue**: what the products bought through the variant sold for, after discounts.
* **Added revenue**: what bundles sold for beyond one unit at full price. A $18 bundle of a $10 product adds $8.
* **Average order value**: revenue divided by orders.
* **Revenue per visitor**: revenue divided by visitors.
* **Bundle orders**, **Subscribed orders** and **Subscription rate**: orders with a bundle, orders with a subscription, and the share of orders with a subscription.
* **Units per order**: units bought through the variant divided by orders.
* **Chance to beat A**: for variants B and on, how likely it is that the variant really earns more per visitor than variant A.

The **Options** table under it breaks each variant down by option: how often each subscription option, bundle option and the one-time purchase was chosen, added to the cart, ordered and what it earned.

A subscription's first order counts toward its variant. Later renewals do not, so a variant that sells more subscriptions can be worth more over time than its numbers here show.

### Ending the test and keeping the winner

Click **End test**. The window lists every variant with its visitors, conversion rate and revenue per visitor, and picks the significant winner if there is one. Choose the variant to keep and click **End and keep** with its letter.

* The winning variant stays published and is shown to every shopper.
* The other variants go back to drafts in **Offers**, where you can delete them or reuse them later.
* The results stay on the test's page, and each variant's numbers stay in **Analytics > Offers**, where each variant is an offer with its own performance page.
* Click **View** above the variants on the test's page to see every variant's widget side by side, as it was when the test ended, even if you later edit or delete a losing variant. Before the test ends, **View** shows the variants as they are now.

A test does not end on its own and never picks a winner for you.

To delete a test, open it and click **Delete**. Deleting a draft test also deletes its draft variants B and on; variant A's offer stays as it is. Deleting an ended test keeps every variant's offer. End a running test before you delete it.

### What is not measured yet

* **Checkout rate**: how many shoppers reached checkout.
* **Profit**: revenue minus your product costs.
* **Refunds and cancelled orders**: an order counts when it is placed, even if it is refunded later.

### Common questions

**How long should I run a test?**
Until the results line names a significant winner. With little traffic that can take weeks. Ending earlier is your call, but the numbers may still be down to chance.

**Will a shopper see a different variant tomorrow?**
No. A shopper keeps their variant on every visit from the same browser. On another device or after clearing their browser data, they may get another one.

**Can I test two different offers against each other?**
A test always starts from one offer and its copies. To compare two existing offers, create a test from one of them and set up variant B like the other.

**Can I test widget designs?**
Not yet. Every variant draws with variant A's design.

**Does an A/B test change how checkout prices work?**
No. Each variant prices its own subscription and bundle options, like any offer. A shopper only ever sees one variant, so they pay that variant's price.

### Related guides

* [A Beginner's Guide to Subscription Offers](https://supr-bundles-and-subscriptions.crisp.help/en-us/article/a-beginners-guide-to-subscription-offers-1ey00x7/)
* [Bundle and Subscription Analytics](https://supr-bundles-and-subscriptions.crisp.help/en-us/article/subscription-analytics-1fn3ntr/)
* [Bundle types: quantity breaks, Buy X Get Y, fixed bundles, and build-a-box](https://supr-bundles-and-subscriptions.crisp.help/en-us/article/bundle-types-quantity-breaks-buy-x-get-y-fixed-bundles-and-build-a-box-1p9zux1/)