A/B Testing (Experiments)

Note: Who can use this feature? This feature is available on the Advanced and Expert plans. If you're interested in taking advantage of it, please reach out to your customer success manager.

A/B testing allows you to test and compare up to five different variations on a single merchandising campaign, each of which you can change using Athos Commerce's Visual Merchandising tools: Boost rules, banners, pinned products, removed products, and custom filter order.

A/B Testing is available on any campaign type, so you can test across your entire site or on a specific page.

Note: A/B testing is NOT available for recommendations.

For example, you could run an experiment with these variations:

• Boost rules and a banner that emphasize a brand
• Boost rules and a banner that emphasize a different brand
• No custom boost rules or banners, as a control group

How it works

Each A/B Testing experiment will segment users into random groups based on how many variations you've set up and the percentage of users desired in each. A user will always be in the segment they're initially assigned, so reloading the page or coming back another day will result in the same experience for the same shopper.

When you begin a subsequent experiment, even on the same exact campaign, the same user will be randomly re-assigned for the new experiment — therefore, a user will not stay in the same variation number over multiple experiments.

Setting up an experiment

An experiment is an individual test run on a campaign over a specific amount of time. To begin setting up an experiment, go to Merchandising and click a campaign to select it. The Campaign Editor will open; from here, select the Experiments toolbox.

Under the "Start an Experiment" section, click "Create Experiment". 

Note: This button will be grayed out if you already have five A/B tests running.

The basic options for an experiment are shown below; name the experiment and select a date range for the experiment to last.

Under "Variations", select the variation you'd like to set up and configure it. (The variation you're currently editing will be marked in blue.) You can add up to five different variations in a single experiment by clicking "Add another variation". The more variations you add, the longer you'll need to run your experiment to get accurate data sample sizes. 

You can also set up what percentage of shoppers are split into each variation by dragging the sliders under "Customer Placement".

To begin your experiment, simply save the campaign as you normally would. A popup dialog will ask to confirm the beginning of the experiment. You will not be able to make any merchandising changes until the experiment is complete. 

Running an experiment

While the experiment is running, no merchandising tools will be editable. However, you may still edit the campaign's title, schedule, experiment end date, experiment default winner, and end the experiment manually.

While an experiment is running, you can still use the experiments editor to change the end date. 

You can also view the setup of the running variations by clicking one of them, and you can end the experiment immediately. 

Completing an experiment

Ending the experiment manually

To end an experiment immediately, you can click on "End Experiment Now". Here you will need to commit to a winner, which will be used moving forward. Once a campaign has reached its end date, the editor will prompt you to select the winner. Once a winner has been declared, save your campaign to complete the experiment. 

Trying it out

You can easily simulate different shoppers on your live site by clearing your cookies for the site. Once you've cleared your cookies, simply refresh the page and you will be re-sorted into a group for the experiment. There is a chance that you'll be sorted into the same variation group again, so you may need to do so multiple times. 

Reporting

You can view the results of your A/B test directly in the console with the Experiments report.

Glossary

• A/B testing =  The name of this feature.
• A/B test =  A common synonym for "Experiment" or "A/B Merchandising"
• Experiment =  An individual test run on a campaign over a specific amount of time. Contains 2-5 variations in a single experiment.
• Variation =  An individual set of elements. Shoppers are divided randomly and shown only one variation.
• Element =  Part of the page's behavior that a merchandiser can change. There are 5 elements =  banners, boosts, pinned products, removed products, and custom filter order.
• Editor =  The panel available on the left side of the editor. Each toolbox within the editor may contain multiple tabs to view/manage other content.
• Context =  The conditions which must be met in order for the campaign to take effect.