TotallyMoney have produced a free split testing tool for A/B/n tests which allows you to find the statistical significance of your results and visualise their uncertainty.
Step by step guide to analysing the results of your A/B/n test:
- Make a copy of our tool from Google Sheets (https://docs.google.com/spreadsheets/d/1jiDtQhxRGRBaTCgSp8D5DaOhSX_AzQ2x-zqABQKsyIE/edit?usp=sharing)
- Input the number of visitors and conversions (or impressions and clicks etc.) for your control and one or more variations
- The variation with the highest conversion rate will be returned as the winner, with its confidence level and statistical significance, as well as whether it is over 95% or 99% certain that your results were not down to chance
- Bear in mind that when conducting an A/B/n test, the question we are asking is ‘can we reject the hypothesis that the different variations have the same conversion rate as the control, given the observed conversion rates?’
- If you would like to visualise the probability distributions of the winning variation and control (or the runner up if the control wins), then click the ‘Update Axes’ button on the graph and follow the instructions at the bottom of the sheet to activate the script