The duration for an Amazon A/B test depends on a variety of factors. Namely:
- The sales volume of your product
- The number of variants in a test
- The size of the difference between variants
- The minimum significance of your test settings
The minimum test duration is 4x(the number of variants), so for a 2 variant A/B test the minimum test duration is 8 days.
After that, we will analyse the statistical significance of the results so far. If you have a large sales volume, it will be easy to hit the minimum significance threshold very quickly, oftentimes tests will end on day 8. This is because we have a larger data set which means more accurate results.
However, products with smaller sales volumes can still reach the minimum significance threshold quickly, so long as the difference between the variants is quite large, and gives very obvious results.
If the sales volume is small and there is no clear winner after 4x(the number of variants) days, Splitly will likely recommend you abort the test and try something else.
If you are very close to achieving the minimum significance threshold we will display an approximate number of days remaining until a winner is successfully found.
For best results, we recommend increasing the minimum test duration to 7 days from the default of 4 that way each variant would be able to run for each day of the week to counteract any possible discrepancies caused by day of the week.
This minimum test duration can be changed under the Advanced Settings of an A/B test.
Lastly, in order to make sure that enough data is being considered for A/B tests, each variant must have a minimum of 10 sales before an A/B test concludes.
TL;DR
By default, Allow 4x (the number of variants) days before we run any analysis. After this, we will give you recommendations of whether to pause/abort the test or continue, with an approximate number of days remaining until we can find a winner.
For more information about statistical significance in A/B testing, please read here.