During an A/B test, Splitly will rotate through your variants automatically, testing each one to see how each one performs. Splitly works a little bit differently to most traditional A/B testing softwares. Usually a variant is shown to only one visitor by placing a cookie in the visitors browser. However, since we are unable to place any tracking codes inside your product listings it is not possible for us to do this. This means there are a few things you should be aware of while a test is running.
Don't edit your product listing
Splitly will automatically make changes to your product listing as and when necessary. If you change your product listing to something else during a test, the test will be voided.
Maintain consistent marketing efforts
If you suddenly boost your marketing efforts, this will skew results data. For example, don't suddenly email your email list promoting your product because this will skew the results for a specific variant. It is best to wait until the end of the test before you run these kinds of marketing efforts.
PPC campaigns typically do not effect the quality of results, so long as they are consistent and traffic is evenly distributed through the entire time your test is running.
Manually void test results
If you believe you may have skewed the test results for a specific day, you should void the result for that day. This means that days worth of data will not be counted towards the final calculations, meaning more accurate results. To void a result, under the rotation log just click void.
Install the Chrome extension
Amazon's MWS excludes some critical pieces of information, namely, sessions data. We use our Chrome extension to extract this vital piece of data including sales, conversions and revenue. See here to learn about how Splitly moves data in and out of Amazon.