There is a rumour in the seller community that there is a penalty each time you update your listing, in that you will see a drop in sessions or ranking after updating a listing.
Now that Splitly has been live and kicking for some months, we have a lot of data from our customers tests, which we can analyse to see if there's any truth to this claim.
We compared the total number of sessions before and after tests are started, to see how running a split test might affect them.
The results are shown below:
This chart shows the relationship between sessions before a test and after a test. As you can see, the number of sessions is relatively unchanged. The small discrepancy lies within a very small error range. The sessions increase with no. of days, as more days are being included. Later they decrease, as less tests can be used to get data.
Overall sessions before is very close to sessions after, with sessions after being slightly improved in parts. We are not sure if this is due to Amazon rewarding listings for being regularly updated, or our customers choosing good variants to test.
However either way, we can be sure that on average split testing is not hurting sellers rankings at all. So long as you comply with Amazons policies - you have next to zero risk of your rankings being hurt.