Statistical significance is a method to produce accurate results. Splitly automatically ends a test when it reaches the minimum statistical significance threshold, or when a certain level of accuracy has been reached.
Broadly speaking, statistical significance is the probability of one variant out or under-performing another. When we have enough data we show the statistical significance values in test results.
We always recommend a minimum significance of 90% - that gives a 10% margin of error. If you decide to abort the test before the significance is satisfied, you should not trust the results.
If you are seeing a small statistical significance or roughly a 50/50 result - this means your variants are too similar and the test will take a very long time to complete before considered accurate. It would be best to end the test there and try something else, such as a bigger change in your variants.
Learn more about statistical significance in Amazon A/B tests here.