Statistical significance is the probability that the result you saw in a test was a true difference and did not happen by accident, at random. We don't want to spend company resources creating designs that have no positive impact, so it's important to job title email list only make changes when the statistical significance is high. The good news is that you don't need to understand exactly how to calculate statistical significance and p-values; most tools like Optimizely or an online "statistical significance calculator" can do a rough estimate for you. However, you need to job title email list understand what statistical significance means. Neil Patel's Meaning Calculator puts the above heads or tails scenario at a 70% confidence level, assuming you tossed a fair coin and it converted to a heads at a rate of 50 % (15 out of 30).
This means that it is 70% sure that the second piece is unfairly weighted towards the faces instead of the result occurring by chance. The industry standard significance level is 95%, which means the equity of the coin cannot yet be determined job title email list without further reversals. For a company willing to take more risk and act on information that may only be due to random circumstances, a level of 90% is sometimes used. So how many more job title email list tosses would you need to determine the equity of the coin? Sample size calculators like this can help you determine that. The base conversion rate is what normally happens without alteration. In this case, a normal piece has a 50% chance of landing heads.
The fact that we see 57% bud (17/30) means that the minimum detectable effect is 14% (the percentage change between 50 and 57). At a 90% confidence level, we would need at least 440 tosses of this coin versus 440 tosses of a fair coin to job title email list realistically determine if it is unfairly weighted. If the percentage difference you expect to see (14%) changes, the sample size needed will also change. A caveat would be to allow for a few test weeks even if significance is reached immediately, as economic cycles can really change the results over the course of a week. I use this coin toss example because we all know that most coins are fair and seeing just a little more job title email list heads than heads does not necessarily mean the coin is weighted towards the heads. Keep this example in mind as you review your experiences. Many marketers will see a positive improvement in Version B with a 70% confidence level and call it a great success. This is dangerous because if you act on information that is likely due to chance (like the example of the coin toss at a 70% confidence level),