And why does it matter?
Imagine that a new drug is being tested to see if it is effective at treating a certain disease.
The study has two groups:
After the study, the researchers compare the results from the two groups. 33 of the 200 placebo participants got sick, while 11 of the 200 in the drug group got sick.
They perform a test of significance and find that the probability of this difference between the two groups occurring by chance is 1% or less.
Therefore, the results are statistically significant.
Imagine a marketing agency that has designed two different logos for a client's new product. They want to determine which branding is more likely to be popular amongst the product's target market.
To confirm this, the agency runs a preference survey on goahead by targeting a precise audience of 500 participants.
The results show that:
Visual Identity B is the winning design, but is the result statistically significant? Yes!
The agency can be 95% confident that Visual Identity B is the most appreciated with this audience.
Asking a handful of people around the office for their preference of one option over the other option is pretty common.
However, such a small sample of people will yield very little certainty that the few people's choice actually reflects the choice the target audience would have made.
By asking a large enough group of people to give their opinion about the option, it’s possible to get a much better idea of what the target audience thinks and mathematically possible to test whether one option was statistically preferred over the other.
This is likely something that is not just true within the sample but also reflects the larger population of the target audience.
