Comparing Clinical Significance & Statistical Significance - Similarities & Differences
MHA Search
For the 2023-2024 academic year, we have 112 schools in our MHAOnline.com database and those that advertise with us are labeled “sponsor”. When you click on a sponsoring school or program, or fill out a form to request information from a sponsoring school, we may earn a commission. View our advertising disclosure for more details.
In extremely broad terms, statistical significance means that it’s likely that something is happening, while clinical significance verifies to what extent that thing is happening. Put another way: statistical significance seeks to disprove a negative, and say an event probably didn’t happen by chance; clinical significance seeks to prove a positive, and say an event did happen in a particular, measured manner.
Unfortunately, it’s also a lot more complicated than that, and this goes beyond simple semantics: a crisis on the non-replicability of scientific studies in the last decade has caused vociferous arguing about what these terms mean, and how they’re decided, and how much importance they should convey.
There’s an informal fallacy that can help illustrate aspects of that crisis, and it’s called the Texan Sharpshooter Fallacy. In this thought exercise, a Texan starts shooting at the side of a barn. When he’s run out of bullets, he walks over to the barn and paints a giant target sign, making sure to draw the center of the target around the area where the most bullets landed. Hey look, he says, I’m a sharpshooter! From a statistically significant point of view, he’s right: it couldn’t possibly have happened by chance that all those shots landed so close to the bullseye.
Now, in the academic world, that type of misrepresentation is far less exaggerated, far less common, and practically never intentional. Still, certain forces have implicitly encouraged small errors in data collection and put a priority on the wrong kinds of significance.
To put it succinctly, some academic journals and studies placed far too much importance on statistical significance, which, it turns out, sometimes isn’t all that significant. If this sounds confusing, don’t worry; it’s confusing to some extremely smart people, too.
To learn the differences, similarities, and applications of clinical and statistical significance, check out our side-by-side comparison chart below.
| Clinical Significance | Statistical Significance | |
|---|---|---|
|
Definition |
In medical terms, clinical significance (also known as practical significance) is assigned to a result where a course of treatment has had genuine and quantifiable effects. | Broadly speaking, statistical significance is assigned to a result when an event is found to be unlikely to have occurred by chance. |
|
Key Terms |
|
|
|
History |
Historically, clinical significance has been applied primarily in pharmaceutical trials and medical research. It also has had major implications in psychology and psychotherapy, where the term clinical significance differs slightly in its criteria and formulation. There’s no simple equation for determining clinical significance, and numerous methods have developed over time for arriving at more accurate results. In the 21st century, the emergence of big data and the field of data science have brought a renewed focus on the concept of clinical significance, which is often referred to as ‘practical significance’ when used outside of a medical research setting. Practical significance now allows researchers to dig deeply into data sets and draw conclusions with a high amount of confidence. |
While the history of statistical significance dates back to the 18th century, it wasn’t until 1925 that British statistician and geneticist Ronald Fisher advanced the idea within statistical hypothetical testing. Fisher put forth the initial threshold of 0.05, or 5 percent, for determining statistical significance, but never intended it to be a standard cutoff point, instead recommending it be adjusted based on contextual characteristics. By the early 2010s, academic journals began to note an alarming overuse of the 0.05 p-value, and an editorial bias towards publishing studies that met that threshold. Such bias led to practices such as ‘p-hacking’ that emphasized p-value over other forms of scientific and statistical reasoning. In 2016, the American Statistical Association issued a statement saying that p-value should never be used as a substitute for scientific rigor, and that it hoped to steer research into a ‘post p<0.05 era’. |
|
Applications |
Clinical significance has key applications in vaccine testing, pharmaceutical testing, and other forms of medical research where the magnitude and specific implications of a particular intervention need to be measured and quantified. But it also has use in non-medical settings, too, where it can provide a more rigorous critique of a data set. | Statistical significance has broad applications wherever one is looking to learn whether something happened by chance, including market research with A/B testing, and opinion research with surveys or polls. It can also be useful in the early stages of pharmaceutical testing to determine whether further research is warranted. |
|
Examples |
A pharmaceutical company is testing the efficacy of a new anti-pain drug with clinical trials. It cultivates a representative participant group, gauges their pre-test pain levels, and then closely monitors their progression, before recording their post-test pain levels. Participants are sorted into tiers of their level of effectiveness, and weighed against a control, with further contextual data taken into account (i.e., side-effects and pre-existing conditions). At the conclusion of the study, researchers will have a more informed sense of the new drug’s effect. | A retail company is considering a new advertising campaign. It shows the new campaign to a sample group, and finds that they spend, on average, more than those who saw the old campaign. If the results are statistically significant, a correlation between the new ad and increased spend is likely, and the company may switch all their marketing material to the new campaign. |
|
Importance |
Clinically significant results are reproducible to a much higher degree than those which are merely statistically significant, and therefore can be used in scenarios with higher stakes, where even a modest margin of error would be too high. Clinical significance is a requirement when performing pharmaceutical testing, and the nuanced results that come with it can produce radical insights. | Statistical significance helps scientists, companies, and other entities understand how strongly the results of an experiment, survey, or poll should influence their decisions. But it’s far from the only factor one should consider when assessing the importance of a particular result. Sample size, contextual characteristics, and more complete research must be taken into account as well. |
|
Resources |
|
|
|
The Bottom Line |
Clinical significance seeks to understand the size and scope of an effect. It’s a critical tool for decision-makers who are dealing with high-stakes pharmaceutical, psychological, and medical research. However, the scientific rigor and reproducibility of this type of data analysis is also attractive to data scientists answering questions in other industries. | Statistical significance seeks to verify that an effect is taking place. It can be a helpful tool for decision-makers when taking into account the results of a particular study. However, it should not be the primary determinant of truth, efficacy, or importance. A stricter focus on scientific rigor and smaller p-values will reduce the chances of false positives and irreproducibility in statistical research. |
