Statistical significance information

In statistical hypothesis testing,^[1]^[2] a result has statistical significance when a result at least as "extreme" would be very infrequent if the null hypothesis were true.^[3] More precisely, a study's defined significance level, denoted by $\alpha$ , is the probability of the study rejecting the null hypothesis, given that the null hypothesis is true;^[4] and the p-value of a result, $p$ , is the probability of obtaining a result at least as extreme, given that the null hypothesis is true.^[5] The result is statistically significant, by the standards of the study, when $p\leq \alpha$ .^[6]^[7]^[8]^[9]^[10]^[11]^[12] The significance level for a study is chosen before data collection, and is typically set to 5%^[13] or much lower—depending on the field of study.^[14]

In any experiment or observation that involves drawing a sample from a population, there is always the possibility that an observed effect would have occurred due to sampling error alone.^[15]^[16] But if the p-value of an observed effect is less than (or equal to) the significance level, an investigator may conclude that the effect reflects the characteristics of the whole population,^[1] thereby rejecting the null hypothesis.^[17]

This technique for testing the statistical significance of results was developed in the early 20th century. The term significance does not imply importance here, and the term statistical significance is not the same as research significance, theoretical significance, or practical significance.^[1]^[2]^[18]^[19] For example, the term clinical significance refers to the practical importance of a treatment effect.^[20]

^ ^a ^b ^c Sirkin, R. Mark (2005). "Two-sample t tests". Statistics for the Social Sciences (3rd ed.). Thousand Oaks, CA: SAGE Publications, Inc. pp. 271–316. ISBN 978-1-4129-0546-6.
^ ^a ^b Borror, Connie M. (2009). "Statistical decision making". The Certified Quality Engineer Handbook (3rd ed.). Milwaukee, WI: ASQ Quality Press. pp. 418–472. ISBN 978-0-87389-745-7.
^ Cite error: The named reference Myers et al-p65 was invoked but never defined (see the help page).
^ Dalgaard, Peter (2008). "Power and the computation of sample size". Introductory Statistics with R. Statistics and Computing. New York: Springer. pp. 155–56. doi:10.1007/978-0-387-79054-1_9. ISBN 978-0-387-79053-4.
^ "Statistical Hypothesis Testing". www.dartmouth.edu. Archived from the original on 2020-08-02. Retrieved 2019-11-11.
^ Johnson, Valen E. (October 9, 2013). "Revised standards for statistical evidence". Proceedings of the National Academy of Sciences. 110 (48): 19313–19317. Bibcode:2013PNAS..11019313J. doi:10.1073/pnas.1313476110. PMC 3845140. PMID 24218581.
^ Redmond, Carol; Colton, Theodore (2001). "Clinical significance versus statistical significance". Biostatistics in Clinical Trials. Wiley Reference Series in Biostatistics (3rd ed.). West Sussex, United Kingdom: John Wiley & Sons Ltd. pp. 35–36. ISBN 978-0-471-82211-0.
^ Cumming, Geoff (2012). Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. New York, USA: Routledge. pp. 27–28.
^ Krzywinski, Martin; Altman, Naomi (30 October 2013). "Points of significance: Significance, P values and t-tests". Nature Methods. 10 (11): 1041–1042. doi:10.1038/nmeth.2698. PMID 24344377.
^ Sham, Pak C.; Purcell, Shaun M (17 April 2014). "Statistical power and significance testing in large-scale genetic studies". Nature Reviews Genetics. 15 (5): 335–346. doi:10.1038/nrg3706. PMID 24739678. S2CID 10961123.
^ Altman, Douglas G. (1999). Practical Statistics for Medical Research. New York, USA: Chapman & Hall/CRC. pp. 167. ISBN 978-0-412-27630-9.
^ Devore, Jay L. (2011). Probability and Statistics for Engineering and the Sciences (8th ed.). Boston, MA: Cengage Learning. pp. 300–344. ISBN 978-0-538-73352-6.
^ Craparo, Robert M. (2007). "Significance level". In Salkind, Neil J. (ed.). Encyclopedia of Measurement and Statistics. Vol. 3. Thousand Oaks, CA: SAGE Publications. pp. 889–891. ISBN 978-1-4129-1611-0.
^ Sproull, Natalie L. (2002). "Hypothesis testing". Handbook of Research Methods: A Guide for Practitioners and Students in the Social Science (2nd ed.). Lanham, MD: Scarecrow Press, Inc. pp. 49–64. ISBN 978-0-8108-4486-5.
^ Babbie, Earl R. (2013). "The logic of sampling". The Practice of Social Research (13th ed.). Belmont, CA: Cengage Learning. pp. 185–226. ISBN 978-1-133-04979-1.
^ Faherty, Vincent (2008). "Probability and statistical significance". Compassionate Statistics: Applied Quantitative Analysis for Social Services (With exercises and instructions in SPSS) (1st ed.). Thousand Oaks, CA: SAGE Publications, Inc. pp. 127–138. ISBN 978-1-4129-3982-9.
^ McKillup, Steve (2006). "Probability helps you make a decision about your results". Statistics Explained: An Introductory Guide for Life Scientists (1st ed.). Cambridge, United Kingdom: Cambridge University Press. pp. 44–56. ISBN 978-0-521-54316-3.
^ Myers, Jerome L.; Well, Arnold D.; Lorch, Robert F. Jr. (2010). "The t distribution and its applications". Research Design and Statistical Analysis (3rd ed.). New York, NY: Routledge. pp. 124–153. ISBN 978-0-8058-6431-1.
^ Hooper, Peter. "What is P-value?" (PDF). University of Alberta, Department of Mathematical and Statistical Sciences. Archived from the original (PDF) on March 31, 2020. Retrieved November 10, 2019.
^ Leung, W.-C. (2001-03-01). "Balancing statistical and clinical significance in evaluating treatment effects". Postgraduate Medical Journal. 77 (905): 201–204. doi:10.1136/pmj.77.905.201. ISSN 0032-5473. PMC 1741942. PMID 11222834.

[Sirkin-1] Sirkin, R. Mark (2005). "Two-sample t tests". Statistics for the Social Sciences (3rd ed.). Thousand Oaks, CA: SAGE Publications, Inc. pp. 271–316. ISBN 978-1-4129-0546-6.

[Borror-2] Borror, Connie M. (2009). "Statistical decision making". The Certified Quality Engineer Handbook (3rd ed.). Milwaukee, WI: ASQ Quality Press. pp. 418–472. ISBN 978-0-87389-745-7.

[Myers_et_al-p65-3] Cite error: The named reference Myers et al-p65 was invoked but never defined (see the help page).

[Dalgaard-4] Dalgaard, Peter (2008). "Power and the computation of sample size". Introductory Statistics with R. Statistics and Computing. New York: Springer. pp. 155–56. doi:10.1007/978-0-387-79054-1_9. ISBN 978-0-387-79053-4.

[:0-5] "Statistical Hypothesis Testing". www.dartmouth.edu. Archived from the original on 2020-08-02. Retrieved 2019-11-11.

[Johnson-6] Johnson, Valen E. (October 9, 2013). "Revised standards for statistical evidence". Proceedings of the National Academy of Sciences. 110 (48): 19313–19317. Bibcode:2013PNAS..11019313J. doi:10.1073/pnas.1313476110. PMC 3845140. PMID 24218581.

[Redmond_and_Colton-7] Redmond, Carol; Colton, Theodore (2001). "Clinical significance versus statistical significance". Biostatistics in Clinical Trials. Wiley Reference Series in Biostatistics (3rd ed.). West Sussex, United Kingdom: John Wiley & Sons Ltd. pp. 35–36. ISBN 978-0-471-82211-0.

[Cumming-p27-8] Cumming, Geoff (2012). Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. New York, USA: Routledge. pp. 27–28.

[Krzywinski_and_Altman-9] Krzywinski, Martin; Altman, Naomi (30 October 2013). "Points of significance: Significance, P values and t-tests". Nature Methods. 10 (11): 1041–1042. doi:10.1038/nmeth.2698. PMID 24344377.

[Sham_and_Purcell-10] Sham, Pak C.; Purcell, Shaun M (17 April 2014). "Statistical power and significance testing in large-scale genetic studies". Nature Reviews Genetics. 15 (5): 335–346. doi:10.1038/nrg3706. PMID 24739678. S2CID 10961123.

[Altman-11] Altman, Douglas G. (1999). Practical Statistics for Medical Research. New York, USA: Chapman & Hall/CRC. pp. 167. ISBN 978-0-412-27630-9.

[Devore-12] Devore, Jay L. (2011). Probability and Statistics for Engineering and the Sciences (8th ed.). Boston, MA: Cengage Learning. pp. 300–344. ISBN 978-0-538-73352-6.

[Salkind-13] Craparo, Robert M. (2007). "Significance level". In Salkind, Neil J. (ed.). Encyclopedia of Measurement and Statistics. Vol. 3. Thousand Oaks, CA: SAGE Publications. pp. 889–891. ISBN 978-1-4129-1611-0.

[Sproull-14] Sproull, Natalie L. (2002). "Hypothesis testing". Handbook of Research Methods: A Guide for Practitioners and Students in the Social Science (2nd ed.). Lanham, MD: Scarecrow Press, Inc. pp. 49–64. ISBN 978-0-8108-4486-5.

[Babbie2-15] Babbie, Earl R. (2013). "The logic of sampling". The Practice of Social Research (13th ed.). Belmont, CA: Cengage Learning. pp. 185–226. ISBN 978-1-133-04979-1.

[Faherty-16] Faherty, Vincent (2008). "Probability and statistical significance". Compassionate Statistics: Applied Quantitative Analysis for Social Services (With exercises and instructions in SPSS) (1st ed.). Thousand Oaks, CA: SAGE Publications, Inc. pp. 127–138. ISBN 978-1-4129-3982-9.

[McKillup-17] McKillup, Steve (2006). "Probability helps you make a decision about your results". Statistics Explained: An Introductory Guide for Life Scientists (1st ed.). Cambridge, United Kingdom: Cambridge University Press. pp. 44–56. ISBN 978-0-521-54316-3.

[Myers_et_al-p124-18] Myers, Jerome L.; Well, Arnold D.; Lorch, Robert F. Jr. (2010). "The t distribution and its applications". Research Design and Statistical Analysis (3rd ed.). New York, NY: Routledge. pp. 124–153. ISBN 978-0-8058-6431-1.

[:1-19] Hooper, Peter. "What is P-value?" (PDF). University of Alberta, Department of Mathematical and Statistical Sciences. Archived from the original (PDF) on March 31, 2020. Retrieved November 10, 2019.

[20] Leung, W.-C. (2001-03-01). "Balancing statistical and clinical significance in evaluating treatment effects". Postgraduate Medical Journal. 77 (905): 201–204. doi:10.1136/pmj.77.905.201. ISSN 0032-5473. PMC 1741942. PMID 11222834.

Statistical significance information

and 19 Related for: Statistical significance information

Statistical significance

Statistical hypothesis test

Significance

Statistics

Clinical significance

Data dredging

Power of a test

Null hypothesis

Type I and type II errors

Stephen Ziliak

Analysis of variance

William Sealy Gosset

Binomial test

Hidden Markov model

Misuse of statistics

Probability of success

Effect size

Regression analysis

Detection limit