A Sensible, Practical Proposal

Nonetheless, we disagree that a statistical significance-based “filtering process is useful to avoid drowning in noise” in science and instead view such filtering as harmful. First, the implicit rule to not publish nonsignificant results biases the literature with overestimated effect sizes and encourages “hacking” to get significance. Second, nonsignificant results are often wrongly treated as zero. Third, significant results are often wrongly treated as truth rather than as the noisy estimates they are, thereby creating unrealistic expectations of replicability. Fourth, filtering on statistical significance provides no guarantee against noise. Instead, it amplifies noise because the quantity on which the filtering is based (the p-value) is itself extremely noisy and is made more so by dichotomizing it.

Amrhein, Gelman, Greenland, & McShane

The controversy about significance testing continues in the pages of JAMA. The latest salvo, in the form of a letter to the editor, can be viewed in preprint form here.

About the Author

Ben Butina, Ph.D.
Dr. Butina, who hosts the Department 12 Podcast, is an industrial-organizational psychologist with interests in training, leadership development, talent management, and positive psychology in the workplace.

1 Comment on "A Sensible, Practical Proposal"

  1. 5/13/2019 @ 00:46:47 In my view, department12.com does a great job of covering subject matter of this type! While often intentionally controversial, the material posted is more often than not well researched and stimulating.

Leave a comment

Your email address will not be published.


*