The relationship between psychologists and statistics can be likened to the relationship between drivers and cars. Most drivers use cars, but have a minimal understanding of how they work. Our understanding of cars–such as it is–depends on the explanations provided by experts–mechanics and engineers. It follows, then, that our understanding is heavily influenced by the explainers we listen to.
For most of us, the first statistics explainers were our stats and research instructors. They were chosen for us by whatever university we attended. Our ability to pass classes–and, in some cases, complete an approved thesis or dissertation–depended on our willingness to buy into whatever statistical paradigm those assigned experts expounded.
After graduation, however, we have our choice of explainers. We read articles by statisticians who, naturally, have competing views on the statistical subjects that interest us. We decide who to read and how much weight to give their arguments.
The problem with this is obvious: as non-experts in statistics, we’re not in a good position to make these choices and judgments. We’re likely to favor the arguments of explainers who reinforce our existing opinions and dismiss or minimize the arguments of those who do not. (If someone would like to argue that psychologists are somehow exempt from confirmation bias, I eagerly await your evidence.)
I’ll use myself as an example. I dislike p-values. I won’t try to explain why in this article. The problems and limitations of p-values have been discussed at length in the academic literature and I don’t have anything original to contribute. Suffice it to say that I agree strongly with its critics.
But I’m just self-aware enough to know that my strong opinion has made me subject to confirmation bias in this area. Like other humans, I tend to give more weight to the opinions and arguments of those who agree with me than those who do not.
The obvious first step to combating this bias is to read more widely and more fairly.
He who knows only his own side of the case knows little of that. His reasons may be good, and no one may have been able to refute them. But if he is equally unable to refute the reasons on the opposite side, if he does not so much as know what they are, he has no ground for preferring either opinion…Nor is it enough that he should hear the opinions of adversaries from his own teachers, presented as they state them, and accompanied by what they offer as refutations. He must be able to hear them from persons who actually believe them…he must know them in their most plausible and persuasive form.
John Stuart Mill
Reading opposing arguments more frequently and with greater sympathy may go a little way toward addressing confirmation bias, but it immediately confronts us with another major obstacle: a lack of expertise in statistics. To what extent are we truly able to understand and evaluate arguments written in field other than our own?
I’m sure that we are, on average, better-informed than the general public on statistics. But I am even more certain that we are substantially less informed on the topic than those who have earned PhDs in the field.
One potential solution to this problem must be dismissed from the start: it is simply not practical for every psychologist to also become a professional statistician.
It is possible, and necessary, however, for all of us to become better-informed about statistics than our graduate programs have made us.
To return to my car and driver analogy, it’s not possible for every driver to become an automobile engineer, but it is possible to become more like a race car driver. Race car drivers are not engineers, but they do know a great deal about cars. They know that if they want to win races, they have to understand something of what’s happening under the hood.
For me, this process will involve setting aside the controversies of the day for a while and spending more time learning the fundamentals. I don’t need to revisit the basics of how to “drive” statistics that I learned in grad school. I need to get a better handle on what’s happening under the hood.
If you found yourself nodding along to any part of this article, I hope you’ll do the same. Otherwise, we’ll remain at the mercies of our cherry-picked explainers forever.
