Dr. Marcus Crede has been threatened, legally and physically. His crime? Pointing out problems in published research. Lack of pre-registration, lack of power analysis, obvious calculation errors, impossible statistical results…authors, editors, peer-reviewers and readers are not doing so well. What is happening and what can we do about it?
Transcript
This transcript is AI-generated and may contain inaccuracies. Please do not quote myself or any of my guests based on this transcript.
[00:00:00] Ben Butina, Ph.D.: Hello and welcome to the Department 12 Podcast, where we talk about everything industrial and organizational psychology. I’m your host, Dr. Ben Butina. My guest today is Dr. Marcus Crede. How are you today, Marcus?
[00:00:12] Marcus Crede, Ph.D.: I am very well. Thanks for having me on the show, Ben.
[00:00:16] Ben Butina, Ph.D.: So you’re an Associate Professor of Psychology at Iowa State University.
And you have a truly impressive publication record. I was looking at it before the interview today, and I think what impressed me the most is the breadth of topics that you’ve published papers on. And it’s clear that you spend a lot of time reviewing the research literature in I-O across just many different topics.
And I’m guessing that what you found is that as a field we really, we really have our quantitative research and statistics game down.
[00:00:48] Marcus Crede, Ph.D.: Well, we’re very sophisticated in how we do our research and that’s certainly what I used to think. When I was in graduate school, we used to snare at our fellow social psychologists as being unsophisticated rub.
But I think my, my view on that has, has changed quite dramatically over the years. Primarily because of some of the review work I’ve done and some of it’s just kind of been accidental stuff. I’ve stumbled. Okay.
[00:01:09] Ben Butina, Ph.D.: My question was a bit of a joke because I talked to a lot of I-O psychologists and overwhelmingly conversations are about quantitative research. We are the numbers people. I guess what I’m wondering is during this review that you’ve done, what have you found? Like what surprised you.
[00:01:26] Marcus Crede, Ph.D.: It’s been a lot of kind of depressing stuff. I kind of break it down into a, a couple of, of broad areas that I I’m concerned about.
The first one, I guess, is that I think our. The readers of our journals. Don’t seem to read stuff very closely because what’s happened time and time again is is that I’ll be reading an article and I’m, I’m literally not looking for trouble. And I I’m reading it for, you know, preparing for a class or because I’m doing a, a meta-analysis and I’m reviewing something and sort of a very obvious and severe error kind of jumps out at me.
And then you look at the paper and you see that it’s been cited a thousand times and you ask yourself, How is it possible that I’m the first person to have noticed this to me, relatively obvious this and very severe error , but I feel like readers aren’t doing a good job. And, and of course the, the errors themselves indicate that, that these individual researchers are often working with.
So either they’re dishonest or they’re, they don’t know what they’re doing. Mm-hmm , which is maybe even more concerning. And then also. The sort of the lack of a response from editors and I’d say our society leaders has also been discouraging. They should be setting policies in place that minimize the degree to which these errors and problems make their way into the literature.
Then also being a little bit more responsive. When you them out and say, well, we’re gonna, you know, issue corrections or re retractions or whatever, whatever may be warranted. So I’m kind of disappointed with the field in general. And I kind of, I like to chop it up into those kind of three constituencies.
[00:03:02] Ben Butina, Ph.D.: So by the time you see an error like that as a reader, That means that the author or authors have either missed it or, you know, they’re deliberately doing something let’s give ’em the benefit of the doubt and say that they just didn’t know what they were doing. The editor or editors has missed it. The peer reviewers have missed it. So who’s not doing their job here. Is it everybody?
[00:03:23] Marcus Crede, Ph.D.: You know, it, it sounds harsh, but I, I feel like it is everybody. The weird thing is. You know, often I’ve, I’ve raised some of these concerns with, with colleagues with the, of course, with the authors, you know, might I shoot them an email and say, Hey, this, you know, this thing is going on.
Or I’ve had extensive discussions with editors as well. And one of the responses I got almost a decade ago now from the editor of a leading journal was like, well, yes, you’re right. These results that you are pointing out are impossible that they can’t possibly be. Right.
But everybody’s doing it right. Why, why are you picking on this particular author? When we know that this is a widespread phenomenon and at the time I was still relatively naive and I was like, what you saying? Everybody’s doing this that, that can’t possibly be true. And then I looked into it and we, you know, we, we started coding some articles and we did a bit of a review and, you know, the editor was kind of right.
It’s not everybody, but the, the, the proportion of articles that have these kinds of errors is so high. That if a IO practitioner came to me and said, Marcus, I, you know, I just read this really cool article in one of our top journals. You know, what do you think about it? Unless I could look at the article, I would probably say to them, you know, disregard it.
You cannot trust the work that’s being published in our fields. And the same way, you know, if I, if I said to you, Ben, I, I wanna take you to one of my favorite restaurants. There’s only a 20% chance that you’ll get food poison. You would probably turn down that invitation, right? Because the, the risk is too high.
It’s not, it’s not a hundred percent, it’s not even 50%, but it’s an unacceptably high level. If I told you, you know, step on this airplane, there’s only a 2% chance that it will crash again. You’d probably decline that offer. So I think the, the, the sort of the base rate of really severe problems in our field is so high.
that I, I honestly have trouble telling anybody to trust what is being published right now.
[00:05:31] Ben Butina, Ph.D.: And that is very disturbing. It’s science is supposed to be generative, right? We’re supposed to be building on each other’s work, but if each of the building blocks below us is junk, then kind of at a standstill or worse reverting as a field. So I learned about some of what you have found in, in your review, work on Twitter. And I’m gonna include a link to your Twitter account in the show notes, because I think you’re a great follow for a number of reasons. But recently you talked about a review that you did.
If you’re a listener and you’re thinking, well, maybe this is a little exaggerated I want to share with you something that, that Marcus shared two weeks ago. I, I think from, from when we’re recording this now in late June, 2022, you completed a big review project of some very recent articles in top I-O psych journals.
Although it wasn’t the primary focus of the review. You also coded whether the hypotheses were preregistered and whether a power analysis was used to inform sample size. So you coded 300 articles from top IO journals. How many of them did you find had preregistered their hypotheses?
[00:06:38] Marcus Crede, Ph.D.: We found one. And so this wasn’t all articles published in the journal. We, we were looking for something very specific, but there were lots of articles that met the sort of inclusion criteria. So we found one paper that had preregistered their hypotheses.
[00:06:51] Ben Butina, Ph.D.: This was a specific literature review for a specific topic.
I don’t know what that topic is, but. Any 300 articles recently published IO articles. The fact that, you know, we could pick any 300 and only one might have a preregistered hypothe. Very,
[00:07:06] Marcus Crede, Ph.D.: these were all, these were all papers with hypotheses. So these were not narrative reviews or theory papers, or even meta lyses.
These were all ones where primary data had been gathered and the people had, you know, were testing a whole series of hypotheses. And it’s, it’s not worthy that if you, if you look at some of the top journals in, in cognitive psychology or social psychology, the pre-registration rates are much, much higher.
You know, the it’s been, it’s been almost a decade now since we, which we learned about. The reason for pre-registration. , there’s no excuse for IO psychology to be so far behind the trend. And it’s not as if pre-registration is difficult, you know, it’s a, it’s a relatively easy process.
[00:07:48] Ben Butina, Ph.D.: So I, I have some students who listen to the show as well. Could you give them a thumbnail sketch of what pre-registration is and why it’s important.
[00:07:57] Marcus Crede, Ph.D.: Yeah. I mean, it’s a it’s a relatively straightforward process. There are many online resources for doing it that most widely used. One is probably the Open Science Forum.
You create an account it’s, it’s free. It’s very quick. And there are now templates, which you can use to sort of write down what your hypotheses are before you gather the data. Before you look at the data, you can also just put it in a word document and. And, and upload it. And the nice thing is it gets a timestamp.
So we can see when you uploaded this document or when you, when you registered , these hypotheses. And then we can be certain that the, the hypotheses were actually created a priori, right. They were before could. They were formulated before you, you looked at the data , and that removes concerns around harking, certainly hypothesizing after results known.
And if you also specify, you know, how you’re gonna analyze the data, you know, are you gonna use, and you can control for certain things, how you gonna treat outliers? How are you gonna treat missing data? Then you can also be more certain that P hacking hasn’t. So it gives people more confidence that you haven’t just stumbled across some sort of a chance relationship or chance pattern in your data, and then constructed a, a theory quote unquote or a hypothesis.
After the fact,
[00:09:10] Ben Butina, Ph.D.: and out out of 300, again, there was just one that had preregistered its hypothesis, shocking. You also looked at power analysis. So I assume we’re talking about 300 research articles and which statistical power is an issue, right?
Outta those 300, how many included a power analysis?
[00:09:27] Marcus Crede, Ph.D.: Not one, there was not a single mention of, of statistical power or calculations for it. Which again, very surprising because we’ve been as a, as a, the field of psychology, obviously been. Talking about and being concerned with, with low power since what, 19 60, 19 56, somewhere on there.
So it’s been, it’s been well over 50 years and, and it’s something that I’m almost certain that 99% of all researchers, when they go through their PhD programs, they’re exposed to that idea at some point So, again, very distracting.
[00:09:58] Ben Butina, Ph.D.: It would be hard to imagine them not being exposed to that pre-registration of hypotheses.
I can kind of buy that. It’s still, even though it’s been around for a while now, I can kind of buy that it’s maybe not emphasized as much in programs, just because so many of the current faculty may not have, have come up where pre-registration of hypothesis was a thing. But power analysis, as you point out is.
Not recent not by a long shot and not so terribly difficult to do. But for the, the sake of, again, those students who might not know what the heck we’re talking about could you just like, sort of explain what statistical power is?
[00:10:35] Marcus Crede, Ph.D.: , you want to be reasonably confident that you can detect the effects that you think are present in, in your data.
And so you wanna have a sufficient sample size to be able to do that. So for example, if you’re, if you want to test some complicated moderation hypothesis, doing that with a sample size of 50 is probably, you know, the chances of detecting an effect, even if it was there is gonna be abysmally low, but those are the kind of kind of sample sizes.
We still find. Relatively often in our, in our leading journals and by leading journals, for those of you who are wondering we looked at the Journal of Applied Psychology, Personnel Psychology, the Journal of Organizational Behavior, and the Journal of Management. So those were the four that we, that we reviewed.
These are career making journals often for young researchers, but you really want to be published as you’re, as you’re looking for a job or coming up for.
[00:11:24] Ben Butina, Ph.D.: Yeah, absolutely. These are not obscure journals. These, these are by I think, any reasonable standard considered top journals in our field.
So we’ve got Researchers who are submitting stuff, no power analysis. They’re not pre-registering the hypotheses. Despite the fact that both things have been talked about an awful lot editors are accepting them for publication and publishing them.
Sometimes in addition to just not doing those, you know, commonly accepted good practices, there’s also just. Statistical problems with the results that you’re seeing editors are missing it or ignoring it, not sure which reviewers also either missing it or ignoring this stuff. You as a reader in, in a pretty small minority of readers that are picking up on the problems you will, you know, bring it up to the editor maybe.
And the response is, yeah, that’s true, but everybody’s doing it. So why pick on this one journal? Why pick on this one person? I’m just wondering, have you given much thought to what could be a solution to this problem?
[00:12:25] Marcus Crede, Ph.D.: Well, you know, I don’t, I don’t have any particularly good ideas.
I will say the, the, you know, the one person, the one editor who I think has done a particularly good job of raising the quality of work published in their journal. Has been John Antonakis, who is, I mean, I think he’s stepping down, but he’s been the editor and chief of leadership quarterly. Yeah. For the last couple of years.
And you know, he’s really required much greater rigor in, in, in the work that is submitted there and You know, I’m, I’m not necessarily an expert in leadership research. You know, I, I do some work on the side with, with one of my colleagues on it. But the stuff coming out of there is, is, you know, you have much greater faith in it.
And the impact factor of the journal has skyrocketed. So people are taking the work more seriously because they have more trust in it. And so, you know, I think ultimately it’s gonna come down to, to editors in chief. Putting their foot down and saying, you know, we, there’s some basic things that need to happen for you to publish your work in our journal.
And if that comes at a short term cost of the number of submissions or how the, the journal is viewed or even relationships with, you know, peers and friends and colleagues so be it. And I, you know, I’m hopeful that we’re gonna have those kind of leaders taking charge of journals. Right now, it still seems like. There’s still, that’s still lacking. There I was, I was the, the person who raised concerns about the work that was published by Fred Walumbwa. A couple years ago with, with editors and the work he did with, with many of these colleagues. And there, the response of the, the journals for the most part was just appalling in my view, I mean was an unbiased source, but it was not great with some exceptions.
You know Leanne Atwater at Leadership Quarterly at the time she was the editor. She, I think, handled the situation very well. Neil Ashkenazi at the Journal of Organizational Behavior also I thought handled it very well. But the editors of other journals, I was deeply disappointed.
[00:14:39] Ben Butina, Ph.D.: Yeah, it’s, it’s interesting that you, you know, you talk about editors and their ability to make a difference.
And, and I follow John on social media as well. I don’t know him personally, but I was also impressed with just how open he is about his editorial practices very open to talking about why he is doing what he’s doing. I’ve been really impressed with that. And, and some of the stuff I’ve read, it’s like, wow, this just sounds like a regular person.
How did that get published? And almost always, I look at, and it’s like, oh, it’s leadership quarterly. And John’s the editor. He allowed them to write like a person I wonder, do you have any any hope for sort of decentralized solutions to this? So when I think about something like researchers, one or other kinds of new sort of peer review publishing platforms that are being proposed do you see any future in that?
I, I guess when I’m thinking about some of these articles, I think, wow. If this was published on a platform that. Post publication review. So in other words, it’s not counting on you to co contact an editor and expect them to do the right thing, but that there’s actually a forum for you. Put a, a comment right along that article there.
Right. Pointing out the issue. Where do you think that goes? Is that gonna be helpful
[00:15:47] Marcus Crede, Ph.D.: I think, I think it would be I think the, the big, the big hurdle for that to kind of be a, have, have a positive impact is that. You know, as a field, certainly in the time that I’ve been involved with it, we’ve had this, this very substantial move towards business schools.
You know, a lot of our, our top faculty understandably have doubled their salaries and moved to business schools and business schools. And I think until they change. How they rank each other what they use for promotion criteria until they deemphasize some of these top journals a bit it’s gonna be very hard to argue that a young researcher should.
Publishing these alternative formats, which are scientifically, perhaps much more rigorous and better for the field, but for their own careers is gonna be detrimental. I mean, there, there are many business schools who will really only consider articles that are published in a very, very narrow range of journals.
And it’s often the same journals that, you know, that have these problematic practices. And then , the greater, the competition. To get published in those journals, the greater the incentive is to, you know, fudge your numbers or to engage in harking. You’ve got some data that you gathered from an organization, what you were looking for initially didn’t pan out, or, you know, the hypothesis wasn’t supported.
So you start, you know, sifting through the results and you look at every two way and three-way interactions and you on some mediation models and some, you. Whatever you can come up with and eventually you’ll stumble across something, right? Yeah. And then you pretend that that you’d expected that all along.
And if, if that’s the way that business is done and that’s the way your colleagues do things, or at least you think they do things, then I think the system just perpetuates itself. And you, look, you look at the authorship of people in those journals and they’re mostly now at business schools. It’s, it’s relatively rare.
Especially in journals, like I think the journal of management for. Sort of traditional IO folks to, to publish articles there. So I I’m, I’m hopeful that eventually this will happen, but I think there are real obstacles in the way.
[00:17:49] Ben Butina, Ph.D.: It seems like it’s an issue of misaligned incentives, particularly in academia.
I’m very much from the practitioner side myself. And, and so for me, as, someone who tries to use the best evidence to make decisions and, and implement interventions, It’s discouraging to hear you. How, how many problems we have in the published research.
And it makes it tougher to go to, to stakeholders and say, look, we shouldn’t just be making decisions on our gut. Or because of what work for us once before, you know, there’s all kinds of biases that come into that we really should be making decisions based on the best available evidence. And it really just kind of sucks when the best available evidence is, is still so lacking in these areas.
I, I guess one of the things that surprises me, or maybe it shouldn’t is. Even when you take the incentives out of it or, or lessen the incentive pressure a little bit after a researcher has tenure, I, I still don’t see anybody making the move to these alternative platforms. I wonder if you’re, are you seeing any signs of that?
It seems like it would be easier. Okay. You you’re now, you know, in a research university, you’ve got your tenure. Now you don’t necessarily need to publish everything that you come up with in a high impact journal, that’s gonna help you get tenure, cuz that was kind of your goal. You can go for a platform that is more rigorous or that allows, you know, post publication review or you know, a single platform for your pre-registration your data sharing your publishing.
Right. I just doesn’t seem to be that anybody’s doing that. Or very few people I should say very few people.
[00:19:16] Marcus Crede, Ph.D.: Yeah, very small minority and you know, it would be nice if tenure gave you that, you know that security, but I think I’d have at least two responses. The first one is that, you know, people develop habits and they develop methods of, of getting their work done.
And you know, if you’ve gone through grad five years of grad school and six years of pre-tenure, so you’ve got 11 years under your belt, at least. Are you really gonna change the way you conduct the business of doing research when it’s kind of worked again for you? Yeah, it’s doubtful. And then of course there are often still massive financial incentives.
You know, I’ve heard through the grapevine and people have confirmed this to me that, that there are quite a few business schools who will offer almost a, a type of boundary for Articles published in the right journal. So, you know, if I’m in a business school and I get an academy of management journal publication, suddenly my Dean might find, you know, three month summer salary for me as a, as a reward, you know, and if I’m, if I’m, if I’m in a business school that probably easily translates into 30 to $50,000.
And so, you know, the, the incentives are perhaps lessened, but they certainly don’t go.
In a psychology department, we don’t have these. I wish I wish that was the case, but, and I’m not saying that you know, that this is all business schools and I’m certainly not saying these all business school professors do this, but again, the, the base rate is so high.
That, you know, I wish the, the honest brokers out there in the business schools would stand up a little bit more and, and shake their fists at, at their colleagues and say this, you know, we’re, we’re destroying our own field, the credibility of our own field. And I, I, I see relatively little of that.
There were very few people who, who seemed to be upset when, you know, when all of this, these, these apparently fraudulent results by the Volo group came up. There were far more people who were upset with me. For, for pointing it out and contacting editors, you know, I got, I got threats, I got legal threats.
I got physically threatened. I, or attempted bribes. It was, it was all sorts of crazy stuff. I think only once did somebody come up to me at a conference and say, you know, thanks for doing that. And that was, that person was, you know, retired and kind of out of the, out of the system. I don’t know. We, we seem to have a bit of a, a culture problem perhaps as well.
[00:21:33] Ben Butina, Ph.D.: It’s exactly the opposite of what we tell non-scientists about how science works. I’ve noticed, you know, it’s, this is why you should trust it is because we have all of these checks and balances and transparency and so on. And then I hear a story like yours, where.
You’re the bag guy for pointing out that there is a problem as if you have some kind of personal crusade against a, a particular researcher.
[00:21:55] Marcus Crede, Ph.D.: Would also help. If we had more, more contact with practitioners like yourself, if the gap was not quite as wide as it were, because, if we saw more value in honestly reporting.
For practitioners, here’s, you know, here’s a new selection method that we thought might be really useful, but turns out not as useful as we thought, or, you know, here’s a new way of training people and it’s, you know, it’s a little bit better than, than the status quo. I think then somehow we, we give back to our roots of, of asking important and relevant questions, not being so theory, obsessed, and also, being more transparent and open about what we really find.
[00:22:33] Ben Butina, Ph.D.: This has been a, a fantastic and enlightening conversation. Marcus, I really want to thank you for taking the time to talk to me this afternoon.
[00:22:40] Marcus Crede, Ph.D.: Thank you for the opportunity. It’s been fun.
