Replication success correlates with researcher expertise (but not for the reasons you might think)

Old businessman holding his glassesBy Christian Jarrett

During the ongoing “replication crisis” in psychology, in which new attempts to reproduce previously published results have frequently failed, a common claim by the authors of the original work has been that those attempting a replication have lacked sufficient experimental expertise. Part of their argument, as explained recently by Shane Bench and his colleagues in the Journal of Experimental Social Psychology, is that “just as master chess players and seasoned firefighters develop intuitive expertise that aids their decision making, seasoned experimenters may develop intuitive expertise that influences the ‘micro decisions’ they make about study selection … and data collection.”

To see if there really is any link between researcher expertise and the chances of replication success, Bench and his colleagues have analysed the results of the recent “Reproducibility Project” in which 270 psychologists attempted to replicate 100 previous studies, managing a success rate of less than 40 per cent. Bench’s team found that replication researcher team expertise, as measured by first and senior author’s number of prior publications, was indeed correlated with the size of effect obtained in the replication attempt, but there’s more to the story.

The correlation between expertise and replication effect size would appear to at least partly vindicate the authors of original studies who have claimed that replication attempts have failed because replication researchers have lacked sufficient expertise. However, there is a twist because Bench and his colleagues also looked to see whether replication researcher expertise correlated with the original effect size found in the research they were attempting to replicate, and they found that it was.

What’s more, the link between replication researcher expertise and replication success evaporated when this other correlation (between replication researcher expertise and strength of the original finding) was taken into account. In other words, it seems that more experienced researchers are more likely to publish successful replications because they tend to choose more robust studies to replicate. Supporting this, there was also evidence that more experienced researchers tended to choose to replicate original studies that were less culturally sensitive, with more obviously generalisable results.

“The current results provocatively suggest” said Bench and his colleagues, “that expertise may not play a role in the execution of replication studies. Rather, expertise contributes to the ability to identify studies that are good candidates for replication.”

Of course, this is not the end of the issue. In the Reproducibility Project, replication teams did not have an entirely free choice of studies to replicate but chose from a pre-selection. It remains to be seen if researcher expertise will have the same relevance in other situations. Also, the current study used a fairly crude method of establishing researcher expertise (they looked at number of previous publications and whether they included data and experiments, but not at the direct relevance of the methods involved). For these reasons and more, Bench and his team admitted that “ironically, these results are themselves ripe for replication in order to truly understand the value of expertise in scientific research.”

Does expertise matter in replication? An examination of the reproducibility project: Psychology

Christian Jarrett (@Psych_Writer) is Editor of BPS Research Digest