While psychology has been mired in a “replication crisis” recently – based on the failure of contemporary researchers to recreate some of its most cherished findings – there have been pockets of good news for certain sub-disciplines in the field. For instance, some replication efforts in cognitive psychology and experimental philosophy or X-phi have been more successful, suggesting that results in these areas are more robust.
To this more optimistic list we may now add personality psychology, or at least the specific area of research linking the Big Five personality trait scores with various personal and life outcomes, such as higher Neuroticism being associated with poorer mental health and reduced relationship satisfaction; higher trait Conscientiousness being associated with less risk of substance abuse; and stronger Extraversion correlating with leadership roles.
In his new paper that is in press at Psychological Science (and available as a preprint at the Open Science Framework), Christopher Soto at Colby College speculates that perhaps it is the tendency for researchers in personality to use large samples of participants, numbering in the hundreds or thousands, and to use reliable, standardised tests, that is to some extent responsible for the relatively robust results in this area. The new findings “leave us cautiously optimistic about the current state and future prospects of the personality-outcome literature,” Soto writes.
Soto sourced 78 previously published trait-outcome associations from a major review published in 2006 “Personality and the Prediction of Consequential Outcomes“. To see if these effects replicated, he recruited four online samples totalling over 6,000 younger and older participants, and asked them to complete an established 60-item measure of the Big Five personality traits, and then to complete various measures of other life outcomes, covering everything from career to relationship success, to political orientation to criminality (to manage the length of the new surveys some of these were abridged versions of the measures used in the original research).
The vast majority (around 87 per cent) of the previously published trait-outcome associations replicated. This is not a perfect result, obviously, but it is far more impressive than the average 36 per cent replication rate achieved in 2015 by the Reproducibility Project’s attempt to replicate findings across social and cognitive psychology, and the less than 50 per cent success rate reported late last year by the Many Labs 2 project after its attempt to replicate various social psychology findings.
A couple of original trait-outcome effects that did not replicate this time included lower Agreeableness not correlating with heart disease and higher Conscientiousness not correlating with greater family satisfaction.
Despite the overall impressive replication success rate, between 63 to 71 per cent of the time the replicated effects were weaker than in the original research, and between 30 to 42 per cent of the time the replicated effect was substantially weaker (the precise figures depended on whether the analyses corrected for the use of abridged outcome measures). This indicates that the existing trait-outcome literature contains at least some false-positive results (i.e. null findings erroneously reported as positive). Overall, the new personality trait-outcome associations were about 80 per cent as large as in the original research. On a positive note, the weakening of significant effect sizes seen here is not as dramatic as in the Reproducibility Project in 2015.
Original results with a larger effect size replicated more often, and it helped if the replication had more statistical power and if it used the same data source and format (for example, if both the original and the replication used self-report questionnaires). The bigger picture, though, is that most results did replicate even though the new research was obviously conducted in a different era (some of the original results date as far back as the 1980s), sometimes in a different geographical locale, and using a different personality measure. This echoes the message from the recent Many Labs 2 Project that found that those psychology results that do replicate tend to be relatively robust in the face of such contextual differences.
Soto’s new replication effort, dubbed “the Life Outcomes Of Personality Replication (LOOPR) project”, meets the highest research standards, with its methods and hypotheses being pre-registered (to avoid the possibility for post-hoc data tweaking and mining) and with all the materials and data made available online at the Open Science Framework. However, as Soto acknowledges, the study is not perfect – for instance, it involved a cross-sectional design with trait scores and life outcomes measured at the same time point, whereas it is advantageous to use a longitudinal design with outcomes measured later; and only a single replication attempt was made for each trait-outcome finding, whereas in future it may be revealing for multiple attempts of the same findings to be attempted, allowing for the influence of moderating contextual factors to be explored.
Despite these limitations, this is surely a welcome news story for the field of personality psychology. Soto is cautiously upbeat in his conclusion, writing that: “… the extant literature provides a reasonably accurate map of how the Big Five personality traits relate with consequential life outcomes, but that personality psychology still stands to gain from ongoing efforts to improve the replicability of behavioral science.”