Have we overestimated the effectiveness of psychotherapy?

By Christian Jarrett

Most people who undertake psychotherapy seem to benefit from it. How do we know? Arguably, the most important evidence comes from meta-analyses that combine the results from many – sometimes hundreds – of randomly controlled trials. Based on this, it’s been estimated that psychotherapy is effective for about 80 per cent of people (meanwhile, between five to 10 per cent of clients may suffer adverse effects).

But now the more concerning news: a team of researchers led by Evangelos Evangelou at the University of Ioannina, Greece has assessed the quality of 247 of these psychotherapy meta-analyses and they report in Acta Psychiatrica Scandinavica that many of them have serious methodological short-comings.

Coincidentally, a separate research group led by Brent Roberts at the University of Illinois, Urbana-Champaign has just published in Journal of Personality some of the first observational data on how people’s personalities change after undertaking psychotherapy. In contrast to what’s been found in the clinical literature, they report that people who’ve been in therapy seem to show negative changes in personality and other psychological outcomes.

In their “umbrella review” of the psychotherapy literature, Evangelou and his team chose to include published meta-analyses covering any form of psychotherapy and almost any kind of target mental health condition, but with the proviso that each meta-analysis had itself combined the results from a minimum of ten studies. Their comprehensive search resulted in 247 unique meta-analyses which collectively synthesised data from over five thousand randomly controlled trials.

Overall, 80 per cent of the published psychotherapy meta-analyses had reported a significant and positive benefit of whatever form of psychotherapy was their focus. This sounds impressive at first, but after applying “state-of-the-art” tests of their robustness, Evangelou and his colleagues report that just 16 of the 247 meta-analyses had provided “convincing evidence”.

The researchers identified a number of issues:

  • Many meta-analyses showed a statistically significant amount of heterogeneity between the trials that they’d combined. The worry is that too many meta-analyses are comparing apples and oranges, although there is scholarly debate about what level of heterogeneity is unacceptable.
  • The researchers found many instances of the “small study bias”, which is the tendency for smaller, less robust studies to report larger effects.
  • They found evidence of “excess significance bias“, which is when an over-abundance of trials seem to report positive findings given what we know so far about psychotherapy’s effectiveness. This suggests negative findings are remaining unpublished for whatever reason.

Evangelou and his team conclude that the field of psychotherapy research needs to work harder to ensure that negative results are published as well as good news results, especially given the findings of another recent paper suggesting that the field has a problem with undeclared researcher allegiance to particular therapeutic approaches. One way round this is to ensure all trials and meta-analyses are preregistered before they are conducted, alongside information on the statistical tests that are planned.

For the reader undertaking psychotherapy or who knows someone who is, it is worth keeping some perspective: this is just one critique and the weight of evidence still suggests that psychotherapy is, more often than not, beneficial.

For scholars, the methodological concerns raised by this new paper feed into an already contentious field. Some experts believe that, high-quality or not, randomly controlled trials (and by extension, meta-analyses based on those trials) are not really an appropriate way to gauge the effectiveness of psychotherapy because of all complex, myriad factors involved in the dynamic between a client and his or her therapist.

An alternative approach is to look at observational data. Rather than signing people up to a controlled trial, with all the contrivances that that entails (such as standardising the delivery of therapy as much as possible), this approach is less hands-on and involves looking instead at the outcomes of people who happen to have been in therapy and those who haven’t.

Brent Roberts and his colleagues found two sources of this kind of data: hundreds of students in Tübingen who were enrolled in a long-term personality study and who’d completed measures twice across four years; and a group of thousands of older Americans who similarly had completed personality and other measures twice across four years.

Crucially, both these longitudinal surveys included a question about whether the participants had undertaken psychotherapy in the intervening period between the two data collection points. One hundred and twenty-eight of the Tübingen students had completed some therapy and, compared to the other students, they showed negative changes in their personality in terms of higher scores in neuroticism, less extraversion and conscientiousness, as well as reductions in self-esteem, increases in depression, and less life satisfaction. It was a similar story for the older American adults who’d been in therapy: they showed negative changes in personality and other psychological outcomes.

There are problems with how to interpret these findings – an obvious shortcoming of observational data of this kind is that it’s less controlled than an experimental trial. For example, perhaps undertaking therapy was a consequence of these unwelcome psychological changes rather than the cause (although this still wouldn’t explain why the therapy hadn’t been more helpful, more often). Whatever the explanation, however, the results stand in contrast to the findings from controlled psychotherapy trials which have pointed overwhelming to positive personality changes arising from therapy. As Roberts and his team conclude, “the gravity of the issue necessitates that researchers investigate the apparent discrepancy between these findings and those from well-controlled trials”.

Does psychotherapy work? An umbrella review of meta-analyses of randomised controlled trials

Therapy experience in naturalistic observational studies is associated with negative changes in personality

Image via gettyimages.co.uk

Christian Jarrett (@Psych_Writer) is Editor of BPS Research Digest

16 thoughts on “Have we overestimated the effectiveness of psychotherapy?”

  1. When considering the efficacy of therapy, you have to consider the same factors as you would consider with other types of health efficacy measures. That is, compliance and consistency of the patient matter. If a person with diabetes sees a doctor and doesn’t institute lifestyle changes or take medication, then the treatment will not be effective and it is not a failure of the treatment plan, but of the ability of the patient to follow through. Psychotherapy is no different, particularly in an age in which evidence-based procedures such as CBT (which ask the patient to do more “homework” than traditional talk therapy alone) are more common and the number of sessions is limited by insurance policies. I know far too many people who go to therapy, but do nothing on their own, or fail entirely to engage with the therapy. They show up and expect magically to get better by just complaining or talking. If clients don’t put in the work, they don’t get better. It is the same as other ailments of the body – working through the therapeutic process matters enormously when measuring efficacy. Somehow, that aspect seems left out when questioning the value and efficacy of psychotherapy while being included in medical studies.

    1. To Orchid64-we do not know what has been considered. On precisely what measures were used to test the robustness of these studies, this article is silent. Science, no matter whether medicine or research, has to follow the same basic principle: empirically test a hypothesis, revise, and apply, repeat.
      If patients aren’t following through, that is a measure of the effectiveness of therpay: i concur with the whole ‘you can take a horse to water…’ point, but, a skilled equestrian will know how and when the horse needs to drink, and will drink, so they will not take it to water until then. A doctor is in a similar role. They must bring the patient healing, when the patient is ready.

      1. “…. skilled equestrian will know how and when the horse needs to drink, and will drink, so they will not take it (sic) to water until then”…….this is a strange analogy for psychotherapy, involving a high level of control attributed to the therapist (although the phrase starts off with the “patient” (client?) in the driving seat for seeking therapy). The “skilled equestrian…will know how and when etc”…….a horse, like all animals equipped with homeostatic mechanisms, when at liberty to do so, will drink as he or she (do we talk about patients/clients as “it?) needs: the idea that anyone at all has to “lead him or her to water” is putting the equestrian/therapist in a position of completely unnecessary authority. Perhaps this is what’s wrong with therapy in the first place?

    2. Very good! Some conditions, the milder ones, don’t demand such hard work. But I found that many people who had suffered long term abuse from drugs and hospitalizations seemed highly motivated to work extremely hard on recovery. One patient who had been self destructive and suicidal for over a decade would have come 24 hours a day! It turned out she’d been in an incubator after a difficult birth and then developed a deadly, but treatable disease. She required a babies bottle for 2 years as part of her therapy.

      I almost died from diabetis over 50 years ago and, similarly, was so motivated for life that I’m now 88 and still love golf and tennis.

      1. could no agree more, i have also had cases where the patient had the most severe history of drug abuse, yet show show much determination with recovery from the start

      2. This seems to miss a great deal ‘For the reader undertaking psychotherapy or who knows someone who is, it is worth keeping some perspective: this is just one critique and the weight of evidence still suggests that psychotherapy is, more often than not, beneficial’

        This might be one critique but there are many others Dr. William Epstein wrote a book many years ago about all of this The Illusion of Psychotherapy and he appears in the interview below:


        This book is also an interesting critique https://www.amazon.co.uk/Therapy-Industry-Irresistible-Talking-Doesnt/dp/0745329861/ref=sr_1_1?ie=UTF8&qid=1536260760&sr=8-1&keywords=the+therapy+industry

        as is David Smails work

        Psychotherapy is most often a short term false relationship that largely views the person as having some sort of personal issue, faulty thinking, behaviour, attitudes etc – yet we are always in dynamic interaction with so much more – power, culture, our bodies – most of which is beyond our awareness.

        When things go wrong we say ‘i’m depressed’ or whatever when really we’re having a depressed or whatever REACTION to the many disorders in the culture/environment – most psychotherapy relies on the person changing through insight and will power – when insight does not necessarily leads to any change – just consider the amount of things in your own life that you have clear insight into but are nonetheless repeating over and over again – will power in the face of what seems more like a deterministic universe living inside the body/mind of a single human being surrounded by powers of ideology, class, economy, media, education seems an illusion too.

  2. Pingback: Unsafe Space

Comments are closed.