Hmm. I’m not sure where I gave the impression that publication is the main or sole problem that mental health faces. In fact, I think that if you read closely I signal that there are many other challenges as well. Like many complex fields, there are multiple problems. Attempting to address them all in one book or, God forbid, blog post would be futile. My intent with this post is simply to point to one problem in particular.
Your proposed product is a wonderful example of an easily tested intervention that would not require anything like the investment involved in testing a medication. One reason that many health professionals (myself very definitely included) get so cranky about “alternative healthcare” is that people sit behind ideas that “my novel approach is already empirically supported” (huh?) or “it has a holistic influence and so isn’t appropriate for research.” (I know you aren’t saying this yourself.) But in reality, the intervention you’d like to put in place is ideally suited to a cheap and easy few tests.
You want to bring this product to an institution, and you have specific ideas about why they would want to make that investment. People will get more minutes of sleep, they’ll have fewer violent outbursts – something. Offer to install the equipment at your own cost, and seek ethics approval – you will need a nonvulnerable population and some form of consent. Then have an independent researcher (not a true believer like yourself) randomize the way it is used (by week, month, or whatever), and OTHER observers who do not know the condition rate the outcome variable of interest. Expectancy should not be an enormous issue, but it must be controlled for, and scorers will certainly be influenced by their own beliefs if they know which conditions are in place when.
Do this properly and you should have no difficulty publishing in a reputable journal (yes, such things do exist). But you can guess my proviso. Only do the research if you are committed to publish the outcomes WHATEVER HAPPENS, even if the disappointing results threaten your business. And get help with the research design (the above few sentences don’t count – as you may see from the howls of outrage that I haven’t eliminated every source of bias in my short reply). The last thing we need is another Critical Incident Stress Debriefing, or Past Life Regression Therapy. Or paroxetine for children.
Agreed. We have tended to focus on the funders and writers of these trials, and many reading about the problem could get the impression that the journals are the poor victimized innocents in the affair – like the drivers of cars hijacked by escaping bank robbers.
But the abject failure of many of the journals to live up even to their own declarations of “policy” doesn’t support this view of the problem. They are driving the getaway car with full knowledge and consent, and can only be viewed as accomplices in an endeavour which has a negative impact on many lives.
I agree that patient opinions need to be paid attention to, and the field has been quite paternalistic. “Quite” is, admittedly, an understatement.
You raise an interesting possibility: What if an independent body could rate journals not in terms of citation count, but in terms of adherence to specific standards of evidence. Things like willingness to publish negative results, unwillingness to publish short-term trials of meds for long-term conditions, and disqualification of studies not previously declared or registered. Then my only-half-facetious argument against reading journals could be more finely tuned, and physicians could make a point of reading better research and ignoring infomercials.
Of course, there’s still the issue of sponsored lunches, clinic “educational” visits, and funded talks at professional conferences…
Thank you for your comment. I agree with the points you have made here. And you are quite right to point out that the same criticisms could be made of psychotherapy outcome trials – publication bias in favour of positive trials, and a tendency of researchers to shelve nonsignificant results rather than attempting to publish.
As well, we in psychology do seem to have become obsessed with demonstrating the effectiveness of specific technique-based therapy models in the absence of good evidence of unique effects (for the most part – specific strategies for panic attacks, and exposure-based therapies for some anxiety disorders do appear to be more powerful than free-form counselling). It is as though we have spent 30 years arguing over the right colour of cardigan to wear in the consulting room, when this clearly is not the prime determinant of outcome.
My focus on Study 329 could be criticized as a slander to the field, attacking as it does the low-hanging fruit of one of the most criticized trials (and publications) in psychiatric history. The truth is somewhat more troubling: Study 329 is simply a recognized example of a much more pervasive problem in the literature.
You point out that as an 8-week trial it should have been disqualified from consideration out of hand. Well, certainly it has little to say about long-term outcome, which is the primary variable of interest. But short-term trials of 8-16 weeks are pretty much the norm in psychopharmacology – in large part because this is all the regulatory agencies demand, and the point of the studies is to get regulatory approval.
If we dismissed all the short-term studies of medications intended for long-term problems we would thin out the psychopharmacology literature to a very surprising extent, and force everyone back to the labs to look at the outcomes of real interest. I’ll leave it to you to guess whether I think that would be a good thing.
I would recommend Goldacre’s book Bad Pharma, which summarizes the problems in much pharmaceutical research. As well, Whitaker’s Anatomy of an Epidemic (on psychopharmacology) and Kirsch’s The Emperor’s New Drugs (which emphasizes antidepressants). From these you can access individual pieces on various problems in the literature. I’ve written a lot on the issue as well, and you can find more at my blog, psychologysalon.com, and click on the keyword “Medication”.
Given your request, I am providing references to discussions of the problems, and I am very conscious on this site of coming across as a dismissive across-the-board “it’s all nonsense” skeptic. I am skeptical of many claims in mental health, but would not consider myself anti-psychiatry, anti-psychology, or anti-therapy – particularly given that I am a psychologist and psychotherapist. I am not one who feels we need to get rid of the mental health field, but to improve it. In part, my suggestion that journal-reading is unproductive is meant as a goad for things to get better, not to disregard the quest for effective, evidence-based approaches to genuine problems.
Although much of the press and the awareness campaign (and hence my post) is based on pharmaceutical research, exactly the same problem pervades much of science, including my own field of clinical psychology.
I think you’re right, that we should have not only the writeups, but access to original data, and this is part of the push right now with alltrials.net and other sources.
You mention the idea of studies not passing the sniff test, and many in fact do not. Study 329 is a prime example of this, and it is hardly an isolated case. But even if a published study is perfectly run and reported and passes the sniff test with flying colours (odours?), the bias problem means that it will not be balanced out by other well-run studies showing different effects.
In years past, journals were sharply limited by printing and distribution costs, resulting in a necessary funnelling and selection effect. With the Internet, such concerns are no longer relevant, and there is no reason for a journal to decline publication of a study showing null results, provided it is well done. (The problem, of course, is that more reviewers would be needed, but this seems to be a concern that could be overcome.) Journals could even have a separate section for “Null Results” or “Failures to Replicate” so that their exciting positive studies are not obscured.
One barrier is the ongoing (and likely temporary) phenomenon of dual publishing (paper and net). It probably won’t be long before most journals cease the print version, thus freeing themselves up.
Hi Fred, Thanks for your comment. I think you’re right that the anger most people feel relates to the misuse of science, rather than to science itself. Jeffrey Lieberman has characterized much of the criticism mental health has received as “anti-science” which strikes me as a) a misperception of what is being said, and b) a stance that some get tempted by when they learn that science is not as pristine as sometimes advertised.
I’m struck by how often I hear the equivalent of “The science isn’t perfect so I’m going with iridology instead.” There is so much pseudo-science in the alternative medicine field one wants to tear one’s hair out. The one advantage some of it has (not to be sneezed at) is that inert treatments are often at least not damaging (though sometimes they are harmful as well). I don’t mean by saying this that all “alternative” practices are bunk – that seems clearly false and much of what I do would be regarded as alternative by many – but that we should also have some humility and scientific curiosity about these as well.
For most of us, most of the time, under most circumstances, we will figure things out on our own, or with the help and support of family and friends. We have become rather too quick to assume that difficulties in life are diseases, or can be treated with a pill (or a cognitive challenging exercise).
There remain certain situations, though, in which our coping resources are overwhelmed or we run out of ideas, and at that time an external viewer/advisor can often be useful. The industrialization of support has been problematic (an understatement, I know). But I continue to believe that, when this happens, external help can have a significant role to play.
You’re right, there are many problems – too many to cover in any single article. This makes it a challenge to discuss any one of them, as I frequently get jumped on for “naively thinking this is the only problem” when I pick one to look at!
One of the very significant problems that I haven’t discussed is the one you focus on here – that medication effects get relabeled as disease effects. The diagnosis of bipolar disorder following antidepressant-induced mania is a good example.
You are right that industry interests are likely to oppose greater transparency in medication research, and that they have been largely successful to date. Hopefully through authors like Goldacre, Healy, and Whitaker, and well-educated medical reporters, and sites like this one, the awareness of the problems and the push to correct them will move things in a more pro-science direction.
The concern I have is that there will simply be a polarization – between the cheerleaders like Jeffrey Lieberman (Shrinks) touting or grossly exaggerating the successes and ignoring the problems, and the “anti-psychiatry” forces wanting to throw it all out and practice Reiki. Rage is a powerful drug, and it’s easy to feel its pull.
Not at all. Studies that are ‘garbage’ have often been a problem, but many of the issues are more subtle.
In the case that I am describing, we can have perfectly good studies – not garbage at all – and incorporate them into a properly-analyzed and dispassionately reported document, and still wind up with a distortion in the data.
Thanks for reading.
One of the limiting factors in the publication of nonsignificant or contradictory results has been the sheer cost of printing and distribution. I think that the development of online journals (preferably with peer review) will eventually change a great deal of how science is conducted and reported. There is no longer any problem with a journal “issue” of 500 pages, and editors will still be able to “highlight” the articles they think are most interesting.
This won’t affect the problem of selective publication by authors, unfortunately. This is a challenge best met through other means – like editors standing by their previously-announced trial registry policies.
You are correct on several counts. Madinamerica generally tries to raise issues with the science side of mental health – ideally with the goal of prompting greater awareness of and adherence to the principles of science within these professions. It runs the risk, however, of becoming a magnet for conspiracy theories and contributing to an anti-science forum of rage. (Watch, for example, the replies to that statement.)
The involvement of pharmaceutical companies in researching their own products, coupled with an extreme version of the profit motive, contributes to much of the problem in dispassionately evaluating medication effectiveness. As Ben Goldacre’s book nicely points out, this is by no means restricted to one medical specialty. It is a problem across all areas of pharmacology.
And further: It is also a problem in other areas of science. Psychologists frequently smirk at the problems turned up in pharmacology, but most of the issues are likewise present in clinical psychology research. Outcome trials of psychotherapy are even more difficult to evaluate than pharmaceutical research – if you wanted to include unpublished studies in a review article, you wouldn’t even know where to look.
Despite all this, some people do benefit – both from psychotherapy and from medication. The problem is how to clean the lens so that we can see what really helps a bit more clearly.
You are also completely correct that the same problems appear in the “natural” therapy field – but often to an exaggerated extent. Often basic assumptions are never tested at all, or the studies are conducted extremely badly, or wild inferences are made from minimal data. As we explore some of the problems with the minutiae of the science in treatment evaluation, we often ignore the fact that much of what people do has simply never been examined at all.
One reason that it is so easy (and has become something of a cottage industry) to pick apart the problems in the science (or attempts at science) underlying many psychopharmacologic approaches is that the studies actually exist.
There are problems with many of them, and with the way they are reported, or not, but at least someone is attempting to try out the treatment, see the effects on real patients and real problems, find the dangers, and figure out whether they work. Profit motive, ego, and optimism all serve as distorting lenses overlaid on this work, resulting in many problems and some treatments that seem worse than the placebos against which they are compared.
My own field is psychotherapy, and I can say confidently that if the same level of scrutiny was applied to empirical investigations of therapy, most of the same problems would be seen (as I state in the article above). I don’t throw up my hands and retreat from the office, but I do recognize that much of the literature needs to be looked at carefully.
Recently a chiropractor near me enthusiastically sent a photocopy of an article in a local newsletter suggesting that people entered an “alpha state” shortly after having a subluxation treatment. No control group, no actual outcome measures of real significance, no followup, etc. On the basis of this he was all set to start offering the service. I don’t know whether he has, but this is a familiar example of the level of analysis undertaken before people declare that they have found the “cure to all disease” or “a revolutionary new approach.”
It’s possible to see this a few times and dismiss it, but after 3 decades of seeing the big new cure appear every few months one gets a bit jaded.
There are some alternative or complementary approaches that likely have great value. Melatonin in the short term seems to be very good for jet lag (long term outcome studies have been disappointing). Trials of St John’s Wort for mild to moderate depression have been encouraging, though inconsistent. There are many other examples.
But they need to be tested. It is not enough to come up with a great new idea that seems to make sense. We need to try it out in controlled settings and see if it really works, while controlling for expectancy and other variables. If not, we run the risk of asking people to spend a great deal on treatments that are worthless, or that delay the implementation of something that may be more effective.
I’m not a fan of the degree to which pharmaceuticals are used to deal with psychological distress. I could get over some of that if a hard look at the data suggested that they were nevertheless effective over the long term and produced better quality of life. But at least there is some attempt to see whether they work. The reason many of us are skeptical is not despite the science, but because of it. This same scrutiny needs to be given to other approaches as well.
Thank you for this – I had not read this review. It echoes (and backs up with actual data) much of what Ben Goldacre talks about. The Editor’s comments focus on how pre-registration of trials and mandatory publication are likely to help the situation. There are two problems with this:
1. The obligation to publish does not necessarily deal with the non reporting of nonsignificant primary outcomes, nor the problem identified by these authors of certain outcomes mysteriously going from nonsignificant to significant in the writing process.
2. Goldacre reports that the stated commitments of journals to publish only pre-registered trials has been largely ignored, with journals routinely publishing non registered trials just as they did before.
This is a valuable paper and I would refer interested readers to it.
Actually, I think they are issues of understanding good scientific technique, the problems of statistics and partial reporting, and the policies of peer-reviewed journals. We can be healed, grown, and conscious and still make all of these errors.
By discussing some of the problems in existing science and reporting, it is not my intention to discard science and the need for careful attention to its principles. That puts us on the road back to armchair philosophizing, snake oil, and unmeasurable “energy fields.” Instead, we need a dispassionate examination of what actually works in therapy and pharmacology – and we will only get there with a more careful attention to science basics.
I’m saying that there are flaws in the process that need to be addressed. There will always be flaws, but some of them are sufficiently well-known and obvious that it is bad practice to allow them to continue.
Cochrane reviews and others do often manage to get reasonably complete data sets with pharmaceutical topics, so the problem is not completely intractable. As well, it is unlikely that so many trials have been conducted and gone unpublished (in most areas) that apparent positive findings are entirely due to chance.
Many of healthcare’s most effective and helpful treatments have been found and validated based on existing research and reporting methods, but the knife is much more blunt than it should be, or needs to be. In the area of psychopharmacology there have been particular problems due to weak effects, short trials, an inattention to long-term outcomes, and significant reporting issues – to the point that confidence is often appropriately low in what the science actually says.
So is medicine all nonsense and misinformation? No. When I look at the literature on alternative treatments the situation is generally much worse. But it could be much clearer than it is.
Thank you. It gets worse: I am even pro-medication! Provided they are used in limited quantities, with a clear eye on the hard data, taking into account side effects and long-term disadvantages.
Why just the other day I had a client contemplating their first flight after a few sessions of treatment for flight phobia, and I agreed that using a benzo for that first flight wasn’t a bad idea (provided that the goal was to go benzo-free for the 2nd or 3rd flight – and that the flight should not be the first time they try the med). I suspect that the pharmaceutical companies would go bankrupt if I was prescribing on their behalf, but if I had a prescription pad would it ever get used? Yes, now and then. Not many refills, though.
When we feel strongly about an issue there is always the temptation to gravitate to the poles and look for all-or-nothing, good-or-bad solutions. The complicated truth usually lies somewhere between the extremes. Most psychiatrists I know are dedicated people who genuinely want to help their patients – and in many instances actually do so.
Are there problems in the field? Absolutely. Just as there are problems in psychology and psychotherapy. Will we solve those problems through polarization? Probably not.
A lot of people feel similarly – that healthcare should not be an industry. It is, however. It costs a great deal of money to provide, and somehow it has to be funded. If privately, then by consumers or insurers. If publicly, then by a single customer – the government.
We can dance around the term, but I think this makes it a business – one that can be run ethically, with standards, and under appropriate regulation, or one that exists in an unregulated free-for-all situation. My own view is that if the bridge-building industry was as badly monitored as mental health there would be few bridges standing – but that’s just my opinion.
Do we need it? My own city has a good model of what can happen with no functioning mental health system, and it is not a pretty sight. So yes, I think a society can benefit from mental health services – provided they are adequately researched and based on what hard evidence we can muster.
I think you are right that much of the problem we have currently is cultural in nature. We (the culture generally, and mental health practitioners like me as well) have created a culture in which normal difficulties in life are often pathologized, unhappiness is a disease, and the experience of reality (which is that life is often difficult, tragic, and disappointing) is viewed as a disorder.
If we were to acknowledge that distress and periods of dysfunction are normal, would there be a role for mental health? Yes, I think there still would be. Many of my own clients have no diagnosable “illness” – or, if it might be possible to find them in the DSM, I choose not to do so. Yet they report benefiting from having a trained listener (and sometimes a good-natured confronter) helping them work out the challenges of life in the 21st century.
I disagree that the drugs have nothing to do with healing. I think they are overused, badly studied, and unethically marketed in many, many instances. And there are some drugs that I wish had never been approved for use in humans or animals.
But I have also seen remarkable benefits for some people from some meds – benefits we would never have had if the resources had not been devoted to medication development. I suspect that there is a problem in psychopharmacology: If the medications were prescribed only to those who stand a good chance of truly benefiting, based on moderately well-conducted research, and used in quantities that are actually helpful, the sales figures might not justify the development costs.
So is the money wasted? Perhaps, if the result is financial pressure to overprescribe to the point that a medication winds up doing more harm than good overall. But is good EVER done? Yes, I think so. But not often enough for my taste.
My post this week involves one of the many factors that distort medical decision-making – I hope that I have not implied that it is the only problem!
You are right – virtually all pharmaceutical research is funded, conducted, analyzed, and written up by pharmaceutical companies (or those whom they fund) – the companies that stand to make or lose millions (or billions) depending on the outcomes of the work. The hope that this work will be carried out dispassionately is a slim one.
But yes, there is science. Even in some of the most egregious examples, such as Study 329 (the trial of paroxetine for adolescents that has received so much attention (see http://www.psychologysalon.com/2015/01/science-in-court-study-329-paxil-and.html for background), the data are there to be found, beneath the layer of misinterpretation. The question is how to get greater impartiality in study design and reporting than we have today.
You’re absolutely correct in this. The pharmaceutical industry has produced drugs which have unquestionably saved thousands of lives. Witness, for example, the almost unbelievable success they have had with HIV. Government agencies do not have the funds to develop drugs, and the profit motive is a great motivator for this work to be done. People who work for the pharmaceutical industry are almost unfailingly pleasant people who believe in what they do.
One of my reservations about this site is that people perceive it as anti-psychiatry, and in some posts and articles it genuinely is. I believe this is wrong-headed. Psychiatry, like the pharmaceutical industry, is a field intended to improve mental health. To this extent I am whole-heartedly PRO-PSYCHIATRY.
The problem is that the reward systems we have in place are often imperfect. It is to a company’s very significant advantage, for example, to market a drug beyond the narrow confines of its approved uses, and we have seen much prescribing that should not have happened. As well, there are virtually no brakes on the amount companies can do to promote their products, including funding heavily biased “Continuing Education” talks that should more properly be considered advertisements.
Nor, and I think this is a more significant concern, are there sufficient ethical brakes on the medical profession. If a politician accepted a cruise, meals, golf fees, free samples, and more from a representative of industry hoping for an advantage, that politician would be rightly accused of gross corruption. For a physician to do it is not regarded as problematic. “Ah,” goes the reply, “because physicians have their patients’ interests at heart (true) and could not be swayed by a bottle of wine.” I’d like to agree. But that would imply that the corporations are simply wasting billions on such efforts unnecessarily and getting no benefit. The research strongly suggests otherwise, and surely no one can believe that the companies are so misguided.
We do need psychiatry, and pharmacology, and a profit motive to drive parts of the industry. But like all industries, medicine and pharmacy need carefully-thought-through guidelines and regulation, without which their noble aims can fall victim to shorter-term considerations.
Interestingly, new developments in psychopharmacology are a bit thin across the board. But if a patented treatment is approved for any specific indication, we can indeed expect that not only will the drug be marketed, so will the disorder.
I agree with you that the term does continue to have some stigma associated with it, and that one might not want to put it on a resume. My emphasis, I suppose, is on the idea of nervous breakdown as a discrete episode – an event that happens to a person – whereas most diagnoses “weld” the label to the person’s character. Increasingly, we see a psychiatric diagnosis as a permanent feature rather than a transitory phenomenon. People who have not had a major depressive episode or panic attack for many years continue to be told that they have Major Depressive Disorder or Panic Disorder.
I am not a fan of viewing mental illness as being “just like physical illness,” at least in most respects, because in most instances it is not. But it would be nice if we could adopt at least one aspect of physical diagnosis: transitoriness. If we are diagnosed with an upper respiratory tract infection today, we do not retain the diagnosis next year. If I am depressed today, why should I become “a depressive” next year even though I have no symptoms.
There may be vulnerability. We can all be viewed as chains – pull on us hard enough and one of our links will prove weaker than the others. One person’s weak link may be a proneness to depression, another will have panic, another may become more irritable. It can be useful to know one’s weak links so that we can be on the lookout for them giving way. But to define us by our weak links seems unhelpful.
I can hear your anger, and believe me when I say that I have seen many, many justifications for it, and respect your opinion. I also share it to an extent, but not completely.
I agree that diagnosis often harms more than it helps. I have seen many instances, though, where a bit more care with diagnosis would have helped more than it harmed. The example that comes to mind is bipolar disorder. Our treatments for this problem are clearly inadequate, and the problem itself may (by being published in a diagnostic manual) have taken on more of a sense of reality than it merits. But there is a current diagnostic fad for the problem that has led to many people being so diagnosed who clearly (with the completion of a quick and clear diagnostic interview) do not have it. Whether what we do when we determine that they DO meet criteria is the most helpful thing is another question.
When I see someone with an anxiety problem, it also helps me to draw a distinction between panic disorder and, say, social phobia. I don’t need a rock-solid DSM diagnosis for this, but knowing the primary subject of the fear can help me figure out which direction to pursue first in helping the person face down their fears.
I disagree that no one takes the utility of the sufferer seriously. In my experience, most people (not all) in the various fields are tremendously motivated to help clients. Indeed, it is this desire to help that often causes problems, blinding us to to the painful possibility that we are doing more harm than good.
The mental health system can indeed be the source of suffering, and hopefully the contributions of madinamerica will help us face these and overcome them. Life, though, is inherently difficult, and can produce suffering all on its own. I’m conscious of the Buddhist First Noble Truth, which can be paraphrased as “Life involves suffering” – a sentiment expressed long before any of the mental health professions were founded. It can be helpful, if nothing else, to have someone to whom to express your suffering. And sometimes that person may have some ideas that can go some way to helping the sufferer overcome it. If we can return to that sense of humility about what we do, we will have done well.
Hmm. I’m not sure where I gave the impression that publication is the main or sole problem that mental health faces. In fact, I think that if you read closely I signal that there are many other challenges as well. Like many complex fields, there are multiple problems. Attempting to address them all in one book or, God forbid, blog post would be futile. My intent with this post is simply to point to one problem in particular.
Report comment
Your proposed product is a wonderful example of an easily tested intervention that would not require anything like the investment involved in testing a medication. One reason that many health professionals (myself very definitely included) get so cranky about “alternative healthcare” is that people sit behind ideas that “my novel approach is already empirically supported” (huh?) or “it has a holistic influence and so isn’t appropriate for research.” (I know you aren’t saying this yourself.) But in reality, the intervention you’d like to put in place is ideally suited to a cheap and easy few tests.
You want to bring this product to an institution, and you have specific ideas about why they would want to make that investment. People will get more minutes of sleep, they’ll have fewer violent outbursts – something. Offer to install the equipment at your own cost, and seek ethics approval – you will need a nonvulnerable population and some form of consent. Then have an independent researcher (not a true believer like yourself) randomize the way it is used (by week, month, or whatever), and OTHER observers who do not know the condition rate the outcome variable of interest. Expectancy should not be an enormous issue, but it must be controlled for, and scorers will certainly be influenced by their own beliefs if they know which conditions are in place when.
Do this properly and you should have no difficulty publishing in a reputable journal (yes, such things do exist). But you can guess my proviso. Only do the research if you are committed to publish the outcomes WHATEVER HAPPENS, even if the disappointing results threaten your business. And get help with the research design (the above few sentences don’t count – as you may see from the howls of outrage that I haven’t eliminated every source of bias in my short reply). The last thing we need is another Critical Incident Stress Debriefing, or Past Life Regression Therapy. Or paroxetine for children.
Report comment
Agreed. We have tended to focus on the funders and writers of these trials, and many reading about the problem could get the impression that the journals are the poor victimized innocents in the affair – like the drivers of cars hijacked by escaping bank robbers.
But the abject failure of many of the journals to live up even to their own declarations of “policy” doesn’t support this view of the problem. They are driving the getaway car with full knowledge and consent, and can only be viewed as accomplices in an endeavour which has a negative impact on many lives.
Report comment
I agree that patient opinions need to be paid attention to, and the field has been quite paternalistic. “Quite” is, admittedly, an understatement.
You raise an interesting possibility: What if an independent body could rate journals not in terms of citation count, but in terms of adherence to specific standards of evidence. Things like willingness to publish negative results, unwillingness to publish short-term trials of meds for long-term conditions, and disqualification of studies not previously declared or registered. Then my only-half-facetious argument against reading journals could be more finely tuned, and physicians could make a point of reading better research and ignoring infomercials.
Of course, there’s still the issue of sponsored lunches, clinic “educational” visits, and funded talks at professional conferences…
Report comment
Thank you for your comment. I agree with the points you have made here. And you are quite right to point out that the same criticisms could be made of psychotherapy outcome trials – publication bias in favour of positive trials, and a tendency of researchers to shelve nonsignificant results rather than attempting to publish.
As well, we in psychology do seem to have become obsessed with demonstrating the effectiveness of specific technique-based therapy models in the absence of good evidence of unique effects (for the most part – specific strategies for panic attacks, and exposure-based therapies for some anxiety disorders do appear to be more powerful than free-form counselling). It is as though we have spent 30 years arguing over the right colour of cardigan to wear in the consulting room, when this clearly is not the prime determinant of outcome.
Report comment
My focus on Study 329 could be criticized as a slander to the field, attacking as it does the low-hanging fruit of one of the most criticized trials (and publications) in psychiatric history. The truth is somewhat more troubling: Study 329 is simply a recognized example of a much more pervasive problem in the literature.
You point out that as an 8-week trial it should have been disqualified from consideration out of hand. Well, certainly it has little to say about long-term outcome, which is the primary variable of interest. But short-term trials of 8-16 weeks are pretty much the norm in psychopharmacology – in large part because this is all the regulatory agencies demand, and the point of the studies is to get regulatory approval.
If we dismissed all the short-term studies of medications intended for long-term problems we would thin out the psychopharmacology literature to a very surprising extent, and force everyone back to the labs to look at the outcomes of real interest. I’ll leave it to you to guess whether I think that would be a good thing.
Report comment
I would recommend Goldacre’s book Bad Pharma, which summarizes the problems in much pharmaceutical research. As well, Whitaker’s Anatomy of an Epidemic (on psychopharmacology) and Kirsch’s The Emperor’s New Drugs (which emphasizes antidepressants). From these you can access individual pieces on various problems in the literature. I’ve written a lot on the issue as well, and you can find more at my blog, psychologysalon.com, and click on the keyword “Medication”.
Given your request, I am providing references to discussions of the problems, and I am very conscious on this site of coming across as a dismissive across-the-board “it’s all nonsense” skeptic. I am skeptical of many claims in mental health, but would not consider myself anti-psychiatry, anti-psychology, or anti-therapy – particularly given that I am a psychologist and psychotherapist. I am not one who feels we need to get rid of the mental health field, but to improve it. In part, my suggestion that journal-reading is unproductive is meant as a goad for things to get better, not to disregard the quest for effective, evidence-based approaches to genuine problems.
Report comment
Although much of the press and the awareness campaign (and hence my post) is based on pharmaceutical research, exactly the same problem pervades much of science, including my own field of clinical psychology.
I think you’re right, that we should have not only the writeups, but access to original data, and this is part of the push right now with alltrials.net and other sources.
You mention the idea of studies not passing the sniff test, and many in fact do not. Study 329 is a prime example of this, and it is hardly an isolated case. But even if a published study is perfectly run and reported and passes the sniff test with flying colours (odours?), the bias problem means that it will not be balanced out by other well-run studies showing different effects.
In years past, journals were sharply limited by printing and distribution costs, resulting in a necessary funnelling and selection effect. With the Internet, such concerns are no longer relevant, and there is no reason for a journal to decline publication of a study showing null results, provided it is well done. (The problem, of course, is that more reviewers would be needed, but this seems to be a concern that could be overcome.) Journals could even have a separate section for “Null Results” or “Failures to Replicate” so that their exciting positive studies are not obscured.
One barrier is the ongoing (and likely temporary) phenomenon of dual publishing (paper and net). It probably won’t be long before most journals cease the print version, thus freeing themselves up.
Report comment
Hi Fred, Thanks for your comment. I think you’re right that the anger most people feel relates to the misuse of science, rather than to science itself. Jeffrey Lieberman has characterized much of the criticism mental health has received as “anti-science” which strikes me as a) a misperception of what is being said, and b) a stance that some get tempted by when they learn that science is not as pristine as sometimes advertised.
I’m struck by how often I hear the equivalent of “The science isn’t perfect so I’m going with iridology instead.” There is so much pseudo-science in the alternative medicine field one wants to tear one’s hair out. The one advantage some of it has (not to be sneezed at) is that inert treatments are often at least not damaging (though sometimes they are harmful as well). I don’t mean by saying this that all “alternative” practices are bunk – that seems clearly false and much of what I do would be regarded as alternative by many – but that we should also have some humility and scientific curiosity about these as well.
For most of us, most of the time, under most circumstances, we will figure things out on our own, or with the help and support of family and friends. We have become rather too quick to assume that difficulties in life are diseases, or can be treated with a pill (or a cognitive challenging exercise).
There remain certain situations, though, in which our coping resources are overwhelmed or we run out of ideas, and at that time an external viewer/advisor can often be useful. The industrialization of support has been problematic (an understatement, I know). But I continue to believe that, when this happens, external help can have a significant role to play.
Report comment
You’re right, there are many problems – too many to cover in any single article. This makes it a challenge to discuss any one of them, as I frequently get jumped on for “naively thinking this is the only problem” when I pick one to look at!
One of the very significant problems that I haven’t discussed is the one you focus on here – that medication effects get relabeled as disease effects. The diagnosis of bipolar disorder following antidepressant-induced mania is a good example.
Report comment
You are right that industry interests are likely to oppose greater transparency in medication research, and that they have been largely successful to date. Hopefully through authors like Goldacre, Healy, and Whitaker, and well-educated medical reporters, and sites like this one, the awareness of the problems and the push to correct them will move things in a more pro-science direction.
The concern I have is that there will simply be a polarization – between the cheerleaders like Jeffrey Lieberman (Shrinks) touting or grossly exaggerating the successes and ignoring the problems, and the “anti-psychiatry” forces wanting to throw it all out and practice Reiki. Rage is a powerful drug, and it’s easy to feel its pull.
Report comment
Not at all. Studies that are ‘garbage’ have often been a problem, but many of the issues are more subtle.
In the case that I am describing, we can have perfectly good studies – not garbage at all – and incorporate them into a properly-analyzed and dispassionately reported document, and still wind up with a distortion in the data.
Thanks for reading.
Report comment
One of the limiting factors in the publication of nonsignificant or contradictory results has been the sheer cost of printing and distribution. I think that the development of online journals (preferably with peer review) will eventually change a great deal of how science is conducted and reported. There is no longer any problem with a journal “issue” of 500 pages, and editors will still be able to “highlight” the articles they think are most interesting.
This won’t affect the problem of selective publication by authors, unfortunately. This is a challenge best met through other means – like editors standing by their previously-announced trial registry policies.
Report comment
You are correct on several counts. Madinamerica generally tries to raise issues with the science side of mental health – ideally with the goal of prompting greater awareness of and adherence to the principles of science within these professions. It runs the risk, however, of becoming a magnet for conspiracy theories and contributing to an anti-science forum of rage. (Watch, for example, the replies to that statement.)
The involvement of pharmaceutical companies in researching their own products, coupled with an extreme version of the profit motive, contributes to much of the problem in dispassionately evaluating medication effectiveness. As Ben Goldacre’s book nicely points out, this is by no means restricted to one medical specialty. It is a problem across all areas of pharmacology.
And further: It is also a problem in other areas of science. Psychologists frequently smirk at the problems turned up in pharmacology, but most of the issues are likewise present in clinical psychology research. Outcome trials of psychotherapy are even more difficult to evaluate than pharmaceutical research – if you wanted to include unpublished studies in a review article, you wouldn’t even know where to look.
Despite all this, some people do benefit – both from psychotherapy and from medication. The problem is how to clean the lens so that we can see what really helps a bit more clearly.
You are also completely correct that the same problems appear in the “natural” therapy field – but often to an exaggerated extent. Often basic assumptions are never tested at all, or the studies are conducted extremely badly, or wild inferences are made from minimal data. As we explore some of the problems with the minutiae of the science in treatment evaluation, we often ignore the fact that much of what people do has simply never been examined at all.
Report comment
One reason that it is so easy (and has become something of a cottage industry) to pick apart the problems in the science (or attempts at science) underlying many psychopharmacologic approaches is that the studies actually exist.
There are problems with many of them, and with the way they are reported, or not, but at least someone is attempting to try out the treatment, see the effects on real patients and real problems, find the dangers, and figure out whether they work. Profit motive, ego, and optimism all serve as distorting lenses overlaid on this work, resulting in many problems and some treatments that seem worse than the placebos against which they are compared.
My own field is psychotherapy, and I can say confidently that if the same level of scrutiny was applied to empirical investigations of therapy, most of the same problems would be seen (as I state in the article above). I don’t throw up my hands and retreat from the office, but I do recognize that much of the literature needs to be looked at carefully.
Recently a chiropractor near me enthusiastically sent a photocopy of an article in a local newsletter suggesting that people entered an “alpha state” shortly after having a subluxation treatment. No control group, no actual outcome measures of real significance, no followup, etc. On the basis of this he was all set to start offering the service. I don’t know whether he has, but this is a familiar example of the level of analysis undertaken before people declare that they have found the “cure to all disease” or “a revolutionary new approach.”
It’s possible to see this a few times and dismiss it, but after 3 decades of seeing the big new cure appear every few months one gets a bit jaded.
There are some alternative or complementary approaches that likely have great value. Melatonin in the short term seems to be very good for jet lag (long term outcome studies have been disappointing). Trials of St John’s Wort for mild to moderate depression have been encouraging, though inconsistent. There are many other examples.
But they need to be tested. It is not enough to come up with a great new idea that seems to make sense. We need to try it out in controlled settings and see if it really works, while controlling for expectancy and other variables. If not, we run the risk of asking people to spend a great deal on treatments that are worthless, or that delay the implementation of something that may be more effective.
I’m not a fan of the degree to which pharmaceuticals are used to deal with psychological distress. I could get over some of that if a hard look at the data suggested that they were nevertheless effective over the long term and produced better quality of life. But at least there is some attempt to see whether they work. The reason many of us are skeptical is not despite the science, but because of it. This same scrutiny needs to be given to other approaches as well.
Report comment
Thank you for this – I had not read this review. It echoes (and backs up with actual data) much of what Ben Goldacre talks about. The Editor’s comments focus on how pre-registration of trials and mandatory publication are likely to help the situation. There are two problems with this:
1. The obligation to publish does not necessarily deal with the non reporting of nonsignificant primary outcomes, nor the problem identified by these authors of certain outcomes mysteriously going from nonsignificant to significant in the writing process.
2. Goldacre reports that the stated commitments of journals to publish only pre-registered trials has been largely ignored, with journals routinely publishing non registered trials just as they did before.
This is a valuable paper and I would refer interested readers to it.
Report comment
Actually, I think they are issues of understanding good scientific technique, the problems of statistics and partial reporting, and the policies of peer-reviewed journals. We can be healed, grown, and conscious and still make all of these errors.
By discussing some of the problems in existing science and reporting, it is not my intention to discard science and the need for careful attention to its principles. That puts us on the road back to armchair philosophizing, snake oil, and unmeasurable “energy fields.” Instead, we need a dispassionate examination of what actually works in therapy and pharmacology – and we will only get there with a more careful attention to science basics.
Report comment
I’m saying that there are flaws in the process that need to be addressed. There will always be flaws, but some of them are sufficiently well-known and obvious that it is bad practice to allow them to continue.
Cochrane reviews and others do often manage to get reasonably complete data sets with pharmaceutical topics, so the problem is not completely intractable. As well, it is unlikely that so many trials have been conducted and gone unpublished (in most areas) that apparent positive findings are entirely due to chance.
Many of healthcare’s most effective and helpful treatments have been found and validated based on existing research and reporting methods, but the knife is much more blunt than it should be, or needs to be. In the area of psychopharmacology there have been particular problems due to weak effects, short trials, an inattention to long-term outcomes, and significant reporting issues – to the point that confidence is often appropriately low in what the science actually says.
So is medicine all nonsense and misinformation? No. When I look at the literature on alternative treatments the situation is generally much worse. But it could be much clearer than it is.
Report comment
Thank you. It gets worse: I am even pro-medication! Provided they are used in limited quantities, with a clear eye on the hard data, taking into account side effects and long-term disadvantages.
Why just the other day I had a client contemplating their first flight after a few sessions of treatment for flight phobia, and I agreed that using a benzo for that first flight wasn’t a bad idea (provided that the goal was to go benzo-free for the 2nd or 3rd flight – and that the flight should not be the first time they try the med). I suspect that the pharmaceutical companies would go bankrupt if I was prescribing on their behalf, but if I had a prescription pad would it ever get used? Yes, now and then. Not many refills, though.
When we feel strongly about an issue there is always the temptation to gravitate to the poles and look for all-or-nothing, good-or-bad solutions. The complicated truth usually lies somewhere between the extremes. Most psychiatrists I know are dedicated people who genuinely want to help their patients – and in many instances actually do so.
Are there problems in the field? Absolutely. Just as there are problems in psychology and psychotherapy. Will we solve those problems through polarization? Probably not.
Report comment
A lot of people feel similarly – that healthcare should not be an industry. It is, however. It costs a great deal of money to provide, and somehow it has to be funded. If privately, then by consumers or insurers. If publicly, then by a single customer – the government.
We can dance around the term, but I think this makes it a business – one that can be run ethically, with standards, and under appropriate regulation, or one that exists in an unregulated free-for-all situation. My own view is that if the bridge-building industry was as badly monitored as mental health there would be few bridges standing – but that’s just my opinion.
Do we need it? My own city has a good model of what can happen with no functioning mental health system, and it is not a pretty sight. So yes, I think a society can benefit from mental health services – provided they are adequately researched and based on what hard evidence we can muster.
I think you are right that much of the problem we have currently is cultural in nature. We (the culture generally, and mental health practitioners like me as well) have created a culture in which normal difficulties in life are often pathologized, unhappiness is a disease, and the experience of reality (which is that life is often difficult, tragic, and disappointing) is viewed as a disorder.
If we were to acknowledge that distress and periods of dysfunction are normal, would there be a role for mental health? Yes, I think there still would be. Many of my own clients have no diagnosable “illness” – or, if it might be possible to find them in the DSM, I choose not to do so. Yet they report benefiting from having a trained listener (and sometimes a good-natured confronter) helping them work out the challenges of life in the 21st century.
Report comment
I disagree that the drugs have nothing to do with healing. I think they are overused, badly studied, and unethically marketed in many, many instances. And there are some drugs that I wish had never been approved for use in humans or animals.
But I have also seen remarkable benefits for some people from some meds – benefits we would never have had if the resources had not been devoted to medication development. I suspect that there is a problem in psychopharmacology: If the medications were prescribed only to those who stand a good chance of truly benefiting, based on moderately well-conducted research, and used in quantities that are actually helpful, the sales figures might not justify the development costs.
So is the money wasted? Perhaps, if the result is financial pressure to overprescribe to the point that a medication winds up doing more harm than good overall. But is good EVER done? Yes, I think so. But not often enough for my taste.
Report comment
My post this week involves one of the many factors that distort medical decision-making – I hope that I have not implied that it is the only problem!
You are right – virtually all pharmaceutical research is funded, conducted, analyzed, and written up by pharmaceutical companies (or those whom they fund) – the companies that stand to make or lose millions (or billions) depending on the outcomes of the work. The hope that this work will be carried out dispassionately is a slim one.
But yes, there is science. Even in some of the most egregious examples, such as Study 329 (the trial of paroxetine for adolescents that has received so much attention (see http://www.psychologysalon.com/2015/01/science-in-court-study-329-paxil-and.html for background), the data are there to be found, beneath the layer of misinterpretation. The question is how to get greater impartiality in study design and reporting than we have today.
Report comment
You’re absolutely correct in this. The pharmaceutical industry has produced drugs which have unquestionably saved thousands of lives. Witness, for example, the almost unbelievable success they have had with HIV. Government agencies do not have the funds to develop drugs, and the profit motive is a great motivator for this work to be done. People who work for the pharmaceutical industry are almost unfailingly pleasant people who believe in what they do.
One of my reservations about this site is that people perceive it as anti-psychiatry, and in some posts and articles it genuinely is. I believe this is wrong-headed. Psychiatry, like the pharmaceutical industry, is a field intended to improve mental health. To this extent I am whole-heartedly PRO-PSYCHIATRY.
The problem is that the reward systems we have in place are often imperfect. It is to a company’s very significant advantage, for example, to market a drug beyond the narrow confines of its approved uses, and we have seen much prescribing that should not have happened. As well, there are virtually no brakes on the amount companies can do to promote their products, including funding heavily biased “Continuing Education” talks that should more properly be considered advertisements.
Nor, and I think this is a more significant concern, are there sufficient ethical brakes on the medical profession. If a politician accepted a cruise, meals, golf fees, free samples, and more from a representative of industry hoping for an advantage, that politician would be rightly accused of gross corruption. For a physician to do it is not regarded as problematic. “Ah,” goes the reply, “because physicians have their patients’ interests at heart (true) and could not be swayed by a bottle of wine.” I’d like to agree. But that would imply that the corporations are simply wasting billions on such efforts unnecessarily and getting no benefit. The research strongly suggests otherwise, and surely no one can believe that the companies are so misguided.
We do need psychiatry, and pharmacology, and a profit motive to drive parts of the industry. But like all industries, medicine and pharmacy need carefully-thought-through guidelines and regulation, without which their noble aims can fall victim to shorter-term considerations.
Report comment
Interestingly, new developments in psychopharmacology are a bit thin across the board. But if a patented treatment is approved for any specific indication, we can indeed expect that not only will the drug be marketed, so will the disorder.
Report comment
I agree with you that the term does continue to have some stigma associated with it, and that one might not want to put it on a resume. My emphasis, I suppose, is on the idea of nervous breakdown as a discrete episode – an event that happens to a person – whereas most diagnoses “weld” the label to the person’s character. Increasingly, we see a psychiatric diagnosis as a permanent feature rather than a transitory phenomenon. People who have not had a major depressive episode or panic attack for many years continue to be told that they have Major Depressive Disorder or Panic Disorder.
I am not a fan of viewing mental illness as being “just like physical illness,” at least in most respects, because in most instances it is not. But it would be nice if we could adopt at least one aspect of physical diagnosis: transitoriness. If we are diagnosed with an upper respiratory tract infection today, we do not retain the diagnosis next year. If I am depressed today, why should I become “a depressive” next year even though I have no symptoms.
There may be vulnerability. We can all be viewed as chains – pull on us hard enough and one of our links will prove weaker than the others. One person’s weak link may be a proneness to depression, another will have panic, another may become more irritable. It can be useful to know one’s weak links so that we can be on the lookout for them giving way. But to define us by our weak links seems unhelpful.
Report comment
I can hear your anger, and believe me when I say that I have seen many, many justifications for it, and respect your opinion. I also share it to an extent, but not completely.
I agree that diagnosis often harms more than it helps. I have seen many instances, though, where a bit more care with diagnosis would have helped more than it harmed. The example that comes to mind is bipolar disorder. Our treatments for this problem are clearly inadequate, and the problem itself may (by being published in a diagnostic manual) have taken on more of a sense of reality than it merits. But there is a current diagnostic fad for the problem that has led to many people being so diagnosed who clearly (with the completion of a quick and clear diagnostic interview) do not have it. Whether what we do when we determine that they DO meet criteria is the most helpful thing is another question.
When I see someone with an anxiety problem, it also helps me to draw a distinction between panic disorder and, say, social phobia. I don’t need a rock-solid DSM diagnosis for this, but knowing the primary subject of the fear can help me figure out which direction to pursue first in helping the person face down their fears.
I disagree that no one takes the utility of the sufferer seriously. In my experience, most people (not all) in the various fields are tremendously motivated to help clients. Indeed, it is this desire to help that often causes problems, blinding us to to the painful possibility that we are doing more harm than good.
The mental health system can indeed be the source of suffering, and hopefully the contributions of madinamerica will help us face these and overcome them. Life, though, is inherently difficult, and can produce suffering all on its own. I’m conscious of the Buddhist First Noble Truth, which can be paraphrased as “Life involves suffering” – a sentiment expressed long before any of the mental health professions were founded. It can be helpful, if nothing else, to have someone to whom to express your suffering. And sometimes that person may have some ideas that can go some way to helping the sufferer overcome it. If we can return to that sense of humility about what we do, we will have done well.
Report comment