An Approach to Making Sense of Psychiatric Research

A few weeks ago on the MIA site, Robert Whitaker analyzed a recently-published review of studies of antipsychotic use. The review concluded that there’s not much evidence that antipsychotics help much in the short term. Since they’re so commonly used, you’d think that the evidence would be pretty strong backing up their use. You might also think that anyone reading that review who prescribes antipsychotics ought to start re-thinking what they do. We’ll see about that.

Whitaker’s article generated a variety of comments, including one from me, my first contribution to MIA. There were several responses to my comments, and I found these exchanges challenging and thoughtful.

I’ve practiced psychiatry for over 4 decades, mostly in outpatient settings, and I’m firmly convinced that scientific approaches to psychiatry (and medicine in general) are helpful and necessary. But science is far from being some revealed truth that should be worshipped, so we have to figure out where it belongs in mental health care. I’m also convinced that both psychiatrists and others have all kinds of misunderstandings and biases about what science says. That is one of the important obstacles to working in a collaborative way. My goal here is to shed some light on how psychiatrists try to use science. Some of that helps patients. Some of it doesn’t. I also want to say some things about what makes for good science. Good science wants to move us closer to truth, and understands that even being proven wrong is a step forward.

Full disclosure, I don’t consider myself a scientist in the usual sense, but I know a lot about what makes scientific findings more valid and useful. Oddly enough, some findings may be valid but not useful. People are complicated and nobody wants treatment that is dictated by science alone (the human connection is essential). Besides, that’s a very boring and unsatisfying way to practice psychiatry!

No doubt science is not the only way we know things. Some things we know using logic. Some things we intuit or just have a feeling about. Some things we’re taught and we accept them as knowledge. Some things we can know using our five senses. Some may come from religious training.

For some things, using a scientific approach is best. The big question is what are those things. Sherlock Holmes said that if you rule out all the other possibilities, then the one that’s left must be the truth. The hard part is how do you rule out those others, and can you be sure you’re not missing still others? All the other ways of knowing are more based on our individual experiences. Science tries to say—not always successfully—“let’s not make this personal and let’s try to agree on a way to narrow down our question”. If person A had a terrible experience with an antipsychotic, and person B had a wonderful one, what can we conclude? Only that antipsychotics can do good and bad things. It’s not much help to person C. The more experiences with antipsychotics we can collect, the stronger conclusions we can draw. So what’s the best way we know to collect that information?

The randomized, double-blind, placebo-controlled trial (RDBPCT) is the best way we know of to try to isolate one factor from the others. It’s not the only way, but other ways make it harder to draw valid conclusions. Back in the early 20th century, they were doing political polling by calling random people and asking them questions. Seems unbiased and fair, right? But back then, not everybody had a telephone at home, not everyone could afford them. So the “random sample” was biased—and not on purpose—to get opinions from people in better financial shape. What about other people? They were left out, so all you could say is that richer people tended to support candidate X. But that’s all you could say. You couldn’t say anything about the whole population.

Studies that aren’t randomized, or double-blind, or placebo-controlled can still shed light on things, but you have be even more cautious about drawing conclusions. Same applies to case studies, or other reports of individual experiences with psychiatric care.

Back to antipsychotics. I’m not going to re-analyze either the specific studies reviewed by Whitaker, or the original authors’ conclusions here. I am going to say a few things about what goes into doing a good study, and the importance of interpreting studies in an even-handed way. I’ll refer to Whitaker’s report at times here.

A study that compares 2 treatments, or a treatment with placebo should strive to compare apples to apples. This is harder than it seems. How similar or identical do the apples have to be? What differences can we assume don’t matter?

A designer of a RDBPCT says “let’s (1) find a group to study that are a similar as they can be, (2) divide them in half randomly, and (3) give each half either treatment or placebo, (4) be sure no one knows who got treatment or placebo, and then (5) measure to see if there’s a difference or not.” Seems simple enough, but let’s go through these factors a little.

If we want to study a population of people with schizophrenia or schizoaffective disorder ( like the study we’ve been referring to), we need to be sure the diagnosis is accurate, meaning that we trust someone to make the diagnosis, including not missing other diagnoses that might muddy the waters. The more people you have making diagnoses, the more the chance that errors are being made, that the diagnostic process may vary. (That’s even more of a concern with studies collecting data from different countries and cultures.) I know we have the DSM, but even that leaves room for interpretation. We can either trust the “apples’ designation fully, partly, or not at all.

And it’s not just collecting people with that diagnosis. It’s being as sure as you can that those people are similar in some other ways, because some of those other ways might affect how they do with treatment or placebo. Because we know that socioeconomic, medical and ethnic factors impact schizophrenia, we’d like to have all of these “apples’ similar in that way. Some factors we can think of might not matter; we probably don’t need an equal number of right- and left-handed people. And they don’t need to have the same religion, as far as we know so far.

Some people might be recruited for the study but decide not to participate. Might they be different in some important way from those who opt in? Could be.

Are the subjects for the study being paid? If you’re paid, perhaps you’re more likely to want to not only complete the study, but also to give answers that you think might make the researchers happy (even if you’re only guessing about that). We’ve known for a long time that research subjects are prone to this ( even if not purposely).

True randomization ought to be pretty neat and clean, but what if, once randomized, some people drop out of the study? This almost always happens to some degree, and it’s going to affect the results. Let’s say we study 100 people, 50 in each group. Suppose we find group A has better outcomes than group B, but half of group A dropped out and only 3 did in group B. That’ll need an explanation.

It may be the case that no one knows who got placebo or active drug, but we know that drugs often have side effects. Placebos can, too. How can we be sure that side effects don’t affect the outcome measures? If I’m getting side effects and I think that means I’m getting “the real thing,” I’m likely to let that affect my view of how I’m doing when the researchers ask me. Also, how do the researchers know if I’m really taking the treatment in the prescribed way?

And who decides what outcome measures to use? There are oodles of validated, standard rating scales for almost all kinds of psychiatric disorders, including schizophrenia. And measures to identify side effects, too. Both provide a way to collect information researchers hope will show the pros and cons of the treatment. But a rating scale might not capture what is meaningful to a patient with a given condition. It might, say, show that a person’s hallucinations are decreased, but maybe that doesn’t matter to that person as much as whether, say, they’re getting along better with their family. To the researcher, the person is better, but the patient, if asked, might say they see no difference. But that will go in the “benefited” column.

Oh, and who is paying for the study? A big pharmaceutical company? Are they putting their thumb on the scale in some way? Maybe we should say that any study funded by them is automatically one we should ignore. I wouldn’t say that so absolutely, but I do want to know that from the get-go. And we know that drug companies have been known to prevent publication of negative studies of their drugs; there are some ways around that, but it’s complicated and beyond the scope here.

Someone might read this and understandably conclude that doing RDBPCTs are so full of problems that even trying them is doomed. I know I can find flaws in just about any study I see. And I’ve not listed here every kind of flaw I can think of.

So here’s where I come down on this. What follows applies to other studies that are not RDBPCTs. I think it applies also to other medical and psychotherapy studies.

We—all of us—in every part of our lives—have to decide who’s trustworthy and who’s not. Some of us are trustworthy about some things, but not so much about others. The more someone (or some part of society, like psychiatric care) gives you reasons not to trust them, the more you either have to erase them from your life or take some time with that person (or that psychiatric study) to decide if you can trust them in that instance. If you like your psychiatrist, but s/he doesn’t return your call in a timely way, you can get a new one, or decide that the benefits of that person outweighs this flaw. And so it is with research. If finding a flaw is only about a “gotcha” moment, take a step back.

Very few people are really skilled in making sense of published psychiatric research. It’s probably an open secret that that applies to most psychiatrists (therapists, too). Few would admit it, but I believe the vast majority of us read relatively few studies from beginning to end. And all of those statistics will make most eyes glaze over. Most will either read the introduction and the conclusion to get the gist of it, trusting that the study didn’t have huge flaws. Most will trust the editors of the journal to have done that work for them. If I’m familiar with one or more of the researchers, or the place where the study was done, that’ll affect how much faith I have in the study.

But what about non-psychiatrists who are trying to make sense of RDBPCTs and other research? A few things to think about:

(1) Never rely on the title of the article alone. This also applies to headlines in social media, TV reports, and any other shorthand story. They’re too often after your attention, not about educating you.

(2) Think about how much you trust the source of the information about the study. Your psychiatrist, or therapist, or family doctor may—or may not—be a reliable interpreter of some seemingly important research. If you read Whitaker’s analysis, how much did you trust him? How much did he trust the article he analyzed?

(3) I said earlier that every study has flaws. Don’t let that be an automatic disqualifier of its findings. Many important and patient-helpful studies have flaws, and if we just ignored them because of that, there’d be more suffering than there is now. Ditto for a Pharma-sponsored study. We all have our own flaws, and we’d not take it well if our own work or words were discounted because of that.

(4) Some people, based on their experiences, have some powerful criticisms of psychiatry, and some are cynical, too. Fair enough, and so to think that that might make someone more likely to see (and perhaps magnify) flaws in research shouldn’t be a shocker. We are all prone to see what we want to see and not see what goes against what we already believe. Just human nature.

(6) It’s not often noted, but almost every study I’ve ever read has a section in which the authors discuss (a) what they already know to be some of the flaws in the study (b) how those flaws limit the strength of the conclusions they draw and (c) that their study is not definitive and how it should stimulate more studies. I can’t think of another line of work where the workers routinely include some self-criticism and humility about what they’ve done. I think that’s praiseworthy, and you won’t hear about that on TV or in social media posts about these studies.

(7) It’s pretty hard to do good scientific research. Imperfect but good research has gone a long way, in my view, to make things better for many people. I wish it were even more helpful, but I think that reflects, mostly, how hard it is to understand humans and the tedious detail involved in doing any good work. And I’m not so naive as to think that research can’t be misused, or that some make a lot of money doing it. They don’t have to be saints, but then again, thankfully, neither do you or I.

My bottom line echoes what you might remember Ronald Reagan saying in a very different context: “Trust, but verify.” I presume a researcher or research team is motivated more by trying to do good than to do evil until I have evidence—more than a gut feeling or a general distrust—otherwise. I expect to find flaws, and then I need to discern if the flaws are minor or worse.

Usually, what I want to know in these studies are these: (1) how close to apples vs apples did they get? (2) are the outcome measures meaningful and did they look at function, not just symptoms? (3) how bad were the downsides of the treatment? (4) were there a lot of dropouts—and why? (5) was the study done for a long-enough time? Depending on the study, I probably will wonder about other things, especially whether it should change something about how I work with patients.