How Do We Know When a Treatment Works? A Primer on the Scientific Method

25
1447

The layperson often has some difficulty understanding the nuance of what “science” is. Science, in its broadest sense, is not the objective truth—rather, it is a method for determining the truth. There are many ways of finding truth, including personal experience and anecdotal reports, but the scientific method is, at its best, the most rigorous method, one which others can replicate in order to confirm the legitimacy of findings.

In this discussion I am limiting the topic to how the application of science can determine the truth about the efficacy and safety of health treatments. Scientific procedures to determine treatment efficacy are in sharp contrast to what might be called clinical opinion or everyday clinical evidence. 

It is not that clinical opinion is necessarily wrong, but if it is not backed up by the scientific method, then there is no way of verifying that it is correct. Indeed, one of the major issues with clinical opinion is that it does not provide a “control group”—a group of people who are not given the drug or who are given a placebo.

If a doctor gives a drug and then notices that the patient has improved, the doctor’s clinical opinion is likely to be that the drug works. However, the patient might also have improved if he did not take the drug, or if he simply believed he was taking an effective drug (the placebo effect). Clinical opinion has no way to “control” for this potential factor—but the scientific method does. It’s called the “randomized, placebo-controlled trial,” or RCT.

The best way to control for researcher bias is with the double-blind RCT. This means that neither the trial subjects nor the researchers know who is getting the drug and who is getting the placebo. Without the double-blind, patients who know they are getting the placebo might not experience the full effect, and researchers might subconsciously rate subjective criteria as more improved if they know the patient is on the real drug.

Unfortunately, the blind is often compromised because the researchers—and often, the subjects—are aware of the drug’s side effects. Thus, the people who experience side effects know they are on the real drug. This problem can be avoided by using active placebos, a drug with side effects but which does not treat the health problem in question. But this is rarely done. For instance, I don’t believe an active placebo was ever used in a clinical trial for an FDA-approved depression drug.

Though there are many issues with the US FDA’s drug-approval process—it does not always work as intended—ideally, it should synthesize the results of RCTs so that only drugs that prove to be better than placebo (ie, better than nothing) get approval.

Off-Label Prescribing

When your physician prescribes you a drug off-label—one not FDA-approved for the condition being treated—you and your physician are both in the dark about the drug’s true efficacy and safety. This is not a minor problem, because it is estimated that 20% of the drugs on the market are prescribed off-label, meaning the FDA did not approve this drug for this specific health problem.

At the very least, the patient should be informed that the drug is being prescribe off-label (so that they know there is not enough evidence that the drug is better than nothing), but I doubt physicians rarely inform the patient of such.

I suspect physicians justify this by never wanting to cast any doubt on their drug recommendations, which might limit the drug’s efficacy! Physicians are very aware of the powerful placebo effect, and they know that if they let the patient know that the treatment may not work, the patient may not benefit as much.

Another concern is the risk/benefit ratio. All treatments have potential harms, and most have potential benefits. For example, Depakote is an anticonvulsive medication that has very severe side effects. When used for seizure prevention, the risk/benefit ratio may be appropriate, because in some cases, seizures are life-threatening. But Depakote is also offered off-label for pain relief, which may not be worth it because the harms of the medication have the potential to be more severe than the condition it’s treating!

A neurologist friend offered me a prescription for Depakote for my left leg pain, but I declined by saying, “I like my brain too much”!

Patient Satisfaction

The “snake oil” salesman used to report that 90% of their users were very satisfied with the results of treating dozens of health problems with their phony remedies. How can we account for the popularity of these treatments? There were two factors: the placebo effect and, often, the effect of drugs like alcohol or cocaine. The alcohol/cocaine in the treatment certainly made the patient feel better quickly, which contributed largely to the patient’s satisfaction with the treatment.

The charlatan’s drug gave the patient hope of improvement, which reduced the patient’s stress, which in turn perhaps strengthened the immune system to fight the health problem. Perhaps more importantly, many health problems are self-limiting—that is, the patient gets better with or without treatment (eg, the flu, upper respiratory infections, and, often, “depression”).

In short, patients are often satisfied with placebos because their symptoms improve naturally. But if a medical treatment only works due to the placebo effect, it exposes patients to the harms of that treatment unnecessarily.

What Reduction in Symptoms Is Scientifically Acceptable?

Physicians and patients would both prefer if the treatment resulted in “remission”—the complete absence of the health problem. However, such a goal is unobtainable for many health problems in the real world. In the real world, patients and their prescribing doctors both have a right to be pleased if the prescribed treatment substantially reduces the magnitude of distressing symptoms.

But what percentage of reduction is appropriate and clinically meaningful? As one trained in science and statistics I must depart both from the FDA and many physicians who generally have accepted a lower standard of efficacy than can be justified by a reasonable risk/benefit ratio.

The appropriate level of efficacy depends on many factors, including the severity of the health problem as well as the adverse effect profile of the prescribed treatment. With a life-threatening health condition, a lower standard of efficacy would be appropriate. Not so for a less-serious health problem, which should demand a substantially higher standard.

Take, for example, antipsychotic drug treatment of “schizophrenia” and psychosis. One large meta-analysis found that all antipsychotic drugs were superior to placebo, with moderate effect sizes in reducing symptoms, which may sound pretty good. However, 51% of the antipsychotic group experienced only a “minimal” response as opposed to 30% in the placebo group. Only 23% of the drug-treated group had a “good” response (compared with 14% of the placebo group).

But I believe this is the most important finding in the study: Since the 1960s, in these antipsychotic trials, the average improvement in those receiving the placebo has increased, while the efficacy has decreased for those receiving the antipsychotic medication.

Why this change? The investigators had some hypotheses to explain the decline of antipsychotic drug efficacy over time, including the notion that the longer people take the drugs, the worse the outcome. The researchers intended this to reassure us about the drugs’ efficacy, but this actually means that antipsychotic drugs have very negative effects after long-term use—in fact, they cause irreversible structural damage to the patient’s brain!

For a first-episode psychotic break, the antipsychotic drug’s brain-impairing effects does seem to make it difficult for the patient to focus on the psychotic ideation, and there is often very clear short-term improvement, which I observed in my early experience treating people with acute schizophrenia. However, studies have also found that psychotherapy, such as cognitive-behavioral case management, is just as effective with or without antipsychotic use.

I concur with the minority of psychiatrists who believe in using these drugs only when the patient is dangerous to self or others or too agitated to function in the family and even on a psychiatric unit.

In the above I have taken the position that many factors must be taken into account when considering the risks and benefits of a treatment, such as the severity of the health problem and certainly the seriousness of the treatment’s adverse effects. As a rule of thumb, I would argue as a general standard that the efficacy of the drug or non-drug therapy be at least one standard deviation or more over the average improvement of the placebo group.

I realize what I am recommending would reduce the pharmaceuticals on the market by at least 50% and probably more!

Biases in the Clinical Trials

If one reviews the RCTs for a treatment, it is very important to read more than just the abstract and conclusion, which are full of researcher “spin.” However, even the results section may not tell the full story. It is also important to look at the section on how the researchers chose their subjects for the clinical trial.

For example, clinical trials for drugs to treat depression almost always include a 10-day placebo wash-out period to exclude those from the study who get better on placebo alone! (If so many depressed patients remit their depression on placebo alone in these trials, why do psychiatrists continue to call depression a brain disease?) In the real world, then, the placebo effect is likely to be even more substantial than in the clinical trials—which is significant, since in 49% of the trials for antidepressants, the drug didn’t even beat the placebo.

One perfect example of how patient selection can substantially bias RCTs is Chantix, the popular smoking cessation drug. In the clinical trials for this drug, the researchers cherry-picked the subjects who are more likely to benefit from the drug than other smokers. In these trials, certain smokers are eliminated from the study—namely those with any psychiatric diagnosis, cardiovascular disease, obese patients, adolescents, pregnant women, and light smokers—so, almost everyone.

This means that the study results are only pertinent to a very small proportion of smokers. No wonder Chantix does so much better in the clinical trials than other smoking cessation drugs! Yet do doctors tell people (those with psychiatric diagnoses, for example, or light smokers, or people with heart problems) that this drug has never been tested for them, and that in fact people like them were deliberately excluded from the trial, likely because the drug wouldn’t work for them? Doubtful.

There are many other practices used to ensure that researchers get the results they want, and they all combine to help ensure that the drug is “superior” to the placebo effect.

But anyone trained in statistics and how to conduct sound scientific research, reviewing the FDA depression clinical trials, must come to the inescapable conclusion that these drugs should never have been approved by the FDA for treating clinical depression. Since the FDA did not insist on the use of sound science in the design of these clinical trials, the FDA must be responsible for the many negative consequences following its approval of these drugs.

Beside exposing a very large section of the population to the serious side effects of these drugs, and the additional cost of drugs to treat the adverse side effects, a major negative effect was the creation of a large industry called Treatment Resistant Depression (TRD), a mythical but major new “disease” when the patient fails to respond adequately to the primary drug.

In the newest TV ads for treating depression, we are told that two-thirds of depressed persons on these drugs continue to have depressive symptoms! The add-on drug these ads suggest to take care of this problem is one of several antipsychotic drugs. However, these add-on drugs also don’t seem to reliably get these depressed patents to a permanent remission. This depression roller coaster does not stop here with these add-on antipsychotic drugs (and their adverse side effects), because when these fail to do the job, there are about 8 to 10 FDA-cleared bio-mechanical treatments (mainly zapping the brain) to treat so called TRD!

Actually, psychiatry and the pharmaceutical industry may be better off financially when drugs fail to treat a health problem, or if they create adverse effects that also must be treated with further drugs, than when drugs effectively treat a health problem and treatment can end! To complete the TRD story, the latest pharmaceutical trend has been the introduction of street and hallucinogenic drugs into the pharmacy mainstream (ketamine, for example). If and when these fail, what’s next?

Despite the above, I certainly concur that there are many drugs and non-drug therapies that do meet both a reasonable efficacy and safety standard (if it were not for blood pressure medications, I would probably be dead by now). But in order to achieve better healthcare at lower cost, we must eliminate those drug and non-drug therapies that do not measure up to an appropriate scientific standard of proven efficacy and safety for the condition being treated.

I am not at all hopeful that the FDA and the medical profession will initiate, on their own, a higher efficacy and safety standard than the present one. If a higher efficacy and safety standard is ever to occur, it will have to be instituted by an entity that only allows payment for health services that meet that higher standard.

 

Dr. Ruthven’s website is http://www.ruthvenassessments.com/ and he can be reached by email at [email protected]. His recent book on healthcare can be found here.

***

Mad in America hosts blogs by a diverse group of writers. These posts are designed to serve as a public forum for a discussion—broadly speaking—of psychiatry and its treatments. The opinions expressed are the writers’ own.

25 COMMENTS

  1. Dear Dr Ruthven:
    You stated that long term use of antipsychotics causes irreversible structural brain damage. Can you point me to the research that confirms this? For those who are going through the hell of withdrawal (after long term antipsychotic use), this must certainly be a matter too important to be stated without strong evidence. Thank you, Ed

    • Here is a randomized study finding that 9 months of “antipsychotic” drug use causes brain loss that is 4 times the brain loss that occurs over an entire lifespan.

      https://jamanetwork.com/journals/jamapsychiatry/article-abstract/2761879

      Here is one that found “illness” severity and recreational drug abuse had no or little correlation with brain damage while the drugs caused brain damage. One of this studies authors was the former editor of the American Journal of psychatry.

      https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3476840/

      This study found depending on the length of use the drugs caused an 8-20% shrink in total brain weight in primates.

      https://pubmed.ncbi.nlm.nih.gov/15756305/

      Another study finding the drugs cause brain damage. This study was funded by drug corporations and done by authors known for putting out flawed/biased research favoring the drugs.

      https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0101689

      The most common tactic of psychiatry is to blame the drug effects and brain damage on their victims.

  2. What I find amusing is the pseudo-double blind, which is frequently used to show that niacin is ineffective in treating the “schizophrenias”. What happens is that the placebo is some inert substance, making the flushing from the B3 a “tell” that you’re not in the placebo group and “unblinding” the experiment. Since the “experimenters” want the experimental process to fail, they never use stuff like niacinamide or inositol nicotinate to make their experiments truly blind, while doggedly insisting on the validity of their “experiments”.

    • This is an interesting article. As far as “drug research” goes, the one thing that is never discussed anymore is the effect of the “experimenter” upon the persons in the study; no matter if it is the placebo or the real drug. The other thing that is not discussed is the effect that these drug studies, like almost all medical science studies use volunteers and that also affects outcomes. Of course, in the latter case, it is unethical and illegal to experiment on someone with a drug, etc. without their knowledge and approval. (Although, it is done.) Now, it can definitely be argued that just prescribing these drugs to someone is an act of a scientific experiment, in and of itself. In fact, I would rationalize that I was a scientific experiment and so, I should expect the good with the bad; however there is more bad. What else is interesting to me about this article is that anti-psychotics (neuroleptics) are supposed to be prescribed after the first psychotic break. I never actually had a psychotic break and yet I was prescribed both abilify and risperidal for long periods of time and some other anti-psychotics, too, including I think haldol. I also noted in the article about what the author said about depakote. Depakote has been a common substitute for lithium for bipolar disorder. I was prescribed depakote at times, instead of lithium or alongside lithium along with SSRIs, antipsychotics, and the Benzo drug (to counteract the side effects of the anti-psychotics.) I, honestly, do not have a complete picture of what this has done to my brain. I feel like I have finished the withdrawal stage from these drugs and now consider myself in the adaptation phase. I refuse to give up and just lay down and die. I feel after all this I am alive for a reason. However, I do wish, we had more substantial information available about what to expect and how to adapt after the terrible onslaught of these drugs upon our brains. There are books and organizations to assist those who have Traumatic Brain Injury from accidents or sports or even “non-psychiatric” illnesses, but nothing for those of us whose brains have damaged by these drugs. But, perhaps, it is because there really is no such thing as a true psychiatric illness, as they are all caused by drugs or in some case by environmental chemicals, etc. But, none are caused by anything near by what could be called natural causes or at least nearly natural causes. This is a reply to not only bcharris, but to all who’ve posted, including the author who wrote this informative article. Thank you.

      • I have taken so few drugs in my life that it is hard for me to imagine how people survive the kind of treatments that you describe. I can only imagine that you are a strong being.

        If one defines “mental illness” rather loosely, then yes they do definitely exist. The major one, overlooked in the DSM, is Psychiatrist Disorder. Its major symptoms include: Making up disorders to label other people with in order to hide your own, and pretending to be a doctor while actually harming patients.

        We could all do with improved mental health. But if we got that, we would probably figure out how to overturn the corrupt ones trying to control the current system. So, they pretend, but in fact, it’s the last thing they are interested in.

  3. I would just like to add one other major way drug companies fool patients into believing a drug is far more beneficial than it is: reporting the relative risk reduction of a drug instead of the absolute risk reduction. Depending on the study it could appear that a drug is effective 90% of the time, using relative risk reduction, while the more meaningful absolute risk reduction might show as little as 1% efficacy.

    Newspaper articles often present the misleading relative risk reduction when writing about a new drug I think partly because the new drug would seem so worthless they’d have nothing meaningful to write about if they reported the absolute risk reduction.

    • Yep, another strategy common in psych studies is to report “relapse” were relapse is considered having symptoms under a random selected value. Someone with a symptom score of 40 is relapsed while another person with a score of 39 is labeled as recovered. This way small score improvement of 2% from baseline over “placebo” can be made to appear as if it reduces relapse by 40%.

  4. “The “snake oil” salesman used to report that 90% of their users were very satisfied with the results of treating dozens of health problems with their phony remedies. How can we account for the popularity of these treatments? ”

    If we only have the salesman’s report, and the accounts of the one who swallowed the product is dismissed as anecdotal reports, then of course it has no statistical meaning.

    And the drug or placebo might only “satisfy” for a limited time, with most of that time spent being hopeful.

    What really sucks is that most of medicine is straying away from honesty. (I do NOT include psychiatry in “medicine”)

    MOST “service providers” have one major concern and that is their jobs.

    I wonder if they would be satisfied if they received nothing but shitty home repairs, auto service, haircuts, crap restaurant foods, dirty swimming pools, dirty hotels……nope, they DEMAND good service.

    But the chattle they serve are supposed to crawl to them, lick their dogshit shoes and eat what is dished out no matter what crap.

      • The FDA is populated by people who are part of the drug industry. That’s what I meant by my comment. We need people who don’t have a financial interest in the outcome. There are plenty of scientists in other fields who know enough to evaluate the studies – they don’t have to be doctors or drug company investigators, in fact, we’d be better off if those evaluating drugs had zero attachment to the medical field or drug research at all.

  5. “In the newest TV ads for treating depression, we are told that two-thirds of depressed persons on these drugs continue to have depressive symptoms! The add-on drug these ads suggest to take care of this problem is one of several antipsychotic drugs.” Despite the fact that all doctors – including the psychiatrists – were taught in med school that prescribing multiple anticholinergic drugs (which both the antidepressants and antipsychotics are) is unwise. Since this can result in anticholinergic toxidrome, symptoms of which are “psychosis” and “hallucinations.”

    https://en.wikipedia.org/wiki/Toxidrome

    “However, these add-on drugs also don’t seem to reliably get these depressed patents to a permanent remission. This depression roller coaster” – which I’m pretty certain, is often upgraded to / misdiagnosed as “bipolar” – “does not stop here with these add-on antipsychotic drugs ….”

    In fact, psychiatry is a gigantic, iatrogenic illness creation, scam. And the industry’s goal is for the living hell they are putting their clients through, to continue ad infinitum. If that weren’t their goal, they wouldn’t be lying to their clients and their families, claiming their DSM disorders are “life long, incurable, genetic mental illnesses.”

    And I agree with what you stated, “psychiatry and the pharmaceutical industry may be better off financially when drugs fail to treat a health problem, or if they create adverse effects that also must be treated with further drugs, than when drugs effectively treat a health problem and treatment can end!”

    It’s all about the money. Which, I’m guessing, most psychiatrists don’t even know is created out of nothing, by the bankers. And our monetary system is now being imploded, just like the bankers did, in pre-WW2 Germany.

    But I do believe our modern day psychiatric “holocaust” is much larger than the “6 million” killed during the Nazi psychiatric “holocaust.” Given the fact that the head of the NIMH said in 2015 that “8 million” were dying from the “invalid” DSM disorders – and of course from the neurotoxic psychiatric drugs – EVERY year.

    https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2013/transforming-diagnosis
    https://www.nimh.nih.gov/about/directors/thomas-insel/blog/2015/mortality-and-mental-disorders

    That would amount to approximately 400 million people, murdered by psychiatry, in the past 50 years. And this time the bankers are imploding – not just Germany’s or the US’s monetary system – but all nations’ monetary systems. Nothing like repeating the worst of history over and over again, but making it worse and worse each time. When will the world wake up?

    “If a higher efficacy and safety standard is ever to occur, it will have to be instituted by an entity that only allows payment for health services that meet that higher standard.” The problem with this is that those with all the money, the so called “elite,” seemingly want the rest of us dead, or at least that’s the rumor on the internet.

    But I am grateful for the ethical doctors who are speaking out against all the fraud within the medical field – particularly, but not limited to, the “mental health” field – and big Pharma’s corrupt nature. Thank you for speaking the truth, Les.

  6. I thought this was going to be an article about how to use science to test a hypothesis (or tell if a drug works). It spent very little time discussing that subject, instead focusing of the vagaries of testing drugs. One reason these problems exist, of course, is because the subjects you are testing are humans, and science – a product of human activity – does not even understand humans that well. Thus the physical sciences and engineering tend to be much more successful using the scientific method than do Medicine and the social sciences (like psychology, economics).

    What you should be able to test for in terms of drug reactions is what changes the drug makes to the body. This would keep the tests in the realm of the objective. To test for the suppression or relief of subjective symptoms is a whole new ballgame.

    I think that for any major leaps forward to be made in the physical or the social sciences, a better understanding of how life works will have to be achieved.

    Sorry to be pedantic, but the fact is that the actual healing agent in biology is Spirit. If Spirit decides to heal, then healing will occur, and the experimental subject will normally report an improvement. And if Spirit decides to stay sick, then the subject won’t get better. This is the power of the placebo, and it could be argued that all those RCTs that use placebos have actually proven the healing power of Spirit!

    But more importantly, if life were better understood, doctors and Big Pharma would be forced to admit that their drugs should take a back seat to therapies that address Spirit more directly. A great way to sober up a drunk is to give him a Locational. And if the real cure for psychosis was to “make it difficult for the patient to focus on the psychotic ideation” then a Locational should also work to treat psychotics! Yet, anyone could be trained to help someone in this way, so where’s the profit in it?

    Some would argue that all this really tells us is that help and greed are incompatible. Perhaps science (if it is meant to help) is also incompatible with greed (profit motive). I suppose this was the original idea behind having so much government-sponsored research. But now that the corporate world has nearly captured government and academia, how do we move forward?

    Some of the most important hypotheses of the last century remained untested by science due to the suppressive influence of vested interests. There is no field where this is more evident than Mental Health, with Medicine running close behind. How can we shake these vultures loose and move forward?

  7. Dear Mr. Ruthven, Thank you for your comment about my comment. One more thing is that when the psychiatrists became accepted into medical circles, because they started using the medical model; the traditional doctors could then say, “I don’t know what’s wrong with you, it must be in your mind ” and refer you to a psychiatrist or “You’re just so stressed from work or whatever…” and refer you to a psychiatrist. Basically, psychiatry adopting the medical model allowed traditional medical doctors to “pass the buck.” In the end, I feel this will be the downfall to any sort of good, decent healthcare and I am concerned that any reforms to healthcare would probably fail. And, I am very usually not a person who sees the glass half empty. Despite it all, I see the glass half full. It is just my experience in what I call “psych world.” And as you can see, I am quite wordy at times and resort to “trite slang expressions” rather than a specific word. Why? Due to the “brain damage” from the psych drugs, I can either no longer spell correctly or find the right specific word. Thank you.

LEAVE A REPLY