Aggregate vs. Anecdote for Med Lovers

Corinna West
24
1940

Mad In America and its writers are often dismissed out of hand as “anti-medication.” But we’re not anti-medication, we are pro fully informed choice. It’s just a shame that people in society don’t get anywhere near full information about how well drugs work or what their actual risks are. But a lot of this boils down to people not understanding the difference between aggregate and anecdote. There’s a lot of people out there who don’t want to hear medication critiques because they like their meds. And for people who like their meds (or like their family member on the medications), how do we reach them?

Of course there are many reasons that people don’t have the information to balance harms and benefits. A lot of these reasons are discussed on this site, such as institutional corruption, dominant society paradigms, or medical providers with conflicts of interest. What I wanted to go into today is the people who can’t hear any of this data because they don’t understand the difference between aggregate vs. anecdote.

1. “I like my meds.”

A lot of people say, “Well, my meds work for me, they were a miracle.”

My response is, “This sounds at first blush like a success story. This is why our messaging needs to be careful. The idea is that medication helps some people, but not everyone, and may be increasing the amount of disability in our country.”

This is because in the aggregate (that is, on average) the vast majority of medications are no more effective than placebo in the long run. But aggregate does not mean everyone. Just like any average of data, there are people who do well, or extremely well, or extremely poorly with any kind of treatment.

There is a very good blog post from the late great Mickey Nardo at 1boringoldman.com about how many drugs don’t even beat placebo. The image below is a little more complex. It is actually depicting how drug trials show a higher placebo response rate through time and a lower drug response rate. But notice how similar the placebo chart is to the drug response chart. This is how the aggregate can be visualized. Your success story is one of these dots. In this image, each dot is a whole clinical trial, which could break out into another clump of dots. This site is talking about all the dots. You can’t talk about all the dots until you look at all the data.

The vast majority of the data shows that people who benefit are statistical outliers. You got lucky. We’re glad you got lucky. But your story is not the whole story.

But what about the others, the ones who are hurt by the medication or it doesn’t beat placebo? Is it worth the risk? Public policy decisions shouldn’t be based on “the lucky ones,” they should take into account all aggregate data. And users who are statistical outliers shouldn’t promote their treatment as a solution for everyone. People who tell a story about one dot on that chart but then don’t listen to the story about all the dots. And there are many more dots on that chart that had no benefit or had harm than the ones who got lucky and were helped.

2. “But the drugs don’t hurt me…”

And then there is another version of the story. Have you heard that one yet as an advocate? If you are trying to share info about medical harm and your friendly drug user friend says, “Well, they don’t harm me….”

My response: You just may not know what you are missing. My personal recovery story included a time where medications seemed very helpful, but later the medications started causing a movement disorder and severe sleep disruption, both of which resolved when I came off medications. Also, many medication critics talk about “numbing emotions,” and I never felt numbed on medications. But when when I came off medications, there was an incredible emotional intensity and beauty to my life that I didn’t even know I’d been missing.

People may not know the harm being done.

People can be advised to work on reducing the medications that are causing the most problems with side effects, and work with an extremely slow taper according to good tapering practices found in this list of resources. Most people find a “minimum effective dose.” Once people know the difference between medication withdrawal effects and the return of the “illness,” many people find that the biggest effect of their medication is simply relief of withdrawal symptoms.

Back to anecdote vs. aggregate — you may be in the lower dots and not the lucky dots without even knowing it. And then there is the thing about our communities dying 25 years younger.

3. “What about suicide?”

But then some people say, “But the drugs keep me alive.”

My response: Well, the medications don’t prevent suicide in the aggregate in the long term. They may prevent suicide for you, as one data point, but the rest of the data points show that some of these drugs actually increase suicide.

Again, for individual vs. aggregate: Some lucky dots might feel that suicide has been prevented. But looking at all the dots, the risk is increased. Mickey Nardo did a whole series of posts on Robert Gibbons who fought the FDA black box warning on SSRIs. The FDA put the black box warning on SSRIs because they were increasing the suicide risk for youth. And Gibbons said, “FDA shouldn’t have done that because the kids might die by… suicide.” But Gibbons basically had to make up data in order to make his point. The FDA’s view of all the dots showed an increase in suicide.

Your “lucky dot success” story might be impressive, but good policy decisions are made by looking at all the dots. Again, anecdote vs. aggregate.

4. “Labels help people.”

There is a dude named Mark Weibe that has started to be my advocacy enemy here in Kansas City. He keeps saying, “I know you talk about harms vs. benefits, but what about people who feel relieved when they get their diagnosis?”

And I have told him, “Well, if the diagnosis was a valid medical entity that might be a useful point. But why trade long term stigma and disempowerment for short term relief?” And finally now there is starting to be more research on the damage done to people by psychiatric labeling. 

Again, aggregate vs. anecdote. Mark may tell one story, one dot about one person who liked their labels, but I am talking about all the dots, all the people. Unfortunately, because I speak with passion, because I don’t always wear a suit, because I am not a white male, Mark gets heard in these rooms instead of me. His anecdotes outweigh my aggregate data because people are scared of conflict. They listen to him because he does the right butt kissing and he does not call out these emperors with no clothes. I’m sure that all advocates reading this have been through this experience.

You quote nine research studies talking about 30,000 data points and you care about the issue because you are tired of seeing your friends die. And some disease model guy who uses recovery language but not our concepts gets up and says, “Yeah, I haven’t read those studies but I know one person…..”

And they listen to him instead of you.

What do you about this? The challenger who says that the labels themselves help people? Especially in an advocacy setting?

Cool cartoon from the Texas Education Agency.

5. “Well, you should support my CHOICES.”

Then there is this other misunderstanding of aggregate. People who say, “You are pro-choice for mental health, so you should support my choice to use psych drugs.”

My response: Well, it’s hard to support an uninformed choice that has been made on false assumptions. IF:

  1. The drugs don’t beat placebo (see Irving Kirsh’s info on this),
  2. And the drugs hurt more people than they help (you can compare NNH to NNT for most drugs, this is the best way to communicate tradeoffs),
  3. And the drugs expose people to serious risks of agitation-induced suicide and violence and horrible withdrawal syndromes,
  4. And the disease labels connected to the drugs increase stigma and make people lose hope,

And you are not told this BEFORE you choose drugs and labels, then how valid was your ability to make that choice? We support people whose choices disagree with ours, but we do try to help people figure out where those choices may have been misinformed.

This is really the crux of this whole issue. Until the medication users and the disease model advocates feel “heard” by medical harm aware advocates, they aren’t going to listen to us. And I have such a heard time “hearing” them because they are so clueless. So they can’t hear me because I don’t hear them.

Dialogue is a two way street. I don’t know how to “hear” these people who LIKE their meds and don’t know any better. Give me some better ideas for these challengers in the comments?

6. “We have our ‘consumer’ input.”

And worst of all offenders who don’t understand aggregate are these ‘consumer’ coalitions. More and more funders are making coalition approaches mandatory to get funding. You know the type: 12 people at the table and one advocate if “we” are lucky. And you say, “We should have more people with lived experience at the table,” and they say “Oh, we do.”

Meaning: “We are going to count the long term admin people who now feel safe enough to come out with their recovery story.” Meaning this admin person is a novice advocate who has not been an advocate before and has not interacted with the advocate community and does not know the perspectives and viewpoints of the advocates.

Or: “We are going to grab some random patients off the clinic floor and say that they are the voice of the ‘consumer.'”

And that results in a completely deadlocked discussion group.

Well, guess what? Advocacy is skillset, advocacy is a profession, and inexperienced advocates are not a good replacement for experienced advocates. You know why? Because anecdote vs. aggregate. These novice advocates don’t know long term outcomes data in the industry, they don’t know about all the dots. And without experienced advocates at the table nothing new is going to happen.

And how do you fight this? When more and more voices “at the table” are blocking or excluding experienced advocates? The #6 challenger who is a high level decision maker, a coalition leader, and STILL doesn’t know the difference between anecdote vs. aggregate?

How do you fight any of this? Give me some fresh ideas in the comments below. How do you educate people about aggregate vs. anecdote?

24 COMMENTS

  1. 3. “What about suicide?”

    ” They found four such prophylaxis trials. They compared suicide risk in patients on placebo compared with patients on active medication. Two suicides (493/100,000 person- years of exposure) and eight suicide attempts (1,969/100,000 person-years of exposure) occurred in the group given an active drug (943 patients), but no suicides and two suicide attempts (1,467/100,000 person-years of exposure) occurred in the placebo group (418 patients). Based on these absolute numbers from these four trials, I have calculated (see Figure S1 showing calculation, and see Figure 2) that active agents are most likely to be associated with a 2.22 times greater risk of suicidal acts than placebo (95% CI 0.5, 10.00). Based on these absolute numbers from these four trials, I have calculated (see Figure S1 showing calculation, and see Figure 2) that active agents are most likely to be associated with a 2.22 times greater risk of suicidal acts than placebo (95% CI 0.5, 10.00). http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0030185 http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0030185

    They came up with that data when people were saying it was unethical to give study patients placebos !! I will be back with an edit when I find the article on that.

    “OBJECTIVE: The authors’ goal was to investigate whether there is a greater suicide risk in the placebo arms of placebo-controlled studies of active medication for the treatment of acute manic episode and the prevention of manic/depressive episode. If so, this would be a strong ethical argument against the conduct of such studies. http://ajp.psychiatryonline.org/doi/abs/10.1176/appi.ajp.162.4.799

    • It is also VERY important to recall that they screen out actual suicidal people before starting these trials. So ALL suicidal participants developed suicidal thinking and/or behavior AFTER they started the trials (outside of the possibility of dishonesty in an odd participant here and there). The actual suicide rate in community practice would be considerably higher.

  2. Corinna,
    “Dialogue is a two way street. I don’t know how to “hear” these people who LIKE their meds and don’t know any better. Give me some better ideas for these challengers in the comments?”

    There’s a saying in some groups that goes, “People don’t care how much you know, until they know how much you care”… As I began to help my wife heal, I made a point of never coming off as if I was ‘fixing’ her. I made a point of never coming off as if our relationship was predicated on her accepting my help or suggestions. I made sure she, all the girls in her d.i.d. system, felt 100% safe in their relationship with me no matter if they ever got healed, period.

    And I think until you can find a way to show you care about people, it will be the same. I realize you can’t have a personal relationship with everyone you talk to, but somehow people have to see the willingness to ‘walk in their shoes’ first…it’s not easy, but at least in my one, anecdotal, experience, that’s how it worked…
    Sam

  3. Now to your major point, Corinna,

    let me first say that I mostly, maybe even wholly, agree with the premise of your article, and yet this entire premise of dismissing ‘anecdotal’ evidence has become a hammer that the ‘experts’ use to put the rest of us in our place while hiding their lack of scientific curiosity…because what is the very first step of the scientific method? Gathering anecdotal observations and seeing if there is a larger pattern possible. But instead the experts throw around their ‘studies’ at the rest of the ‘unenlightened’ and ‘ignorant’ and expect us to bow to their greatness.

    Again, I mostly support your premise but this entire biomedical juggernaut that has been forced down everyone’s throat was done, supposedly, because of the ‘weight of the aggregate evidence’ and now other experts like you are trying to be heard and say, ‘don’t look at their evidence, look at ours!’

    Again, I’m mostly in support of your premise and the demolition of the biomedical model, but this isn’t the only scientific theory that I have enough knowledge of to tell me the ‘experts’ are full of it and aren’t even following the basic premises of the scientific method when they supposedly give us science-based methods in so much of what we accept as common knowledge…let’s just start at Galileo or any other great discovery and remember how hard the consensus keepers work to invalidate the innovators that only have anecdotal evidence available to back them no matter how careful the innovators were in their method. The consensus-keepers throw their ‘vast, aggregate’ knowledge around to demand our obedience, and when any of us can prove otherwise using their OWN methods, yet we don’t have the standing to go out and do a massive study to get ‘aggregate’ evidence, we are told to “shut up, your evidence is just ‘anecdotal’ and therefore irrelevant.”
    Respectfully,
    Sam

    • Samruck, that is a thorn in my side too.

      I am not a professional, I am a peer support and caring friend, with my own madness experience.

      In my work at Surviving Antidepressants, I have seen as many case studies (anecdotes) as some doctors.

      After seeing someone accelerate at 70 mph towards a brick wall, it becomes easy to predict how that crash is going to unfold. This isn’t the best example because it’s pure physics instead of the weaselly chemical actions of these drugs.

      But in general, I have noticed that people who cling to their diagnosis stay diagnosed. People who insist that doctors (or others) “fix” them, do not take responsibility for their own wellness. People who take charge of their self education and healing find ways to get better. People who cold turkey from the drugs and feel well at 1-3 months out, tend to have more experiences waiting for them, some of which will be so extreme as to send them flying back to doctor or hospital. People who bounce their doses and change drugs get destabilised / symptoms. People who carefully taper generally do better (but not always).

      These are generalisations based on an aggregate of anecdotes. But like bcharris says, I don’t have a lab (or white coat) or blood tests, clinic, MRI’s, or any “scientific” evidence to prove it. Just years (and possibly decades) of experience.

      The shocker for me is this: if I, in my limited experience, can see this (anecdotally, and in the aggregate of anecdote after anecdote after anecdote) – why don’t the doctors?

  4. When I think about problems with psychiatric drugs, I think about someone in a fog from a heavy dose; when I think about anecdotal advocates for psychiatric drugs, I think about someone taking a light dose. It seems like a discussion about drug therapies should include information about dosages.

    • Steve: I agree with you partially but I think it gets even more complicated than what you suggest. Not everyone metabolizes drugs at the same rate and individuals who are self medicating the side effects of their drugs with other substances such as nicotine, may be affecting the metabolism of their prescription drugs even more, making them build up to toxic levels in their bloodstream or rendering them ‘sub-therapeutic.’ Once again, it goes to the heart of why are we court ordering people to take neurotoxins when we know so little about how they will affect people, who may have a near infinite number of responses. When you consider polypharmacy, which isn’t even tested for safety, it gets even more complicated If I was convinced that psychiatrists and other prescribers actually listened to their patients and would take into account individual differences and would take the time to get to know their patients and build a trusting relationship with them, then it would make sense to talk about dosages. But even questioning the dosage with a prescriber is not necessarily a safe option for some individuals. Questioning the dosage may result in another psychiatric diagnosis such as personality disorder, being oppositional, having anosognosia, etc.

    • Here in Australia, they have a medical measurement for cigarette smoking damage called “pack years.” How many packs for how many years.

      I think that any mention of dosage should also include time on the drug, number of switches between drugs, age when the drugs were started, and other drugs being taken.

      All of these are factors. A 13 year old started on 150 mg of seroquel and on it for 10 years will have a different profile to a 30 year old started on 150 mg of seroquel for 10 years.

      Then there’s polypharmacy, like madmom suggests. There’s no testing for polypharmacy, all we have is the anecdotal collective evidence of this gigantic experiment loosed upon the general populace.

      When someone says to me, “My drugs are working,” my first question is: how long have you been on them?

      Usually, it is less than 5 years (more often, less than a year. Sigh). After that, they are starting to get sick, or having sleep problems, or unexplained symptoms that may not be connected to the drug. My next question is, “how many other drugs have you tried?” When that starts to get up in the 3-5 drugs (not polypharmacy, but switching), then there are often problems.

      How many other drugs are you on? I include statins, PPI’s, blood pressure medication, blood thinners as possible contributors to major problems. For me, statins were as deadly as the psych drugs (even though the lithium was starting to deteriorate my kidneys)

      How old were you when you went on them? People who were put on the drugs before fully forming in adulthood often have a lot of learning to do – in order to come to healing.

      It’s complicated! We are each an experiment of N=1.

  5. My dad tells this anecdote about a user of street drugs. She wrote a nasty letter to “Dear Abby.” At the end of the letter she told Abby she bet she wouldn’t have the nerve to publish it because her arguments were so brilliant–and she wrote the whole thing under the influence of said drug.

    Abby wound up publishing the letter in its unedited form. Not exactly the best argument for recreational drug use.

    The magic of spellbinding! 😀

  6. The main reason anecdotal information is preferred is because of the financial incentives. Unless those change, it’s hard to imagine the industry actually respecting the real data. There is no money in honesty.

    As for those who are themselves convinced by the propaganda, it’s very important to start by acknowledging the reality of their personal experience. I think the next question would be to ask, “Do you think that everyone has the same experience as you do, or do you imagine that some people have different experiences?” If they can’t handle that possibility, I would not bother with further discussion. Some people are like that. If they can, I’d ask, “Would you like to hear some more about that?” and then share some anecdotes (after all, this person is basing their assessment only on anecdote, right?) where things didn’t work out so well. At that point, you can introduce the concept of aggregate data, maybe starting with, “So let me ask you to imagine this: imagine that some people (like you) feel like they do better taking these, some people feel like they do worse, and a lot of people don’t feel very different. Can you imagine that happening?” If they can, you’re in a place to start educating about aggregate data. But you have to get the person to allow that aggregate data is relevant or important before it will be a viable argument.

    • Even if they get the concept, Steve, there’s still going to be the question of whether the data is trustworthy. Most folks are intimidated by math as simple as fractions, percents, probability. Statistics is math that’s way too complicated for most people to understand, and people just don’t trust what they can’t grasp. Research is suspect.

      • That is very true. It’s important to keep the statistics very simple, and to have stories available as illustrations of what the stats mean for normal folks who don’t have advanced degrees. That’s why I tell the stories first before getting into the data. But for some people, data is irrelevant, at which point, I would stop using it as an argument.

        • You will quote the kind of data and stats in this article. People on the pro-psychiatry side (check out some of these “skeptic” “science” blogs) will “debunk” this data, quote their own data and hammer into people’s heads that “psychiatry is a scientific field”, “psychiatrists are medical doctors and experts” (because they underwent medical training prior to psychiatry), “DSM labels are medical diagnoses” (even if a some of the stuff that goes on in those places has nothing to do with medicine or science at all, and even whatever does seem like medicine and science, are things which, the consequences of, one can only realise when it is too late and they are neck deep in it, with a barrage of labels, drugs, and their identities altered forever).

          There are some things you can only learn from practical experience and not even if you read all the published journal papers in psychiatry and know the most advanced stats in the world.

          To me, all this data and these words (“science”, “medicine”) associated with psychiatry have become simple argumentation tactics between the pro and anti psychiatry sides. There are very few people doing actual groundwork, but many endlessly debating “data, data, data” and whether or not “psychiatry is a science”. This “data” game is a never ending game. And who cares about the words in the end? It’s the consequences on the lives of individuals that matters.

          There are plenty of these men with knowledge of stats, but with dilettante knowledge of googling things, going through journal papers and crunching numbers who write blogs, post comments etc.; and then other men (even psychiatrists) who take part in these Cochrane Collaboration dramas, who will improve their own research profiles, but never do a practical thing in their lives to truly help someone.

          Are these people going to rescue kids from abusive homes? Prevent gaslighting of individuals? Fight legal cases on behalf of people who have been screwed over by DSM labels? No. It would put their lives and careers in jeopardy.

          People are much more concerned about what happens to individuals in real life practice in those places, and these things you can only find out from the real life practical experiences of people. Such experiences can be substantiated in a legal sense too, in terms of audio-visual recordings, documentation etc.

          • There’s a lot of magical thinking that goes on today. Rather than think for themselves, the public will swallow anything an “expert/scientist/doctor” dishes out. In this case quite literally.

    • I pay a lot of attention to anecdotal information. It has spared me quite a bit of suffering from the side effects of medication. There is an excellent source of this kind of info on the internet. Google “ask a patient” (using quotes) and go to the first website it lists. It’s a gold mine — and free!

      Mary Newton

  7. The way I see this is that each person’s experience with a given drug is one anecdote, as in I know this person who…. and so on. I thought the point was, if you then plot each anecdote on the graph (I assume there would be more dots on the graph than shown), you would see the aggregate picture, you get to see all the anecdotes in one place. Then you can get a rough idea of the odds that you will be helped or harmed by this drug. Along the horizontal axis in the time line, so this is when the anecdotes were recorded. I think the graph makes the point.

  8. On the subject of labels. They’re useful if the label implies courses of actions to take to stay healthy or sane, as do elements of the diabetic label (little or no sucrose, regular meals, exercise). Similar things apply for “schizophrenics”, like avoiding junk food, keeping regular hours, avoiding stimulants (including caffeine), alcohol and hallucinogens, exercising. It would also permit you, if you are negatively labeled, to tell the labelers (for example) how lucky they are to be able to shoot meth until they’re having philosophical discussions with their pets (particularly if they’re animals like goldfish).

  9. I think it’s a matter of trust. People trust what they know, what they can observe for themselves or hear second-hand. But aggregate data is too many steps removed, and is subject to so many levels of manipulation there’s no integrity in it. From the outset of a study, the researchers are seeking a particular outcome and make choices that are most likely to yield the desired result. Who do we recruit to participate, and how do we access them? Is there a control group? Are they truly a control group, or are they cold-turkeyed off a drug right before “testing” another? Do effects get reported as stated by the participants, or interpreted (as in suicidality being watered down to “emotional lability”)? And then there are the definitions, and whether the researchers’ idea of a good outcome, of “treatment success” looks remotely like what the participant would consider to be successful. And that’s before we even venture into statistical manipulations! Wasn’t it Mark Twain that said, “there’s Lies, Damned Lies, and Statistics?” Most people find math as basic as fractions and percentages intimidating- don’t ask them to trust advanced statistics! Personal stories are always going to have more of an impact. Research is dry; stories are juicy. Research is elitist; stories are accessible, people can relate to them, can empathize with them, can trust them. Not so with stats. A voice spouting stats is never going to ring as true as the voice of experience.

  10. I think you should read “Pedagogy of the Oppressed” by Paulo Freire. Is a great book, left uknown by univesities and media alike, which talks at great length, as the title suggests, about the way to humanize society. At least it would be a starting point, i hope, to solve the problem of communication with all kinds of oppressors either who rule or certainly dehumanize the world.

  11. I think thinking about the underlying reasons people prefer to take medications is really helpful. From what I’ve seen people mostly don’t WANT to take them, but they would RATHER take them than engage in the other options (therapy etc) because they are too scared, too self-critical, too ashamed, in too much pain etc etc, sometimes cost, family pressure, previous bad experiences with therapists etc are in the mix too. I really see it as people doing the best they can to get quick relief from psychological pain. Connecting and empathising with this, before challenging it, is more effective, partly because it’s more effective for all conflicts (connect then conflict) but it also acknowledges the person’s humanity and suffering, rather than just seeing them as someone who needs to be educated and converted. People on meds are often themselves internally conflicted about being on them (sometimes a little subconsciously) and so when they are directly challenged, they are polarised and have to take the other side (be defensive). Not polarising the conversation by taking both sides with them (the empathising part), gives them the freedom to also take both sides, because they feel connected with, not just attacked. It’s just too much for many people to deal with the question “if not meds, what…” Perhaps highlighting the effectiveness of alternatives is more helpful as it gives hope and suggests new directions, as opposed to just informing why meds are a bad option.

    • I think this is a very wise post. Us vs. Them is the oldest meme in human history. We have to find ways to let the person who disagrees know it’s OK to disagree, and to listen to WHY they believe what they do. It is also almost axiomatic that a person being defensive about something is ambivalent, or they wouldn’t have to be so emotional – they’d just be kind of incredulous or scoff but they don’t get upset about it if there isn’t some uncertainty there, usually a LOT of uncertainty. Helping the person locate the pro vs. con in him/herself can enable them not to project the con onto you as the “enemy.”

      Thanks for your wisdom!