Critical Psychiatry Textbook, Chapter 5: Psychiatric Diagnoses Are Not Reliable (Part Two)

Editor’s Note: Over the next several months, Mad in America is publishing a serialized version of Peter Gøtzsche’s book, Critical Psychiatry Textbook. In this blog, he discusses the poor reliability and validity of psychiatric diagnoses and the concept of overdiagnosis. Each Monday, a new section of the book is published, and all chapters are archived here.

Two concepts are essential when discussing diagnostic tests, their validity and reliability.

The validity of a diagnostic test refers to its ability to measure what it is purported to measure, which involves its ability to distinguish between people with and without a particular disease.

The two principal measures of test validity are sensitivity and specificity, which are the proportion of those who are ill that test positive, and the proportion of those who are healthy that test negative, respectively. Most people believe that the predictability of positive and negative diagnostic tests are constants, which they are not, as they depend on the prevalence of the disease that is being tested for.¹⁰⁵ The more uncommon a disease is, the more false positives will there be. This is why screening for depression is a bad idea. The screening test for depression recommended by the WHO is so poor that for every 100 healthy people screened, 36 will get a false diagnosis of depression.^7:46,106,107

When I criticise my colleagues for using such poor tests, I am told that they are only a guideline in the diagnostic work-up and that additional testing will be performed. In an ideal world perhaps, but this is not what most doctors do. Many patients report that there was no further testing and that they got a diagnosis and a prescription in about ten minutes.¹⁰⁸ This is expected, as about 90% of prescriptions for depression pills are written by general practitioners,^{7:256,108,109} and they don’t have much time.

The reliability of a diagnostic test depends on the accuracy and reproducibility of the test results. The accuracy is defined by comparing the test results with a final true diagnosis. There is no such final true diagnosis in psychiatry, and it is therefore not possible to determine the accuracy of a diagnostic test. But its reproducibility can be determined in observer variation studies where two or more psychiatrists suggest a diagnosis for the same patients.

Four of the five textbooks did not mention a single result from observer variation studies and gave the erroneous impression that psychiatric diagnoses are both valid and reliable. With rare exceptions, e.g. the admission that no questionnaires for diagnosing adult ADHD have been validated,^17:615 diagnoses were not doubted. One book claimed that the reliability of the diagnoses is good and noted that, to ensure that the criteria-based diagnoses are sufficiently reliable, they were tested before usage in a big international study, and diagnoses that showed low reliability were either removed or the criteria were strengthened.^16:23

It is not clear what the authors referred to, but what they wrote is wrong.^7:32,110,111 As one commentator put it after the appearance of the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5): “Real sciences do not decide on the existence and nature of the phenomena they are dealing with via a show of hands with a vested interest and pharmaceutical industry sponsorship.”¹¹²

One of the books showed a figure demonstrating that the number of patients diagnosed with schizophrenia in Denmark had quadrupled from 2000 in 1971 to 7400 in 2010.^19:225 This should have woken up the authors but they did not comment on this stunning finding, even though something must be wrong with the validity of the diagnosis. This diagnosis can often not be sustained, e.g. it was rejected in 64% of 1023 people.^1:173

A psychiatrist wrote to me:^7:360 “In my twenties, I had a massive breakdown. At the time, I instinctively resisted all psychiatric labels and medical treatments. When I look back now, I can easily see how, in the wrong hands, I could have been labelled schizophrenic, as I heard voices and had delusions and severe anxiety. Now I know my breakdown was no different to what my patients experience.”

Diagnoses stick to the patient. Once made, everything the patient does or says during a hospital admission becomes suspect, as the patient is under observation, which means that the initial, perhaps tentative diagnosis, all too easily becomes a self-fulfilling prophecy.^7:30 The doctor’s intuition and experience may suggest very quickly what the problem is, and there is a considerable risk that the doctor from then on asks leading questions, which yields the required number of error points and leads to a misdiagnosis.

There is much overlap between the different diagnostic categories, often called high comorbidity, although the problem is not that the patient has several “diseases” but that the diseases are so vaguely defined, with overlapping symptoms, that many patients can get several diagnoses.

Even the book that was critical of psychiatric diagnoses failed when it came to observer variation studies. When the two authors, a psychologist and a psychiatrist, discussed the validity and reliability of the diagnoses, they mentioned kappa, Cronbach’s alpha, Hamilton’s Depression Scale and a wealth of other scales and issues over 25 pages, but not a single result from inter-observer variation studies.^17:165

Kappa values measure to which extent two observers agree beyond chance. If kappa is 0.60, it means that the agreement is only 60% of the difference between chance agreement and perfect agreement, which is pretty poor. There are many problems with kappa.¹⁰⁵ It presents statistical problems and does not tell us, for example, if the disagreement is important, which it surely is for psychiatric diagnoses because a diagnosis almost always lead to drugs, often for many years without interruption,^113,114 and also often to a downhill chronic course for the patient.^5:8,119:24

The claim that the new diagnostic checklist system introduced by the American Psychiatric Association for its DSM-III in 1980 is reliable has been convincingly refuted in a book.^7:32,102,110 The disappointing results when two psychiatrists assess the same people have been buried in a smoke of positive rhetoric in surprisingly short articles, given the importance of the subject.

The documentation is hard to find, but two people did the work, which was a huge task.¹¹⁰ Even the largest study, of 592 people, was disappointing despite the fact that the investigators took great care in training the assessors.¹¹¹ For bulimia nervosa, which is very easy to diagnose, the kappa values when two physicians interviewed the same people were above 0.80, but for major depression and schizophrenia, two of the most important diagnoses, the kappa values were only 0.64 and 0.65, respectively. This is frightening considering the devastating consequences of false positive diagnoses.

When researchers interviewed 463 people about 91 key symptoms for psychiatric disorders, they found that all of them experienced thoughts, beliefs, moods, and fantasies that, if isolated in a psychiatric interview, would support a diagnosis of mental illness.^1:168,115

If the general population is exposed to just a few of the various diagnostic checklists that are being used, a large proportion will get one or more psychiatric diagnoses. When I lecture and try three diagnostic tests on the audience—for depression, ADHD and mania—about a quarter test positive for one or more diagnoses. Imagine if you tested people suspected of having cancer with a test that gave a quarter of them an erroneous cancer diagnosis. We wouldn’t allow such a poor test to be used.

DSM-III from 1980 was replaced by DSM-IV in 1994, which was even worse than its predecessor and listed 26% more ways to be mentally ill. Allen Frances, chairman for the DSM-IV task force, has argued that the responsibility for defining psychiatric conditions needs to be taken away from the American Psychiatric Association because new diagnoses are as dangerous as new drugs: “We have remarkably casual procedures for defining the nature of conditions, yet they can lead to tens of millions being treated with drugs they may not need, and that may harm them.”¹¹⁶ Frances noted that DSM-IV created three false epidemics because the diagnostic criteria were too wide: ADHD, autism, and childhood bipolar disorder.

Psychiatric diagnoses are uncritically believed not only by psychiatrists but also by the media. Even websites that are critical towards overdiagnosis of diseases and overtreatment with drugs and advocate for a new biomedical and social model, convey information like, “One in four people in the world are prone to be affected by mental disorders at some point in life. These mental disorders are the leading causes of ill-health and disability worldwide.”¹¹⁷

Several things are wrong with such commonly seen statements. First, many people are over-diagnosed. They do not suffer from a mental disorder but have problems in their lives. Second, they are not affected by a mental disorder. As already explained, to label people’s problems does not create a being that attacks people. Third, mental disorders are not leading causes of ill health and disability. People suffering from deprivation, poverty, unemployment, and abuse suffer ill health and disability; they are not attacked by some psychiatric monster.⁹⁶

The bottom of journalism was reached when the United States established The Carter Center’s Guide for Mental Health Journalism, which is the first of its kind.^8:162,118 This institution educates journalists to write flawed articles and to never question psychiatric diagnoses. Journalists should pin down exactly what a professional says is wrong with a patient and use that information to characterize a person’s mental state. There is no encouragement for journalists to consider how people so diagnosed see themselves, or whether they accept their diagnostic label, or if the professional might be wrong.

According to the Carter Center, the DSM-5 is a reliable guide for making diagnoses. There is no mention of the fact that the diagnoses are arbitrary constructs created by consensus among a small group of psychiatrists, or that they lack validity, or that psychiatrists disagree a lot when asked to examine the same patients, or that most healthy people would get one or more diagnoses if tested enough.

Reporters are told to write that behavioural health conditions are common and that research into the causes of and treatments for these conditions has led to important discoveries over the past decade. They should also inform the public that prevention and intervention efforts—meaning drugs—are effective and helpful. This is the same message that the American Psychiatric Association and leading psychiatrists all over the world have been promoting for many years.

The guide prompts reporters to echo the message from the American Psychiatric Association that psychiatric conditions are often undiagnosed and undertreated, and that psychiatric treatment is effective. The guide avoids any discussion about how ineffective and harmful the drugs are and makes people believe that “treatment” also includes psychotherapy, even though this is rarely offered.

Nothing is mentioned about overdiagnosis. Reporters are not encouraged to explore why it is that the public health burden of mental disorders has grown dramatically in the past 35 years, at the same time as the use of psychiatric drugs has exploded.^5:8,119:24

The guide states that between 70% and 90% of people with a mental health condition experience a significant reduction in symptoms and improvement in quality of life after receiving treatment. The source of this false information is the National Alliance on Mental Illness (NAMI), a corrupted patient organisation.^7:357 It is true that most people improve, but that would have happened without treatment. Like many of the textbook authors, the Carter Center seems to have “forgotten” why we do placebo-controlled trials, and it has never been documented that psychiatric drugs improve quality of life; in fact, they worsen it (as will be explained in Chapters 7 and 8).

Reporters are told to emphasize the positive and avoid focusing on the failures of psychiatric care. The guide does not provide any resources for obtaining the perspectives of people with lived experience, most of whom would speak critically of the conventional wisdom.

Unfortunately, the Carter Center is seen as a leader in training journalists on how to report on mental health. It encourages journalists to act as stenographers repeating conventional dogma.

It is difficult to see much hope for America. Journalists are told to convey the strongly misleading narratives created by the drug industry and US psychiatrists on industry payroll to the great harm of our patients and societies.^5-7

It is very strange that there is such an institution in America. What the Carter Center does is like telling Chinese journalists that if they want to know what it is like for the Chinese people to live under a dictatorship, they should not ask the people but the Chinese leaders.

One book noted that a good rule of thumb is not to make a depression diagnosis in the first two weeks after stopping drug abuse or intake of medicine.^16:258

This principle should apply to all patients. Diagnoses can make it difficult to get the education patients dream about, a job, certain pensions, to become approved for adoption, to get an insurance or child custody, or even just to keep a driver’s licence.^120,121 Psychiatric diagnoses are often being abused in child custody cases when the parents get divorced.¹²⁰ Even when the diagnosis is obviously wrong and the psychiatrist herself seriously doubted it when she made it, it cannot be removed.¹²¹ It sticks to the patients forever, as if they were branded cows.

Already on the next page, this book ventured in the opposite direction saying that older people are at risk for underdiagnosis of depression because relatives and sometimes the doctor accept and explain their sadness as understandable, based on the many losses of friends and perhaps the spouse and physical capacity.^16:259 The truth is the opposite. Old people are overdiagnosed to an unbelievable extent and sadness is a normal feeling, not a psychiatric diagnosis.

The book about child and adolescent psychiatry mentioned that diagnoses are designations for a condition, a kind of snapshot, and not designations for people.^19:36 It advises that diagnoses should be continuously assessed, re-evaluated and changed, and be considered dynamic tools with limited applicability outside of clinical and research contexts.^19:36

This is brilliant, but why do psychiatrists not say the same about adults? They also change over time and a person in deep distress will not always be in deep distress. That person might be fine both before and after the visit to the doctor. Why is it then impossible to get a wrong diagnosis removed?

The authors warned that one must not indulge in uncritical use of diagnoses, e.g. they are often used as an admission ticket to social services. They claimed that if clinicians respect the limitations and scope of diagnoses and limit their use of diagnoses for administrative and official purposes, diagnoses do not in themselves imply a risk of stigmatisation.

This looks like a tautology. If diagnoses are used correctly, they do not lead to stigmatisation. If people are stigmatised, it is because diagnoses are not used correctly.

The reality is that diagnoses are not being used correctly, which leads to a lot of stigmatisation and misery.^7,8 Think about other issues. If people drove correctly, there would be no traffic deaths. If people drank alcohol correctly, there would be no alcoholics. If people ate correctly, no one would be overweight. What does this tell us? Nothing.

I shall end this chapter by praising Australian psychiatrist Niall McLaren whom I have met several times. He has written a very instructive book with many patients’ stories telling us that anxiety is a key symptom in psychiatry.⁹ If a psychiatrist or family doctor doesn’t take a very careful history, they might miss that the current episode of distress, which they diagnose as depression, started as anxiety many years earlier when the patient was a teenager. They should therefore have dealt with the anxiety with talk therapy instead of handing out pills.

Niall has developed a standard way with which he approaches all new patients in order not to overlook anything important. It takes time, but the time invested initially pays back many times over and leads to better outcomes for his patients than the standard approach in psychiatry.

Niall explains that “the value of biological psychiatry is that it isn’t necessary to talk to a patient beyond asking a few standard questions to work out which disease he has, and that can easily be done by a nurse armed with a questionnaire. This will give a diagnosis which then dictates the drugs he should have.” Sarcastic? Yes. True? Yes.

It doesn’t seem to matter whether a diagnosis is correct or wrong. It will follow you for the rest of your life.

***

To see the list of all references cited, click here.