The DSM-5 Field Trials: Inter-Rater Reliability Ratings Take a Nose Dive

Jack Carney, DSW
0
1120

Does anybody know what’s going on around here?

Groucho probably had the best answer:

“I don’t know what they have to say,
It makes no difference anyway,
Whatever it is, I’m against it.
No matter what it is or who commenced it,
I’m against it.

Your proposition may be good,
But let’s have one thing understood,
Whatever it is, I’m against it.
And even when you’ve changed it or condensed it,
I’m against it…”

(The Marx Brothers, Horse Feathers, 1932)

The American Journal of Psychiatry (January, 2103) recently published a series of articles that analyzed the outcomes of the field trials that were conducted by the DSM-5 Task Force, to determine the inter-rater reliability of the multiple diagnostic categories that will comprise the DSM-5. For those who might not be familiar with the concept, inter-rater reliability refers to the extent to which two clinicians agree on a diagnosis when presented with the same clinical picture.

The measurement used to determine reliability is called the kappa statistic, described by Kirk and Kutchins (1992), DSM-III’s foremost critics, as a mathematical formula often utilized in social sciences research – e.g., anthropology, sociology – to minimize reliability occasioned by sheer chance. To quote from the first article I wrote on the subject of DSM-5 (November, 2011) “… High reliability would be signified by a kappa score of .7. In the DSM-III reliability field trials, five Axis I or II diagnoses achieved the .7 score; for the remaining diagnoses, the data analysis came close but never got there. The follow-up field trials for DSM-IIIR and –IV scored low in reliability particularly as regards personality disorders and disorders of childhood and adolescence.”

To go on … “… no alarms were raised, no protests made that the DSM authors’ claims for construct validity and reliability were never realized. Kirk and Kutchins’ explanation is that psychiatrists and other clinicians bought the rhetoric; further, the effective propaganda campaigns undertaken by the insurance and pharmaceutical industries completed the tautology — the unquestioning reliance of those industries on the DSM were proof of its reliability. Similarly, construct validity was no longer an issue because the medications, prescribed to treat biomedical illnesses, worked. Didn’t they?”

Richard Bentall made similar observations about DSM-IV in his 2004 review.

Now we have the American Psychiatric Association (APA) presenting its own critique of the newest DSM, presumably beating harsher analysts to the punch. (Any readers interested in reviewing the AJP articles published in its January, 2013, issue can access them at http://www.ajp.psychiatryonline.org, pages 43-82. The editorial, which I will reference below, can be found at American Journal of Psychiatry, Jan 2013; 170 (1); 1-5. doi: 10.1176/appi.ajp.2012.12091189.) Remember, the DSM-5 will be published on May 13; even now, the APA is pushing advance sales on its website. Interestingly, the AJP Editorial, which was signed by thirteen psychiatrists, advances the notion “that reliability is the first test of validity for diagnosis …” As it turns out, a noble but misplaced sentiment.

Immediately below, courtesy of Suzy Chapman of Dx Revision Watch and colleagues of the International DSM-5 Response Committee, is a table that tracks the downward progression of kappa scores of ten diagnoses over the course of the past thirty-plus years, i.e., from DSM-III through DSM-5.

 Kappa Scores: DSM-III thru 5

You’ll note that only one putative DSM-5 diagnosis, Major Neurocognitive Disorder, tops the magical .7 kappa, which the APA considers to signify “very good agreement” or high inter-rater reliability. That fact doesn’t deter the APA or the APJ editorialists, who simply move the goal posts. The magic now starts at .6 kappa, which means that three presumed disorders, two of them for children – Post Traumatic Disorder, Autistic Spectrum Disorder and Attention Deficit Disorder – are also considered to have “very good agreement.” Fifteen other disorders – four of them for children – whose kappa scores range from .4 to .59, are deemed to show “good agreement,” while twelve disorders, including six new children’s disorders (which are not listed above), whose kappas range from -.03 to .36, are considered “questionable” or “unacceptable.” Of greatest significance is the “questionable agreement” accorded Major Depressive Disorder, whose kappa scores have declined from a high of .8 in DSM-III to .32 in DSM-5. This result appears to have mortified the editorialists, particularly, I assume, since so many more of their patients, with the demise of the bereavement exclusion in DSM-5, are now candidates for that diagnosis.

So who’s in charge here? Ultimately, of course, the question is moot, since none of these diagnoses has any construct validity, i.e, any scientific basis for its existence. As the president of the APA reminded us only this past December, none of DSM-5’s presumed diagnoses have any biomarkers. Which makes me wonder all the more who the folks are who dream up these diagnoses and what they look like, particularly in the midst of their learned discussions. I imagine they look very much like the bearded and mortar-boarded professors in Horse Feathers who surround Quincy Adams Wagstaff, played by the inimitable Groucho Marx, the new president of Huxley U., as he’s introduced to the university community. There he is, onstage, fending off the academy’s wise men – could they be anything else? – whom he proceeds to tell, “Whatever it is, I’m against it.” Click on it and have a good laugh.

It all makes me wonder where our new Grouchos are, those irreverent anarchists who mock those who presume to claim authority over us and who inspire in us the courage to defy them. I’d like to think that, in a small way, that’s what our DSM-5 Boycott represents; opposition to irrationality and oppression and hope for a life free of arbitrary reductionism.

To quote another anarchist, this one a defiant rock’n’roller from Detroit, “… I’m not a number!

Remember… organize and say “No!” to the DSM-5 when it’s published on May 13.

We’re all prisoners of hope!

References:

American Psychiatric Association, “DSM-5 Field Trials,” Psychiatric News Alert, October 30, 2012

Bentall, R., Madness Explained: Psychosis and Human Nature, Penguin Books, London, 2004

Carney, J., “1984 Revisited: The New DSM,” November, 2011, Behavioral Healthcare

Committee to Boycott the DSM-5, “Position Statement & Petition & Blog Posts”
(latter also contains links to FB Cause page and Twitter feeds)

Frances, A., “DSM-5 Field Trials Discredit APA,” Psychology Today, October 30, 2102

Freedman, R., et al, “The Initial Field Trials of DSM-5: New Blooms and Old Thorns,” Editorial, American Journal of Psychiatry, January, 2013

Kirk, S.A., Kutchins, H., The Selling of DSM: The Rhetoric of Science in Psychiatry, Aldine de Gruyter, New York, 1992

Marx Brothers, Horse Feathers, 1932

Seger, Bob, “Feel Like a Number,” Stranger In Town, 1978

Support MIA

MIA relies on the support of its readers to exist. Please consider a donation to help us provide news, essays, podcasts and continuing education courses that explore alternatives to the current paradigm of psychiatric care. Your tax-deductible donation will help build a community devoted to creating such change.

$
Select Payment Method
Loading...
Personal Info

Credit Card Info
This is a secure SSL encrypted payment.

Donation Total: $20.00