In the PLOS Blog Mind the Brain, psychologist James Coyne critiques a recent study in Translational Psychiatry — widely hailed in the media — where researchers from Northwestern University claimed to have discovered a blood test to identify depression.
Coyne begins by describing the press release and video that appeared alongside the study. “I do not know where the dividing line is between unsubstantiated claims about scientific research and sheer quackery, but this video tests the boundaries, when evaluated in light of the evidence actually presented in the article,” writes Coyne.
The study itself, says Coyne, was likely fatally flawed from the outset with its tiny sample size of 32 depressed patients and 32 controls, when compared to previous failed attempts to find biomarkers for depression that involved tens of thousands of participants. “So, just looking at the number of available patients and controls, we are not expecting much good science to come out of this study that is pursuing significance levels to define results,” writes Coyne. “I think that many persons familiar with these issues would [have] simply dismissed this paper out of hand after looking at these small numbers… Bottom line is that the authors cannot rule out lots of alternative explanations for any differences that they find.”
Coyne then provides a lengthy, detailed critique of virtually every step of the study’s process and analysis, and notes that, deep in the study, the authors themselves admit that some of their own findings debunked their conclusions. However, observes Coyne, the authors of the study “simply ignored having done this procedure and went on to discuss results as significant. If you return to the press release and the video, you can see no indication that the authors had applied a procedure that eliminated their ability to claim results as significant.”
Coyne also points to a US Institute of Medicine (IOM) expert panel report which sought to develop “an orderly process for validating” the growing plethora of biomarker tests aimed at the commercial market that virtually never “perform as advertised.”
“The committee came up with some strong recommendations for discovering, validating, and evaluating such tests in clinical practice,” writes Coyne. “By these evidence-based standards, the efforts of the authors of the Translational Psychiatry [article] are woefully inadequate and irresponsible in jumping from their modest size study to the claims they are making to the media and possible financial backers, particularly from such a preliminary small study without further replication in an independent sample.”
“I could go on about other difficulties in the study,” says Coyne, “but I think you can get the picture that this is not a credible study and one that can serve as the basis in search for a blood base, biomarker for depression. It [is] simply absurd to present it as such. But why get upset?” Coyne then explains exactly why we should all be upset.
How to critique claims of a “blood test for depression” (Mind the Brain, PLOS Blogs, September 25, 2014)