Lancet Psychiatry’s Controversial ADHD Study: Errors, Criticism, and Responses

On February 17, 2017, Lancet Psychiatry published a controversial article that claimed to have found brain differences between children with a diagnosis of attention deficit hyperactivity disorder (ADHD) and children without. The authors, led by Martine Hoogman at the Donders Institute for Brain, Cognition and Behaviour, Netherlands, claimed that their evidence supported the theory that ADHD is a brain disorder.

In fact, Hoogman and colleagues go so far in their original article as to say “We confirm, with high-powered analysis, that patients with ADHD have altered brains; therefore ADHD is a disorder of the brain.”

This overstated conclusion ignited an immediate firestorm of controversy. Researchers as prominent as Allen Frances, chair of the DSM-IV task force (who wrote the book on diagnostic categories), joined to denounce the study. An article on which Frances is co-author explicitly stated:

“The most important argument against the authors’ conclusion that ‘patients with ADHD have altered brains’ is that it is not supported by their own findings.”

They add that the conclusion of Hoogman and colleagues “is wildly speculative and dangerously misleading at a time when ADHD is already overdiagnosed and overtreated with medication in high-income and middle-income countries.”

This was part of a larger push by researchers in numerous countries to examine the strange, problematic data presented by Hoogman and colleagues. The most obvious error in the original article was the swapping of the two variables in one analysis, which resulted in a chart showing that individuals with ADHD appeared to have higher IQs than healthy controls—a surprising finding that contradicted previous research.

In response to the criticism, Hoogman and colleagues re-examined their data and found that this was, in fact, an error—their findings were consistent with previous research suggesting that ADHD symptoms are associated with poorer performance on standardized IQ measures.

Researchers, led by Susanne Bejerot at Orebro University, Sweden, re-analyzed the corrected data and concluded that, when controlling for this new IQ data, many of the observed brain differences disappear.

“We found no significant difference between individuals with ADHD and those in the control group in any of the investigated areas of the brain when IQ difference is controlled for.”

In their response, Hoogman and colleagues acknowledge that “Because a slightly lower IQ can be a feature of ADHD, adjusting for IQ will remove disorder effects in brain regions associated with both ADHD and IQ.” That is, they acknowledge that when controlling for the differences in IQ, their supposed brain differences disappear. They argue that lower IQ is caused by ADHD and should not be statistically controlled. This is a controversial approach to ADHD research, as causation has not been demonstrated and low IQ is not a criterion for a diagnosis of ADHD. Thus, any brain differences found are likely to be due simply to differences in IQ, not to ADHD diagnosis.

Critics of the study (see MIA Report) also see this error as illuminating its fundamental problem: they contend that Hoogman and colleagues were too eager to prove their hypothesis and didn’t thoroughly examine what results might be supported by their data. Critics argue that this error should have been caught by Hoogman and colleagues, by the peer reviewers who approved the article, or by the editorial staff of the journal that published it. That such an unusual and compelling finding went completely unnoticed and unmarked indicates to critics that the authors, reviewers, and editorial staff were so biased by their perception of ADHD as a brain disorder that they ignored substantial evidence to the contrary.

Although this led to a petition to retract the article, the editorial staff of Lancet Psychiatry instead decided to publish a correction and then devote an entire issue to airing the criticisms and a rejoinder from Hoogman and colleagues.

Amongst the articles critical of the original study was an analysis led by Trudy Dehue at the University of Groningen, Netherlands. Dehue and colleagues found that the original results actually show that for adults, there was no difference in brain volume between those with ADHD and those without. Even for children, Dehue and colleagues argue that the small effect size found indicates that there was a considerable overlap in brain sizes between the two groups (95% of the two populations overlapped). That is, even on average there was so much overlap that only the extreme outliers were relevant.  Another way of looking at this finding is that 95 out of every 100 children diagnosed with ADHD had a brain volume that matched that of a child in the “healthy control” group.

According to Dehue and colleagues, the original findings “showed no significant differences in the brains of adults with ADHD, which suggests that the minor differences in children mostly vanish when they grow up. This finding could have been actual headline news given the claims by commercial pharma companies and sponsored experts that ADHD is a life-long disorder in need of life-long treatment.”

Dehue and colleagues, in fact, go so far as to explicitly state:

“There is no point in conveying that a child with ADHD has a brain disorder.”

Another article published by Lancet Psychiatry this month addressed the original finding that emotional processing areas of the brain were implicated as differing between children with an ADHD diagnosis and children without the diagnosis. The authors, Alison Poulton and Ralph Nanan at the University of Sydney, Australia, argue that, as before, Hoogman and colleagues should have highlighted this unexpected finding. Poulton and Nanan point out that there is no emotional component to a diagnosis of ADHD, and question why researchers would find such a brain difference. Their argument is that the diagnosis of ADHD often overlaps with other diagnoses commonly given in childhood, such as oppositional defiant disorder (ODD), the prominent feature of which is emotional dysregulation.

Poulton and Nanan argue that many of the findings of Hoogman and colleagues are actually due to the low specificity of the ADHD diagnosis and its overlap with other diagnoses. If the results are related to ODD rather than ADHD, this calls into question whether the data can be used to suggest that ADHD is a brain disorder.

Hoogman and colleagues simply agree with this criticism and state that they hope to investigate further. They do not address the implications that this has for the interpretation of their results.

The diagnosis of ADHD has been a controversial one. The primary criticism has been the loosening of diagnostic categories, which leads to overdiagnosis. Critics argue that the criteria for a diagnosis of ADHD are so loose that almost anyone could receive the diagnosis (one criterion is “feelings of restlessness,” for instance). Other studies have shown that it is the youngest children in classrooms who were more likely to receive a diagnosis, and they are more than twice as likely to receive stimulant medication than the oldest children. Researchers theorize that the youngest children struggle more in school and are at an earlier stage of development than children who are nearly a year older—which can make them more likely to fit the behavioral criteria for a diagnosis of a mental disorder.

A 2013 New York Times article exposed misleading pharmaceutical industry advertising practices (to both doctors and consumers), which have resulted in numerous FDA investigations and punishments, and linked them to increased ADHD overdiagnosis and overprescription.

Dr. Keith Connors, who helped shepherd the idea of ADHD into being and who conducted the first studies on methylphenidate (Ritalin), says that within the past few years he began to understand that his work has been disastrously misused. He says he “announced to stunned colleagues that the over-diagnosis of ADHD was “an epidemic of tragic proportions.”

“Tragic because many kids get the wrong diagnosis and really have a different problem that needs a different treatment: or they are normal youngsters given a treatment they don’t need; or the drugs prescribed for them are given away or sold to other students wanting a quick fix for studying or partying—a reason why schools and colleges now have huge numbers of students using stimulant drugs, and why emergency rooms are increasingly overwhelmed with overdosing youngsters.”

Hoogman and colleagues were obliged to respond to arguments by prominent researchers that their data supports the idea that ADHD is not a brain disorder. They had the good grace to admit some limitations, such as acknowledging that they had erred in reporting the IQ scores and that this would affect their results. However, despite their findings suggesting the exact opposite, they continue to argue that “ADHD is a disorder by all standards of psychiatric nosology.” That is, even if it is unsupported by their data, they claim that we must accept the common wisdom that ADHD is a brain disorder.

Hoogman and colleagues themselves sum up the argument of their critics nicely:

“The criticisms of small effect sizes raised by these letters imply that we should only use the term brain disorder when everyone with the disorder shows the same pattern of brain abnormalities. By that definition, no psychiatric disorder would be a brain disorder.”

 

****

Batstra, L., te Meerman, S. Conners, K., & Frances, A. (2017). Subcortical brain volume differences in participants with attention deficit hyperactivity disorder in children and adults. Lancet Psychiatry. http://dx.doi.org/10.1016/S2215-0366(17)30107-4

Bejerot, S., Nilsonne, G., & Humble, M. B. (2017). Subcortical brain volume differences in participants with attention deficit hyperactivity disorder in children and adults. Lancet Psychiatry. http://dx.doi.org/10.1016/S2215-0366(17)30160-8

Dehue, T., Bijl, D., de Winter, M., Scheepers, F., Vanheule, S., van Os, J. . . . Verhoeff, B. (2017). Subcortical brain volume differences in participants with attention deficit hyperactivity disorder in children and adults. Lancet Psychiatry. http://dx.doi.org/10.1016/S2215-0366(17)30158-X

Hoogman, M., Buitelaar, J. K., Faraone, S. V., Shaw, P., & Franke, B. (2017). Subcortical brain volume differences in participants with attention deficit hyperactivity disorder in children and adults – Authors’ reply. Lancet Psychiatry. http://dx.doi.org/10.1016/S2215-0366(17)30200-6

Hoogman, M., Bralten, J., Hibar, D.P., Mennes, M., Zwiers, M. P., Schweren, L. S. J. . . . Franke, B. (2017). Subcortical brain volume differences in participants with attention deficit hyperactivity disorder in children and adults: a cross-sectional mega-analysis. Lancet Psychiatry, 4, 310–19. http://dx.doi.org/10.1016/S2215-0366(17)30049-4

Poulton, A., & Nanan, R. (2017). Subcortical brain volume differences in participants with attention deficit hyperactivity disorder in children and adults. Lancet Psychiatry. http://dx.doi.org/10.1016/S2215-0366(17)30105-0