In a new article in JAMA Psychiatry, researchers suggest that “data pollution” impedes psychiatric research. They write that there are many aspects of data pollution, and experts in specific psychiatric research are ill-equipped to account for all of them.
“Neuropsychiatric research is substantially impeded by issues surrounding data collection and analysis. While these issues have been extensively discussed, their severe impact on neuropsychiatric effect sizes is not as widely recognized,” the researchers write.
The researchers were Alessandro S. De Nadai at Texas State University, Yueqin Hu at Beijing Normal University, and Wesley K. Thompson at the University of California, San Diego.
De Nadai, Hu, and Thompson focus on data pollution, which they define as “inadvertent errors” in the data. This is distinct from “data poisoning,” which involves “intentional attempts to feed inaccurate data into models.” The current article focuses on well-intentioned researchers whose results are misleading by accident.
This is common, according to De Nadai, Hu, and Thompson. Moreover, they write that researchers in neuropsychiatry come from such varied backgrounds that none of them are experts in every potential form of data pollution and how to mitigate it.
For instance, data pollution can come from any of the following areas: “(1) unreliable measurement, (2) heterogeneous construct definition, (3) population mixtures with differing biopsychosocial mechanisms, (4) behavioral reporting bias by both patients and clinicians, (5) selection bias, and (6) data that are not missing at random.”
What these have in common is unreliability or “noise.” All of the tests and definitions in psychiatry have varying levels of subjectivity and are influenced by an almost infinite array of factors in a person’s life. Especially when a study uses multiple tests or attempts to account for moderation or mediation (whether certain factors are influenced by others), this noise can add up. In the end, the effects that researchers find are unreliable and often inflated.
“Inconsistent and inaccurate effect size estimation pollutes the research literature and makes it nearly impossible to build incrementally on small but important findings, which will be critical for future progress,” the authors explain.
They note that if physics research had the same level of unreliability, systems like GPS would be impossible to develop.
De Nadai, Hu, and Thompson also focus on the reliability and validity of psychiatric diagnoses. They note that even clinicians often disagree about whether a patient meets the criteria for a specific diagnosis, and patients often have a very different perspective. They add that diagnoses like depression and schizophrenia are extremely heterogeneous, lumping people together who have very different traits, feelings, and behaviors. This makes it very difficult to do research that might generalize to real-world patients.
The authors suggest that there are specific ways of accounting for the various types of data pollution and that researchers should have a “data pollution mitigation plan” before beginning their study.
“Without attending to data pollution,” they write, “much of our progress will be illusory, and true findings that improve patient welfare will remain undetected.”
****
De Nadai, A. S., Hu, Y., & Thompson, W. K. (2021). Data pollution in neuropsychiatry—an under-recognized but critical barrier to research progress. JAMA Psychiatry. Published online December 1, 2021. (Link)
“They add that diagnoses like depression and schizophrenia are extremely heterogeneous, lumping people together who have very different traits, feelings, and behaviors. This makes it very difficult to do research that might generalize to real-world patients.”
Part of why the DSM should be flushed.
Report comment
“De Nadai, Hu, and Thompson focus on data pollution, which they define as “inadvertent errors” in the data. This is distinct from “data poisoning,” which involves “intentional attempts to feed inaccurate data into models.” The current article focuses on well-intentioned researchers whose results are misleading by accident.”
—–
Where I read this is where I get mad. Or perhaps not quite as mad as in frustrated.
Especially when I hear the words “well-intentioned.” Because I feel like saying “no, there IS ill intent.”
Somewhere, at least. But I feel as if passivity and not caring IS ill intent or should be regarded so. Or people should be afraid enough of being accused of doing something intentionally so that they will be more than well meaning dopes but will think hard enough to avoid the “well intentioned error.”
There is also such a thing as ill intent in education. Perhaps no one person having malicious intent, but everyone showing lack of responsibility and passing the buck. “So long as I get my money, I don’t care.”
But, you know what? Sometimes you have an affirmative obligation to take responsibility. If your work is having an impact on people’s lives. And isn’t just about you climbing the ladder and having your career.
Actually, quite often, researchers will just do anything to get published, because they need to get published in order to have their careers. And they become slaves to “what the media will publish.” And that comes first rather than the whole notion of doing good.
And then they go have families and want to support them financially and all that. Well, ok, their children come first. They need to get published and make a splash so they can get promoted.
And they were well-intentioned and thought the research data was sound, and as for selection bias or other problems, well they just hadn’t thought of that.
Actually, how do you know someone is well intentioned? A lot of people are very good at coming across as well-intentioned, even while they are systematically doing all that is in their self interest. How do we know that they didn’t privately think of it in their own heads, some problem regarding selection bias that only could have been fixed by them getting a second grant to help them investigate or clarify certain issues? So they know FOR SURE?
Well, it’s not going to make as much of a splash if researchers say “we did a study that suggests one thing — however, we realized, if you look at it this way and if you look at it that way, it’s possible the data might be ambiguous, and we can only find out for sure if we get more funds to do a more careful analysis.”
I have a feeling their departments tell them “NO, don’t go there.” In other words, don’t ask those tough questions in the first place. Because there is a system and it’s like a machine and you have to work it in order to get ahead.
I’m reminded of the Bill Clinton Monica Lewinsky saga, where democrats were all saying “it’s only about sex, he lied about sex, everyone does it, everyone lies about sex, so he should get away with it.” It made me disgusted, even though I was otherwise a libertarian, and perhaps that’s because I’d already, by then, dealt with the culture of academia. Where indeed, well, you can maybe call them ill-intentioned. But, at the same time, you can almost excuse them by saying “well, everyone does it.”
Everyone, in isolation, could convince themselves “everyone does it, so it’s ok if I do it too.”
OK, fine, but we have a nation of fat people now and it’s partly because nutrition research was corrupted by food industry donations in the 60’s, and researchers were intimidated by an academic HERD.
How can we get to a place where we, as a nation, are NOT a nation of fat people due to such corruption but, instead, are healthy?
At some point I think the solution is that we need to make those people who are “only well intentioned” but goofed in one way or another pay the price AS IF they had evil intent. Start doing that, and watch how quickly people start cleaning up their act so fast.
Maybe we need to say, as a society: “you need to have accurate research findings, and if you put something out there that’s problematic, and people listen to it and follow it and it hurts them, you need to go to jail. EVEN IF it’s not your fault and you didn’t mean it but you were just too stupid to have thought of something.”
How much value do we place on the health of the American people as a society?
Report comment
Actually, you know what’s interesting? My mother was involved in the nutrition research of the 60’s at the Harvard School of Public Health. And she told me, the nutrition research of the 60’s was bogus and bought and paid for by the food industry.
https://ur.booksc.eu/book/73513993/02afd8
Actually, here is one of her articles — you can download the PDF.
This was about obese adolescent girls.
This was a paper where two men had tried to write an article, it didn’t get published. Or wait, they tried to write two articles. She combined the two articles into one and this was published. And then Harvard fired the two men and tripled her salary.
However, she later on had criminal things done to her, by the mafia. And, actually, she was herself a guinea pig as a child, where her mother was forced to feed her on a super strict rigid schedule — every six hours as an infant — and this messed up her appetite regulation mechanism of her brain, so she had trouble keeping her weight down unless she took amphetamines. And had an artificial weight problem as a result, which wasn’t what she naturally would have been. Which was a bit sad because she was otherwise a very attractive woman and was able to look quite glamorous when her weight was down.
Anyway, my mother told me about being bullied by groups of women acting as a gang and engaging in social ostracism, or otherwise having lots of problems with women engaging in vicious intimidation of her. Some of which I witnessed as a kid. And which went above and beyond normal melodrama. E.g., it was probably incentivized by the mafia. And in part something food companies paid bribes for. In other words, my mother was just bullied and bullied and bullied for years, mostly by women, and this was an important tactic which very effectively worked to silence anyone who would be whistleblowers over the nutrition corruption.
Anyway, one of the bits of “inside information” my mother told me about this article I post was that Harvard researchers withheld information. The obese girls were all catholic and the non-obese girls were all protestant.
I think this suggests that something went on with the mafia which resulted in systems of extortion and blackmail which effectively worked to help enslave Irish and Italians so they could be used as guinea pigs for unethical medical experiments. And a lot of Jews too, actually. Pretty much anybody was at risk, but newly arrived immigrants the easiest prey for these schemes.
I can say this because I know it was true for my own family — on my mother’s side at least. Italian – Jewish. I won’t go into details but several family members did have medical care that was just plain abnormal and had to have been some sort of unethical experiment. On both the Jewish and Italian side.
And it’s possible same thing might have happened on my father’s side, which was German and English. Well, yeah, I heard a few things there that didn’t sound right. But I was not as close to them.
Report comment
Date pollution clouds research and horrible media coverge encourages people to say things like this commentator’s thoughts in reaction to another piece of crap barely one sided piece on psychiatry, this one found in the Seattle Times:
https://www.seattletimes.com/seattle-news/mental-health/why-its-often-hard-for-people-to-recognize-their-own-mental-illness/#comments
Ghost of Hitler speaks: “This is why we need a robust involuntary commitment program where parents, teachers, and the medical community can have a person put into a treatment center. A good indication of mental illness is someone living on the street.”
Report comment
Peter, thanks for another great article! This study (hi Alex!) makes some great points that together help explain why decades of biological psychiatry research has accomplished essentially nothing of value “that might generalize to real-world patients.” But undoubtedly the same kind of research will continue with the same enthusiasm, unfulfilled promises that are never held accountable that we are on the cusp of a revolution that will transform everything, publications, grants, academic jobs, tenure and promotion, strong reputations, pharmaceutical company gifts, and so on.
Why? Because this entire research enterprise is not and never has been about helping “real-world patients.” It is about acquiring resources for researchers. That is the point of a publication for a researcher – to pad the CV, get a job, get a grant, become a journal editor, get a book deal, get invited to present highly paid seminars, have people stare at you in awe at conferences, and so on. That is why researchers naively and intentionally engage in scientific sloppiness and misconduct. Trust me, I was trained to do so and accepted this fake world as a seductive and valid reality before a few years of real-world practice knocked some sense into me.
Research is not about the patients. It’s about the researchers. And it’s not even really the researchers’ fault unless their work is deliberately fraudulent. It’s how the entire system is designed. It’s the natural consequence of incentives being followed.
The sooner we can all understand this, the sooner we can collectively dismiss almost all of 40 years of biological psychiatry research (and that’s just for starters) into the dustbin of history and start over. But that won’t happen because the influential leaders needed to do this are all beholden to the incentives that drive shitty research. It’s a vicious cycle and it’s difficult to imagine a way out.
There is a reason why wise practitioners ignore 99% of what counts for psychiatric research and it’s not that they are ignorant anti-science buffoons. It’s that the are switched on enough to see through the con. Because for them, it’s all about the patients.
Report comment
Good to see some honesty about the absolute mess that is psychiatric research. A steaming toxic dump of unintended consequences. Still somehow millions are being ploughed into genetic research that will eradicate madness from the Earth.
The difficult part is coming to terms with your own personal take on madness, your own sub-routines which link in and branch off. And then locating your personal madness within a broad historical trend of cultural insanity. There are legitimate and sensible reasons why most people put a lot of energy and effort into avoiding that process.
Fun also to see that naughty word “pollutant” being used metaphorically, while at the same time lots of good, legitimate science is revealing that a lot of mental illness might be better described as environmental illness or climate illness or diet illness, as unaccountable suffering and agony is no doubt being caused by the polluted air we breathe, the polluted soils and the polluted food chains and the polluted rains and the polluted rivers that flow into the polluted seas.
But hey, let’s not be too pessimistic about all this. We’ve got 5 years to turn all this around. It simply requires science, positive thinking, setting personal vanity goals, and finding new and inventive ways to distract oneself from one’s self.
It’s going to be okay because it has to be okay. Keep crunching the numbers, popping the pills or resisting the pills. At some point it will all somehow resolve itself, magically.
It’s why psychiatry does so well, despite all the many shortcomings. It offers a straight face to magical thinking and the belief that the apocalyptic “soul” of human beings can be resolved, long-term, with careful thinking and sustained determined effort.
Report comment