The Failure of “Spit For Science”: No Genetic or Neurological Pathways for Substance Abuse

Despite finding no meaningful correlation between genes and substance use, high-profile psychiatric geneticists misleadingly conveyed their results in a series of papers over several years. That’s according to prominent researcher Eric Turkheimer and his co-author, Sarah Rodock Greer.

In a recent article, Turkheimer and Greer critiqued the overblown conclusions drawn from a large, long-running, high-profile genetic study, Spit For Science (S4S). They reviewed every research paper published by the S4S researchers. Their finding: The S4S researchers repeatedly presented optimistic claims that weren’t supported by the data.

“We review every empirical paper involving genetic data that has emerged from the program, and reach a surprising conclusion: the study has never identified a single genetic effect of more than trivial magnitude,” Turkheimer and Greer write.

They add, “Most of the papers ignore the tiny effects, reaching optimistic conclusions about the prospects for future genetic explanations of alcohol use.”

Turkheimer and Greer are at the University of Virginia. Their paper was published in Philosophy, Psychiatry, & Psychology. A number of researchers responded to their article, and Turkheimer and Greer followed up in the pages of the same journal.

“We cannot predict the future any better than anyone else, but we will climb out on this limb: there will never be meaningful genetic or neurological pathways for alcohol-related behaviors, any more than there are for, say, bankruptcy. Refusal to accept this possibility, masquerading as ‘optimism,’ is what got S4S in trouble,” Turkheimer and Greer write in that follow-up.

College Kids Spit—For Science

S4S was the brainchild of Kenneth Kendler and his protégé, Danielle Dick, at Virginia Commonwealth University (VCU). Kendler is one of the most highly-cited and often-quoted researchers in psychiatry, whose work on the genetics of schizophrenia—a theory that has never panned out—captured the zeitgeist of the late 1990s and early 2000s. (Kendler has also been criticized for whitewashing the crimes of Nazi eugenicists.)

Dick and Kendler’s S4S began enrollment in 2011, aggressively recruiting from the incoming freshmen at VCU. The goal was “to put genetic knowledge to work in ways that might aid university administrators and mental health professionals in the prevention and treatment of substance abuse.” To achieve this end, the S4S researchers wanted a huge sample size, hoping to detect previously hidden genetic signals to provide individual-level clinical guidance around substance abuse for young adults entering college.

Turkheimer and Greer note that the S4S researchers did not seek to investigate if there was a genetic component to substance abuse, which would be the expected hypothesis for a study of this type. Instead, they assumed there was, and then designed their study to find evidence that would support this conclusion.

“The relevance of individual-level genomics was not a matter of investigation in S4S, not a hypothesis to be evaluated carefully by skeptical researchers: it was an assumption, and the repeated empirical contradictions of that assumption left the investigators with no rational framework to explain what was happening,” write Turkheimer and Greer.

By 2021, over 12,000 college students had participated in the program, with their DNA collected from saliva (hence the name “Spit For Science”) and their drinking habits and other behaviors assessed by survey every year. By the time Turkheimer and Greer were working on their article, there had been 16 empirical articles reporting genetic data from the S4S project, and about the same number of peer-reviewed articles about the study that did not report genetic data (reporting on study design, qualitative responses, and theorizing about implications, for example). In their analysis, Turkheimer and Greer focused on the 16 articles that provided actual data.

In most cases, the S4S researchers had found nothing—no genes that correlated with drinking behaviors. In some cases, they found an effect, but then another study proved that to be a false positive (the correlation disappeared when they attempted to replicate it). After trying a variety of statistical methods to come up with positive effects, the S4S researchers sometimes managed to gain statistical significance. But, according to Turkheimer and Greer, there is no individual clinical relevance to effects that require having 12,000 people in order to spot a tiny correlation (which could be a false positive anyway). What is that supposed to mean for an individual college student?

“Again and again, the results of S4S investigations demonstrated tiny, and frequently null, contributions of genetics to differences in drinking and related behaviors among college students,” Turkeimer and Greer write. Genome-wide association studies (GWAS) “revealed practically no genome-wide significant SNPs, and the SNPs identified at less stringent levels of significance were never mentioned more than once. The only candidate gene with a greater than trivial association with alcohol use, ALDH2, was already well-known from linkage analysis. SNP-based heritabilities were small, and most of the time not significantly different from zero. Despite considerable optimizing, predictions from PGSs [polygenic risk scores] rarely accounted for more than 1% of the variance, and were mostly not statistically significant.”

As Turkheimer and Greer concluded: “Nothing even resembling biologically based ‘prevention and intervention programming’ was developed.”

“The Authors Ignore Their Own Results”

However, in their articles reporting these null findings and meaningless tiny correlations, the S4S researchers spun these results into something supposedly positive. According to Turkheimer and Greer, “the most striking feature” of these articles is the “disconnect between the goals of the studies as specified in the introductions of the papers, the quantified findings tabled in the results, and the conclusions reached in the discussions.”

“The authors ignore their own results,” write Turkheimer and Greer.

For instance, in many cases, genetic correlations showed up, but were likely false positives. Once Turkheimer and Greer corrected for multiple testing (which leads to inflated results by chance), these correlations disappeared.

R² statistics, in which a value of 0 indicates no relationship, and a value of 1 means that the genes are responsible for 100% of an effect, were typically lower than 0.01, sometimes much lower. This means that genetics explains less than 1% of whether someone drank or not, with many of the articles finding that it explains a lot less than that, even—about as close to zero (no relationship) as one can get without actually being zero.

Here are some results from the studies:

• “The median effect size of the 135 associations was R² = 0.0009”
• “The PGS [polygenic risk scores] accounted for 0.002 of the variance in alcohol consumption (p = 0.031); the interaction accounted for an additional 0.0006 (p = 0.035).”
• “Using the standard formula of R² of χ²/N gives an approximate value of R² = 0.002.”
• “Across different ethnic samples, PGSs and outcome measures, variance accounted for by the PGS ranged from 0.0006 to 0.0019 of the phenotypic variance, except for the score for post-traumatic stress disorder (PTSD) in the AA sample, which accounted for 0.0042 of the variability in resilience, but was in the wrong direction, with higher PTSD PGSs associated with greater resilience. None of the associations with PGS reached statistical significance.”

Yet, these tiny correlations were misleadingly reported by the S4S researchers as supporting their assumption that genetics is the key to understanding—and preventing—substance abuse.

As another example, Turkheimer and Greer note that ALDH2 has long been associated with drinking, since it is the gene that leads to alcohol-related flushing in Asian populations. So, they write, identifying this gene provides no additional information.

Indeed, S4S actually demonstrated that ALDH2 (despite its known effect on flushing) may not have anything to do with drinking behavior. When the S4S researchers incorporated peer pressure into the equation, they found that ALDH2 no longer had a statistically significant effect on drinking behavior. Peer drinking significantly influenced whether an individual drank, but ALDH2 did not.

This was a consistent finding, with peer drinking showing a larger effect than genetics across numerous studies, often including cases where the genetic correlation was nonexistent.

Even when the S4S researchers admit that they found nothing, they argue that the impact of genetics is still there—it must be there—and they just need more funding, more participants, more statistical analyses, and they will finally prove it.

For instance, Turkheimer and Greer highlight this quote from one of the S4S studies: “No significant main effect of alcohol dependence polygenic risk scores or [gene–environment interaction] effect were found for predicting alcohol use disorder symptoms, suggesting the need for larger sample sizes in gene discovery efforts to better characterize genetic risk for alcohol problems, with a particular need for increasing representation of populations of non-European descent in these efforts.”

Moreover, as Turkheimer and Greer note, even the trivial correlations occasionally identified by the S4S researchers are likely due to questionable research practices, ways of analyzing the data that are known to inflate effects and give misleading results.

“The reports have a consistent tendency to emphasize statistical significance over effect size, and moreover to use statistical significance as a means of distracting from tiny effect sizes,” they write.

“Statistical significance has little meaning under these circumstances in any event, with numerous ad hoc hypotheses that have not been pre-registered, and significance cutoffs that were usually not corrected for multiple comparisons.”

Don’t Create the Torment Nexus

What is the core belief of the psychiatric genetics movement, led by these prominent researchers with millions of dollars in federal grants? Eugenics, based on Danielle Dick’s own writing. Turkheimer and Greer cite a 2022 paper by Dick, in which she expresses her excitement about the possibilities of eugenics found in the movie Gattaca:

“In an opening scene, a nurse rattles off a baby’s genetic propensity to develop a number of health problems: ‘Neurological condition: 60% probability; Manic depression: 42% probability; ADHD: 89% probability.’ The movie imagines how that information could reshape society and have a profound impact on the lives of individuals who know their genetic proclivities and vulnerabilities from birth,” Dick writes.

Gattaca is a science fiction film in which a dystopian society uses eugenics to create classes of undesirables who are discriminated against and forced to do manual labor. The protagonist, Vincent Freeman, is one of these undesirables who uses his intelligence and charm to rebel against the oppressive regime and achieve his dream of going to space.

To watch the movie Gattaca and identify with the fascist dystopia, rather than the protagonist railing against its discrimination, is a mind-bogglingly bad take.

Luckily, according to Turkheimer and Greer, the S4S results show that we are very far from actualizing the kind of dystopian society that psychiatrists like Dick and Kendler dream of: “The empirical facts of applied psychiatric genetics have shown that current science is not up to the task of improving behavior, even if we allow ourselves to wish that it were.”

Still, “The goal of improving human behavior via genetics has reliably led to moral disaster,” they note. “Unrealistic optimism about the applied value of science always comes with a price.”

Responses

Philosophy, Psychiatry, & Psychology devoted a whole issue to the conversation around Spit for Science. In addition to Turkheimer and Greer’s critique, there were responses from six other groups of researchers, plus another response from Turkheimer and Greer summing up the conclusions.

Some of the researchers wrote to add a bit of context, explanation, or even further critique S4S. For instance, Bennett Knox, Hannah Allen, and Stephen M. Downes added to the eugenics discussion by writing that even if the S4S researchers had found clinically-relevant genetic correlations with substance use, “it is still not clear how this information could be used in an ethical manner to address the problem of campus drinking.” Essentially, creating a Gattaca-like dystopia on a college campus doesn’t sound like a good idea to them.

An “Exciting Sales Pitch”

John Z. Sadler wrote that the misleading spin by the S4S researchers may be due, in part, to systemic issues like the grant-writing process. Obtaining grants is imperative for academic researchers, as grants fund their labs (and make them appealing for tenure by bringing millions of dollars into their university). But grants are given on the basis of promised findings, on optimistic future data. So, Sadler writes, of course the S4S researchers had to imply that they were always just on the verge of finding the genetic data that would validate the whole project.

Sadler adds that genetics is a lucrative field, quite appealing to the big donors—at the expense of funding research that could actually lead to clinically relevant findings. “The psychiatric genetics research enterprise was and had been an extraordinarily expensive enterprise, when other research questions that could be immediately implementable into clinical care improvements remained marginally funded at best,” he writes.

One reason is that investors believe they’ll get a monetizable device or algorithm out of funding genetic research. To Sadler, this highlights how the money goes to corporate interests rather than to immediate clinically relevant questions. “The power of money and corporate alliances in academic medicine serves to marginalize less-profitable enterprises in academic medicine,” he writes. “The search for magic bullets in medicine and psychiatry makes for exciting sales pitches to legislators, pundits, and lobbyists on Capitol Hill.”

The Defenders

There were also plenty of responses, including one from Danielle Dick herself, that aimed to defend S4S. The striking thing about the defenses, though, is that they don’t actually argue the points made by Turkheimer and Greer. All the defenders admit that S4S found no meaningful genetic information, no result that can be used clinically—that it was, overall, a complete disappointment.

However, just as in the original S4S papers, the defenders argue that the supposed genetic correlation must still be there—despite all odds, despite all the evidence, despite the failure of every study—still there, waiting in the wings for the next new technology to discover and finally prove what these defenders know to be true—that it is our genetic code that makes us who we are.

For instance, here’s a defense of the S4S project by Awais Aftab: “I have no substantial disagreements with Turkheimer and Greer. The main point I wish to convey is that there is reason to hope that findings from genetic research will contribute to clinical progress in psychiatry in an incremental and iterative manner.”

“There are hints,” Aftab writes, “that genetic associations can point towards brain mechanisms.”

This assertion begs a question: what are these “hints”? Aftab points to the same tiny, meaningless correlations that Turkheimer and Greer critique for being false positives or having no actual relevance. For instance, Aftab cites a genetic study with 435,000 participants that managed to create a polygenic risk score (PRS) containing hundreds of phenotypes… that explained less than 1% of whether a person had “problematic alcohol use.”

Aftab admits, “At present, PRS for problematic alcohol use has little predictive value at the individual level and is practically worthless from the standpoint of clinical practice.”

However, Aftab, even after agreeing with Turkheimer and Greer that false positives are meaningless, and acknowledging the larger failure of genetics research in psychiatry to produce any meaningful results, still finds that “it is reasonable to believe that genetics can and will have a role to play in understanding and ameliorating psychiatric distress and disability.”

Turkheimer and Greer respond that Aftab is just “indulg[ing] the same impulse that got S4S in trouble: setting the evidentiary bar for psychiatric genetics so low that the particulars of actual research findings cease to have any meaning.”

In a similar vein, well-known twin-study researcher Claude Robert Cloninger agrees with Turkheimer and Greer that genome-wide association studies will never find the genes responsible for behaviors like substance use. However, according to Cloninger, that’s because the supposed connection is so complex that we haven’t been able to figure it out yet. Again, the assumption is that the connection is there, and the next technology, the next big leap will surely prove what we all know to be true.

“He never explicitly says so, but it is hard to escape the conclusion that he agrees with most of the other commenters that at some level of complexity, this must be the case,” Turkheimer and Greer write (emphasis in original).

According to Cloninger, the issue is that psychiatry is “hostile” to the idea of using machine learning and artificial intelligence to figure out the complex genetic interactions. Turkheimer and Greer dispute this as well: “Complex behavioral phenotypes, we continue to insist, do not necessarily have meaningful representations in genes or neurons, no matter how sophisticated the model.”

They conclude, in their response to Cloninger, that “some phenomena defined on a larger-scale level of analysis must be explained at that same level of analysis, and cannot be translated in meaningful ways into the molecules of which large-scale entities are undeniably composed . . . We do not ask economists to build their science on the chemical composition of money.”

“The Sound You Hear Is the Genetic Goalposts Being Moved Out to Midfield.”

Perhaps the most telling defense of S4S comes from its own leader, Danielle Dick, who distances herself from the project: “As of 2021, I am no longer involved with the project, having ceded its stewardship to another VCU faculty member.”

Dick argues that S4S was not intended to be “research” at all, but just a “registry,” a database of genetic material (despite the fact that the initial proposal documents showcase clear intentions for individual, actionable clinical results from their project).

Indeed, Dick herself has created a “Personalized Feedback Program” for college students, ostensibly based on the S4S findings. The problem with that program, though, is that “not a single finding from S4S is cited in either of the papers that have been written about it,” according to Turkheimer and Greer. Instead, it is more of the same from psychiatry: a questionnaire that asks about psychological factors that are known to predispose to drinking, like impulsivity. There is nothing “genetic” about it.

“Committed to the idea that risk for alcohol abuse is substantially genetic, but unable to produce any concrete genetic evidence in her participants, she devised a Personalized Feedback Program that includes no genetic data at all,” write Turkheimer and Greer.

Dick, in a moment of self-praise, writes of how she “raised awareness about substance use” in college kids and how their 2021 paper “was used to advocate for resources to support mental health at a time before there was widespread recognition of how mental health challenges impact educational performance.”

Ultimately, she writes, “I do not view understanding biology as the central goal of psychiatric genetics.” Yet, she then explains, she has told “thousands of parents, medical providers, journalists, and individuals with substance use disorders and their families” about how substance use is genetic, and thus, biological. This audience, she writes, has found her words “enlightening and useful.”

Dick claims she isn’t part of the project, it was never a research project anyway, genetics is about something other than biology, and also somehow her research project about genetics and biology was a success that everyone finds “enlightening and useful.”

As Turkheimer and Greer write, “The sound you hear is the genetic goalposts being moved out to midfield.”

“It has become commonplace, now that it is clear that we are not going to find genes for complex human behavioral phenotypes, to deny that the goal of behavioral GWAS was once to find them. Now that we know that genetic effects are tiny, it is claimed that we never expected them to be large. Now that we have failed to identify meaningful biological bases of complex behavior, it is denied that we ever thought those biological bases existed. Now that we know samples in the hundreds of thousands are required to identify anything at all, it is claimed that we never thought 12,000 constituted a large sample. In her reply, Dick goes so far as to deny that S4S was a research program at all. Perhaps it was just a registry, designed to combine with other studies by other investigators while giving students an opportunity to participate in research,” they write.

“Unscientific Wishcasting”

The last ardent defense of psychiatric genetics comes from James J. Lee and Damien Morris, who write that S4S is small fry compared with the big GWAS studies that include hundreds of thousands of participants. They “characterize tiny effect sizes as a feature, not a bug, of psychiatric genomics,” according to Turkheimer and Greer. Lee and Morris argue that bigger and bigger sample sizes will find more and more of these infinitesimal effects—but Turkheimer and Greer ask, “so what?”

“Willingness to accept small, correlational, causally eerie associations as proof of a vague concept like the ‘biology of drinking behavior’ is exactly what led S4S down a conceptual dead end in the first place,” Turkheimer and Greer write.

Ultimately, if it requires hundreds of thousands of participants to detect the tiniest correlations, which may be false positives or confounded with environmental effects, then the impact of genetics is so weak, so meaningless, that it is irrelevant to any understanding of “causation.” If it takes researchers hundreds of thousands of people to determine that a combination of hundreds of genes is responsible for less than 1% of whether you drink alcohol, then the conclusion to draw is that your genes are irrelevant.

The defenses of S4S, then, can be fairly summarized in this way: We researchers failed to find anything clinically relevant, but we can be confident that with bigger studies and new technologies we will definitely figure out what we all know to be true.

Turkheimer and Greer have the perfect answer to this: “Hypothetical discussions of future scientific outcomes usually turn out to be unscientific wishcasting. It is like insisting that cities need to develop traffic control systems for flying cars.”

In conclusion, write Turkheimer and Greer, “psychiatric genetics at scale has failed to solve the fundamental problem for which it was designed, that of understanding the “biology” that explains complex differences in human behavior.”

They add, “The problem was not that S4S failed to estimate the relevant genetic effects on alcohol use; the problem was that it succeeded.”

 

****

MIA Reports are supported by a grant from Open Excellence and by donations from MIA readers. To donate, visit: https://www.madinamerica.com/donate/