Editor’s note: This is a copy of a letter that Ed Pigott and Jay Amsterdam sent to Ned Kalin, M.D., editor-in-chief of the American Journal of Psychiatry.
***
Ned H. Kalin, M.D.
Editor-in-Chief
The American Journal of Psychiatry
Dear Dr. Kalin,
Dr. Jay Amsterdam and I are writing this Open Letter to call for the retraction of the five STAR*D articles published in the American Journal of Psychiatry (AJP) because the main findings of these articles are not accurate.1-5
We are the primary investigators (PIs) of the BMJ Open article that reanalyzed the STAR*D patient-level dataset.6 The reanalysis demonstrates that the rates of remission were dramatically lower than reported in AJP when adhering to the NIMH-approved protocol and related publications.
The reanalysis was funded by the Restoring Invisible and Abandoned Trials (RIAT) initiative. In 2019, we published a Call to Action in the British Medical Journal documenting our concerns regarding protocol violations,7 gave the STAR*D PIs fair warning of our intentions, and asked whether they would undertake the reanalysis themselves. The STAR*D PIs declined.
Prior to publication in July 2023, BMJ Open’s editor invited the STAR*D PIs to respond to our article and once again, they declined.8 We therefore were surprised to read in AJP’s December 2023 issue the letter by the STAR*D PIs titled, “The STAR*D Data Remain Strong: Reply to Pigott et al” in which they sought to discredit our reanalysis and reiterated their claim of a 67% cumulative remission rate.9
Coincidentally, the December 2023 cover story in Psychiatric Times (PT) focused on our reanalysis titled, “STAR*D Dethroned?”10 It was written by Dr. John Miller, PT’s Editor-in-Chief. Dr. Miller lauded the scientific rigor of our RIAT reanalysis and stated:
“For us in psychiatry, if the BMJ authors are correct, this is a huge setback, as ALL of the publications and policy decisions based on the STAR*D findings that became clinical dogma since 2006 will need to be reviewed, revisited, and possibly retracted.”10, p.16
Dr. Miller concluded PT’s cover story stating:
“In my clinical opinion, it is urgent for the field of psychiatry to reconcile the significant differences in remission rates for patients with MDD as published in the original STAR*D article in 2006 with the reanalysis just published in the BMJ article this year.”10, p. 17
Dr. Amsterdam and I concurred with Dr. Miller’s assessment, and therefore submitted an article to PT that responded to both the cover story and the STAR*D PIs’ letter that AJP published. Once accepted, we asked Dr. Miller to forward our article to the STAR*D PIs so that they could publicly respond to both the PT cover story and our article documenting their scientific errors.11
Despite having authored seven articles documenting STAR*D investigators’ scientific errors, my coauthors and I were never successful in getting the PIs to respond to our published criticisms.6, 7, 12-16
However, the calculus changed with the hard-hitting December 2023 PT cover story—and the PIs’ foreknowledge of our soon-to-be-published article—whereby the STAR*D PIs were essentially forced to provide their best response in PT’s March edition titled, “Rejecting the Accusation of a Violated STAR*D Protocol”.17 Unfortunately, their responses did not adequately address the main points that we made in our PT article.
The following are the five key scientific errors we documented in our PT article—each one of which is grounds for retraction. Each error we list is followed by the STAR*D PIs’ response, and then our reply.
Scientific Error #1
Without public disclosure, STAR*D PIs used data from 931 patients they deemed ineligible for analysis in step 1 because these patients lacked a blindly-administered Hamilton Rating Scale for Depression (HRSD) score of ≥14 at entry into the study, in their steps 2-4 and summary articles. This included 99 patients who scored <8 on their baseline HRSD—STAR*D’s remission criterion.
In contrast, our reanalysis of treatment steps 1-4 adhered to the STAR*D PIs’ stated criteria; namely, that patients must have a blindly-administered baseline HRSD score of ≥14 at entry into the STAR*D study to be included in data analysis.1, figure 1 This resulted in 3,110 evaluable patients.
STAR*D PIs’ Response:
“In total, 941 patients included in our original analyses were eliminated from Pigott et al’s reanalyses based on their post hoc criteria.” 17, p.7
“Effectiveness trials by design aim to be more inclusive and representative of the real world than efficacy trials. By removing the data of more than 900 study participants from their reanalyses, Pigott et al failed to recognize the purpose of inclusiveness. It appears that the authors created rules to define post hoc which participants to include.” 17, p.7
RIAT Investigators’ Reply:
We applied the STAR*D PIs’ stated criteria for inclusion in data analysis: i.e., a baseline HRSD score ≥14 at entry into the study. 1, figure 1 Furthermore, we also applied the PIs prespecified exclusion criteria that stated “participants who begin a level with HRSD <8 will be excluded from analyses.”18, p. 130
This is the EXACT opposite of creating “rules to define post hoc which participants to include.” We simply applied the rules for inclusion in data analysis that the STAR*D PIs’ prespecified. It is the STAR*D PIs who changed these rules post hoc and did so without public disclosure.
Scientific Error #2
Without public disclosure, in their steps 2-4 publications, the STAR*D PIs included 125 patients who scored as remitted prior to starting their next-step treatment. This occurred despite the PIs prespecifying that “participants who begin a level with HRSD <8 will be excluded from analyses.”18, p.130 Moreover, the STAR*D PIs double-counted the vast majority of these patients as remitted in treatment steps 2-4.
In contrast, our reanalysis counted such patients as remitted only once—as prespecified—and then excluded them from subsequent analyses in steps 2-4.
STAR*D PIs’ Response:
“The cover story appears to accept the contention by Pigott et al that we further ‘inflated’ the cumulative estimate of remission by ‘violating’ the STAR*D protocol to include the data of patients who had low HRSD scores at various baseline assessment points (some patients had baseline assessments for up to 4 steps of treatment).”17, p.7
RIAT Investigators’ Reply:
The STAR*D PIs prespecified that “participants who begin a level with HRSD <8 will be excluded from analyses”18, p.130 as this was their primary outcome criterion for remission.
Despite this explicit prespecified prohibition, STAR*D PIs do not challenge the FACT that they included patients in their data analyses who were scored as REMITTED on the HRSD prior to starting treatment:
- 99 patients who scored as remitted at entry into STAR*D, and an additional
- 125 patients who scored as remitted prior to starting their next-step treatment.
Scientific Error #3
Without public disclosure, the STAR*D PIs failed to acknowledge the FACT that the NIMH-approved protocol specifically excluded all clinic-administered assessments such as the QIDS-SR from use as research outcome measures because they were nonblinded and used to guide treatment.19
Despite this explicit protocol-stipulated prohibition, STAR*D PIs used the clinic-administered QIDS-SR as the secondary outcome to report remission rates, and sole measure to report response rates, in their steps 1-4 articles and the sole measure to report outcomes in their summary article.
The protocol-specified primary outcome, the HRSD, should have been used to report remission and response rates in all articles.
STAR*D PIs’ Response(s):
STAR*D PIs state, “A primary criticism leveled in the Pigott et al paper is that ‘the STAR*D investigators did not use the protocol-stipulated HRSD to report cumulative remission and response rates in their summary.’ As previously reported, what Pigott et al fail to appreciate is that the overall outcomes of patients across 1 year of treatment reported by Rush et al was not an a priori–identified analysis in the protocol, but a secondary post hoc report, specifically requested by the editor-in-chief of the American Journal of Psychiatry at that time, with the goal of summarizing the clinical outcomes—as measured by the QIDS-SR (capturing the symptom status of each patient at the last visit regardless of level and regardless of whether or not the HRSD was obtained at study exit)—of this complicated multilevel trial.”17, p.7
STAR*D PIs also assert that “it needs to be acknowledged that, in our 2006 paper, STAR*D investigators did not violate the study’s protocol or ‘NIMH-approved’ data analysis plan.”17, p.7
RIAT Investigators’ Reply:
While “the editor-in-chief made us do it” is a novel defense, the STAR*D PIs still fail to acknowledge that the NIMH-approved protocol specifically excluded all clinic-administered assessments such as the QIDS-SR to report research outcomes. The protocol states:
“Recall that the research outcomes assessments are distinguished from assessments conducted at clinic visits. The latter are designed to collect information that guides clinicians in the implementation of the treatment protocol. Research outcomes assessments are NOT collected at the clinic visits.” 19, p.47-48; emphasis in the original
The STAR*D PIs never disclosed their change to the “a priori–identified analysis in the protocol” when they used the clinic-administered QIDS-SR to report research outcomes in their steps 1-4 and summary articles. It is the lack of transparency in each of their 2006 AJP articles that makes this a protocol violation by the STAR*D PIs.
To address the issue of the many patients who exited the study without taking the HRSD, the STAR*D PIs should have reported the steps 1-4 and summary outcomes two ways as we publicly disclosed in our reanalysis. First, using the protocol-specified HRSD and then imputing the final QIDS-SR score to the HRSD for those patients missing an exit HRSD. This resulted in no missing outcome data.
Scientific Error #4
Without public disclosure, STAR*D PIs failed to acknowledge that all patients were started on citalopram in their baseline visit. The PIs then excluded from analyses the 234 patients who dropped out prior to their first post-baseline visit scheduled two weeks after having been prescribed citalopram. This is a violation of well-established intent-to-treat analysis procedures that are the gold-standard in clinical trials.
STAR*D PIs’ Response:
STAR*D PIs replied, “It is standard practice in clinical trials to exclude from the analyses patients who never come back for a visit after antidepressant treatment and, for that reason, we excluded from our analyses 234 patients who were prescribed citalopram and dropped out prior to their first postbaseline visit.”17, p.8
RIAT Investigators’ Reply:
We applaud that for the first-time STAR*D PIs have publicly acknowledged that they excluded from analyses the 234 citalopram-prescribed patients who did not return for subsequent visits. Consistent to form though, they failed to describe accurately the “intent-to-treat” analyses that are fundamental in reporting treatment trial outcomes. The PIs used intent-to-treat analyses in their steps 2-4 clinical trials, and should have done the same in step 1 as this is the widely accepted standard for reporting the outcomes from clinical trials.
Furthermore, the STAR*D PIs state in their step 1 article, “our primary analyses classified patients with missing exit HRSD scores as nonremitters a priori.”1, p. 34 These 234 early-dropout citalopram patients did not have an exit HRSD score and therefore should have been counted as nonremitters as prespecified.
Scientific Error #5
Whether by happenstance or design, STAR*D PIs sanitized treatment-emergent suicidality data by excluding from analysis the 234 early-dropout citalopram patients when they were at their greatest risk for suicide.20
In their step 1 article, the PIs state: “There were no suicides in the 2876 participants in this acute-phase citalopram study.”1, p.33 This declaration was based on the PIs excluding from analysis the 234 early-dropout citalopram patients when they were most at risk.
In 2009, the STAR*D PIs published a secondary analysis of emergent suicidality during citalopram treatment. They stated, “One of the participants who did not return after baseline was reported as a likely suicide secondary to ‘falling’ off his apartment complex roof 10 days after citalopram was prescribed.”21, p. 69
STAR*D PIs’ Response:
The STAR*D PIs state regarding the apparent suicide: “In fact, any adverse event that happens to patients who are given a prescription but never return for a first visit could be attributed to not getting any treatment rather than to the treatment itself, as patients may not return to their first visit because they never started the medicine or changed their minds about participating in the trial.”17, p.8
RIAT Investigators’ Reply:
In 2004, JAMA published an article on completed suicides and severe suicide ideation/attempts in 159,810 depressed patients prescribed an antidepressant. This analysis found that suicide completions were 38 times more likely to occur within the first 9 days after being prescribed an antidepressant compared with patients prescribed an antidepressant 90 days or more before their completed suicide.20
The JAMA 2004 article was highly-cited, and therefore should have been well-known in 2006 by the STAR*D PIs.
By not counting suicides during patients’ first two weeks of citalopram treatment, the STAR*D PIs sanitized treatment-emergent suicidality data and then highlighted the lack of suicides in those patients they chose to report on in their step 1 article published in AJP.
STAR*D Still Matters
Almost two decades later, STAR*D is still the go-to study on antidepressants’ “real-world” effectiveness and continues to be highly-cited in the psychiatric literature and mainstream media. For example, a recent New York Times article stated:
“The largest study of multiple antidepressants—nicknamed the STAR*D trial—found that half of the participants had improved after using either the first or second medication that they tried, and nearly 70 percent of people had become symptom-free by the fourth antidepressant.”22
This is simply not true as it is based on a study riddled with scientific errors. As our reanalysis documents, if the STAR*D PIs had adhered to the prespecified analytic criteria, they would have reported a cumulative remission rate of only 35%, which is a far cry from the 67% rate they persist in claiming.
Following publication of the March cover story, in April PT published Drs. Badre and Compton’s letter titled, “STAR*D: It is Time to Atone and Retract.”23 Their conclusion states: “Lastly, and most importantly, we have an ethical duty to our patients to take an honest look at the evidence derived from our research when making decisions that will impact their mental health. Our patients, our field, and our integrity demand a better explanation of what happened in STAR*D than what has thus been provided. Short of this, the best remaining course to take is a retraction.”
Dr. Kalin, the STAR*D PIs responses to our documentation of their scientific errors are marked by misdirection and disinformation gaslighting the scientific and lay public. As you know there is no statute of limitations for retracting peer-reviewed articles. Medical journal editors are duty-bound to retract articles when their “findings are no longer considered trustworthy due to scientific misconduct or error.”24
We hope that you have the courage to choose scientific integrity over guild interests and retract the five STAR*D articles AJP published in 2006.
Sincerely,
Ed Pigott, Ph.D.
Jay D. Amsterdam, M.D.
Cc:
Monica Bertagnolli, M.D, Director of the National Institute of Health
Shelli Avenevoli, Ph.D., Acting Director of the National Institute of Mental Health
Sheila Garrity, JD, MPH, Director of the Office of Research Integrity, U.S. Department of Health and Human Services
Retraction Watch
News Media
References
- Trivedi MH, Rush AJ, Wisniewski SR, Nierenberg AA, Warden D, Ritz L, Norquist G, Howland RH, Lebowitz B, McGarth PJ, Shores-Wilson K, Biggs MM, Balasubramani GK, Fava M. Evaluation of outcomes with citalopram for depression using measurement-based care in STAR*D: implications for clinical practice. Am J Psychiatry. 2006;163:28–40.
- Fava M, Rush AJ, Wisniewski SR, Nierenberg AA, Alpert JE, McGrath PJ, Thase ME, Warden D, Biggs MM, Luther JF, Niederehe G, Ritz L, Trivedi MH. A comparison of mirtazapine and nortriptyline following two consecutive failed medication treatments for depressed outpatients: a STAR*D report. Am J Psychiatry. 2006;163:1161–1172.
- Nierenberg AA, Fava M, Trivedi MH, Wisniewski SR, Thase ME, McGrath PJ, Alpert JE, Warden D, Luther JF, Niederehe G, Lebowitz BD, Shores-Wilson K, Rush AJ. A comparison of lithium and T3 augmentation following two failed medication treatments for depression: a STAR*D report. Am J Psychiatry. 2006;163:1519–1530.
- McGrath PJ, Stewart JW, Fava M, Trivedi MH, Wisniewski SR, Nierenberg AA, Thase ME, Davis L, Biggs MM, Shores-Wilson K, Luther JF, Niederehe G, Warden D, Rush AJ. Tranylcypromine versus venlafaxine plus mirtazapine following three failed antidepressant medications trials for depression: a STAR*D report. Am J Psychiatry. 2006;163:1531–1541.
- Rush AJ, Trivedi MH, Wisniewski SR, Nierenberg AA, Stewart JW, Warden D, Niederehe G, Thase ME, Lavori PW, Lebowitz BD, McGrath PJ, Rosenbaum JF, Sackheim HA, Kupfer DJ, Luther J, Fava M. Acute and longer-term outcomes in depressed outpatients requiring one or several treatment steps: a STAR*D report. Am J Psychiatry. 2006;163:1905–1917.
- Pigott HE, Kim T, Xu C, Kirsch I, Amsterdam, JD. What are the treatment remission, response and extent of improvement rates after up to four trials of antidepressant therapies in real-world depressed patients? A reanalysis of the STAR*D study’s patient-level data with fidelity to the original research protocol. BMJ Open 2023; 13:e063095
- Pigott HE, Dubin W, Kirsch I, Amsterdam JD. Call to action: RIAT reanalysis of the sequenced treatment alternatives to relieve depression (STAR*D) study. BMJ March 6, 2019. Available: https://www.bmj.com/content/ 346/bmj.f2865/rr-10 Pigott HE, Dubin W, Kirsch I, et al. . Call to action: RIAT Reanalysis of the sequenced treatment alternatives to relieve depression (STAR*D) study. BMJ March 6, 2019. Available: https://www.bmj.com/content/346/bmj.f2865/rr-10
- See: What are the treatment remission, response and extent of improvement rates after up to four trials of antidepressant therapies in real-world depressed patients? A reanalysis of the STAR*D study’s patient-level data with fidelity to the original research … | BMJ Open
- Rush AJ, Trivedi MH, Fava M, Thase ME, Wisniewski SR. The STAR*D data remain strong: Reply to Pigott et al. Am J Psychiatry 2023; 180:919–920; doi: 10.1176/appi.ajp.20230869
- Miller JJ. STAR*D dethroned? Psychiatric Times. 2023;40(12). STAR*D Dethroned? (psychiatrictimes.com)
- Pigott HE, Kim T, Xu C, Kirsch I, Amsterdam, JD. What was STAR*D’s cumulative remission rate and why does it still matter? Reply to John Miller. Psychiatric Times. 2024;41(3). STAR*D’s Cumulative Remission Rate and Why It Still Matters (psychiatrictimes.com)
- Boren J. Leventhal A, Pigott, HE. Just how effective are antidepressant medications? Results of a major new study. Journal of Contemporary Psychotherapy 2009;39(2): 93-100.
- Pigott HE, Leventhal AM, Alter GS, Boren JJ. Efficacy and effectiveness of antidepressants: current status of research. Psychother Psychosom. 2010;79:267–279.
- Pigott HE. STAR*D: a tale and trial of bias. Ethical Hum Psychol Psychiatry. 2011;13(1):6–28.
- Pigott, HE. The STAR*D trial: it’s time to reexamine the clinical beliefs which guide the treatment of major depression. Canadian J Psychiatry. 2015;60:9–13.
- Kirsch I, Huedo-Medina TB, Pigott HE, Johnson B. Do outcomes of clinical trials resemble those “real world” patients? A re-analysis of STAR*D antidepressant data. Psychology of Consciousness: Theory, Research, and Practice. 2018 https://doi.org/10.1037/cns0000164
- Rush AJ, Trivedi MH, Fava M, Thase ME, Wisniewski SR Rejecting the accusation of a violated STAR*D protocol. Psychiatric Times. 2024;41(3). Rejecting the Accusation of a Violated STAR*D Protocol (psychiatrictimes.com)
- Rush AJ, Fava M, Wisniewski SR, Lavori PW, Trivedi MH, Sackeim HA, Thase ME, Nierenberg AA, Quitkin FM, Kashner TM, Kupfer DJ, Rosenbaum JF, Alpert J, Stewart JW, McGrath PJ, Biggs MM, Shores-Wilson K, Lebowitz BD, Ritz L, Niederehe G. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004;25:119–142.
- National Institute of Mental Health (NIMH). Sequenced Treatment Alternatives to Relieve Depression (STAR*D) Research Protocol. Washington (DC): NIMH; revised June 28, 2002.
- Jick H, Kaye JA, Jick SS. Antidepressants and the risk of suicidal behaviors. JAMA 2004;292:338-343.
- Zisook S, Trivedi MH, Warden D, Lebowitz B, Thase M, Stewart JW, Moutier M, Fava M, Wisniewski SR, Luther J, Rush AJ. Clinical correlates of the worsening or emergence of suicidal ideation during SSRI treatment of depression: An examination of citalopram in the STAR*D study. J Affective Dis 2009;117:63–73.
- Caron C. What you really need to know about antidepressants. New York Times. April 25, 2024. https://www.nytimes.com/2024/04/25/well/mind/antidepressants-side-effects-anxiety-stress.html
- Badre N, Compton J. STAR*D: It is time to atone and retract. Psychiatric Times. 2024;41(4). STAR*D: It’s Time to Atone and Retract (psychiatrictimes.com)
- See: Retracted science and the retraction index – PubMed (nih.gov)
“In my clinical opinion, it is urgent for the field of psychiatry to reconcile the significant differences in remission rates for patients with MDD as published in the original STAR*D article in 2006 with the reanalysis just published in the BMJ article this year.”
All this fuss is based in a lack of scientific literacy. STAR*D had no control group. It could, by design, never answer the question whether SSRI are effective or save. This reflects very poorly on the standards in medical education concerning foundamentals of empirical research.
Report comment
“The public expects a medical speciality to be an honest purveyor of scientific findings about the benefits and harms of its interventions, and if the trial results tell of treatments that worsen the long term outcomes, then the medical speciality must inform the public about it and rethink its practices”.
Professor Peter C. Gotzshce: Is Psychiatry A Crime Against Humanity. 2024.
Surely the same expectation arises when meticulous scientific re-analysis of a major published trial/study finds evidence (and publishes evidence) of alleged academic errors?
Report comment
I would recommend that women get clinical and investigate why all men have become such dick heads. Get on your white coat and put on your white gloves. Open up our brains and our minds and see what you can make of it. Your tender intellectual care and caress is all we need. We created so many brain diseases in masculinity and society that we need femininity to help us in order for us all to become free. Because we are a whole human process, a whole humanity, and any form of separatism in interests, ideology or action will destroy the one total process of humanity. This isolation of groups is what’s destroying humanity and we all need to see inside each other’s closets in order to become free. I don’t care if you are male, female, other, or unsure. Let the whole world into your closet and be free. There is no need for privacy in love. And this is why love and truth are one. There was not yet the violence and ignorance that forced them apart.
Report comment
As of 2018, there is a higher percentage of women than men entering medical school. In my experience, men have no monopoly on intolerance and close-mindedness. Yes, it has become fashionable to blame men but the reality is that there are intolerant folks of both genders. We all want certainty in life and some will give up inquisitiveness to seek that security. We are all human and have issues that are uncomfortable. It takes a lifetime to resolve some of them.
Report comment
It is true that certainly men of today have no monopoly on intolerance and close mindedness, but if you look historically, the inconvenient truth is that this is because women have been heavily conditioned by a masculine society and seeking power, have tended to imitate male aggressive stupid and destructive egoism. The true woman has never been. And only half the man has ever been, and that was the shit half. She is trying to emerge, as are we all, but so far only scrambled wobbly puddings have been coming out of the Manfactory, very few of whom are sane, male or female. Anyway I’m a man with a female fire in my heart and I can assure you the masculine aspect of me is really like Manuel off of that sitcom Faulty Towers. A perceptive person compared me to him and it’s true. He’s the butler who can’t speak English and is always trying their best to please but mistranslating all the words (male aspect) and signals of the heart and body (feminine aspects) get’s everything wrong pleasing no-one. It is the male and female within us, the thought and feeling, the yin and yang, the good and evil all in one. We may or may not like it, but it’s our consciousness. It’s not set in stone – we can change it, I have changed it. I’m not Manuel anymore but Margarette Thatcher, i.e. a female Mussolini only in a man’s body. This is not actually true but it’s what the dark half of the feminine aspect in me is saying through the brain. She’s in league with the dark half of the brain (left side). The good side of the feminine aspect doesn’t speak endlessly and poke forks in to passers by though. But they are the same and love the full range of masculine, both light and dark, though they may shun one or the other for spiritual reasons. That’s why women often like to sleep with a demon and men often want to sleep with an angel. Or vice versa. The male aspect has the same duality to wrestle with and all the repressed and dark (e.g. violent, sexual, power hungry) rests in the dark aspect. If you understand it you go beyond it but harvest from it it’s powers. We all have both aspects in us and it’s a matter of which one predominates. The masculine predominates in me when I’m fighting or igniting or writing, and the feminine when I’m meditating or naturing or going off to otherdimensionaling.
Report comment