A study described in a JAMA Internal Medicine letter showed that mentions of the limitations of observational studies are often buried deep in the discussion sections of papers, and then the frequency of mentions of the limitations drops steadily thereafter in abstracts, press releases and news stories. The study won praise from HealthNewsReview.org.
Observational studies generally look to discover relationships between different types of events that may simply be associated (or correlated) with each other due to coincidence or some other common factor, or that may actually be influencing (or causing) one another in some way. Common examples at either end of that spectrum are observational studies that have found that ice cream sales and murders both increase in summer months, and that the risks of developing lung cancer increase alongside smoking. Observational studies are common in psychiatry and psychology, often detecting, for example, apparent associations between certain events or lifestyle behaviors and certain types of emotional experiences.
HealthNewsReview.org summarized the outlines of the study: “The authors, from the University of Auckland in New Zealand, analyzed a combined 538 documents including major medical journal articles, accompanying editorials in those journals, news releases by those journals, and news stories written about all of the preceding.” They then quoted the authors of the study: “Observational research is abundant and influences clinical practice, in part via publication in high-impact journals and dissemination by news media. However, it frequently generates unreliable findings. Inherent methodologic limitations that generate bias and confounding mean that causal inferences cannot reliably be drawn. Study limitations may be inadequately acknowledged and accompanied by disclaimers that diminish their importance.”
HealthNewReview.org provided an analysis of the study, but their reproduction of the study’s chart told the main story:
Wang MM, Bolland MJ, Grey A. Reporting of Limitations of Observational Research. JAMA Intern Med. Published online June 08, 2015. doi:10.1001/jamainternmed.2015.2147. (Excerpt)
Weak reporting of limitations of observational research (HealthNewsReview.org, June 8, 2015)