A recent article published in the journal Plos One has raised concerns about the representativeness of participants in psychological studies. The research discovered that individuals who volunteer for such studies tend to exhibit more symptoms associated with personality disorders than those who do not participate. This finding calls into question the generalizability of results from these studies to the broader population.
The authors contend that this self-selection of participants with a higher prevalence of personality disorder symptoms may skew research data, leading to conclusions that might not accurately reflect the general public.
Izabela KaĆșmierczak and Anna Zajenkowska from Maria Grzegorzewska University in Poland led the research. They assert in their report:
“Participants who alone applied to take part in paid psychological studies had more symptoms of personality disorders than those who had never before applied to take part in psychological studies. The current results strongly translate into a recommendation for either the modification of recruitment strategies or much greater caution when generalizing results for this methodological reason.”
In light of these findings, the researchers urge reconsidering current recruitment practices to ensure a more representative sample of the population in psychological studies. This could involve reevaluating incentives, broadening recruitment channels, or implementing other strategies to attract a more diverse pool of participants.
The implications of this research extend beyond the academic sphere, as psychological study findings often influence public policy, clinical practice, and our understanding of human behavior. Greater caution in generalizing results and increased efforts to improve the representativeness of study participants are essential steps toward producing more accurate and reliable findings that can benefit the entire population.
The current work has four goals. First, to assess whether being invited to talk about regular (RLE) or critical life (CLE) events attracts people with more psychological symptoms. Second, to determine whether face-to-face or online modalities made a difference in terms of participants’ symptoms. Third, to compare groups that volunteer for psychological studies to groups that do not volunteer in terms of psychological symptoms. Fourth, determine if people who volunteer for psychological studies have higher depressive and anxiety symptoms than average.
To accomplish these goals, the authors examined three subsets of participants. The first subset was invited to a face-to-face study using three different invitations as an experimental manipulation. The first invitation was to discuss a CLE (such as the death of a family member) that led to a low mood. The second invitation was to discuss a CLE that did not lead to a depressive episode. The third invitation was to discuss the psychological impact of an RLE. One hundred twenty-seven people aged 21 – 41 volunteered for this portion of the study. About half (50.7%) were college graduates.
The second subset of participants was composed of people who had never previously volunteered to participate in a psychological study. Researchers in their local communities invited these participants to a face-to-face study. This subset consisted of 100 participants between the ages of 20 – 48. Most (66%) had either a bachelor’s or master’s degree.
The third subset of participants was invited to participate in an online survey to earn points that could be redeemed for prizes. Seven hundred twenty participants aged 25 – 45 volunteered, of which 56.7 had earned university degrees. All the participants in all the subsets were from Poland and self-identified as white.
The authors used a structured clinical interview and the borderline personality inventory to assess differences in personality disorder symptoms among their participants. In addition, they used the hospital depression and anxiety scale to examine the levels of depression and anxiety among their participants.
The highest number of symptoms and personality disorder diagnoses came from the CLE with the low mood group. This group had more symptoms of avoidant, dependent, passive-aggressive, schizoid, and narcissistic personality disorders than all other groups. This group also had increased symptoms of schizotypal, paranoid, antisocial, and obsessive-compulsive personality disorders compared to the non-volunteer group.
Participants from the CLE without low mood group exhibited more symptoms of schizoid personality disorder than the RLE and non-volunteer groups. The non-volunteer group showed fewer symptoms of personality disorders than any other group. Non-volunteers also scored lower on measures of anxiety and depression.
The authors acknowledge several limitations to the current work. First, all participants were from Poland and identified as white. This severely limits generalizability to other populations. Second, the present work did not use deception, which resulted in their participants knowing about the methods to be used. Participants without such knowledge may create more reliable data. Third, the results of the current work depend on the measures the authors chose. Different measures could bring different results. They conclude:
“Researchers often take for granted that the way they advertise their studies and who they recruit do not appreciably affect their outcomes. In our studies, we have shown that those who have more personality pathologies are more drawn to studies where they can express their trauma and may be simply more likely to volunteer for studies. While we cannot dismiss the fact that all our samples were of Poles, if we assume that they are like others around the worldâan assumption we see no reason to doubt, which is, after all, the null hypothesisâ then our results have meaningful implications for how researchers interpret their results and how clinicians estimate the prevalence of various disorders. In short, our field may be conducting research on an atypically disordered and motivated group of people leading to biased views of the reality of psychological effects.”
The psy-disciplines are no stranger to biased research practices. Research has shown that implicit bias can impact clinical care. Similarly, implicit racial bias can change both diagnosis and treatment. Misreporting research results and publication bias are also reasonably common. Industry sponsorship routinely biases research in favor of industry products.
Self-selection bias is problematic for research as only a certain type of person will often volunteer. This can result in findings that cannot be generalized. For example, previous research has highlighted problems with self-selection bias in research around stress. Similar research found self-selection bias in people willing to participate in research around eating disorders.
****
KaĆșmierczak I, Zajenkowska A, Rogoza R, Jonason PK, ĆcigaĆa D (2023) Self-selection biases in psychological studies: Personality and affective disorders are prevalent among participants. PLoS ONE 18(3): e0281046. https://doi.org/10.1371/ journal.pone.0281046 (Link)
Anyone who thinks that psychological/psychiatric research isnât implicitly biased needs their own head examined.
Report comment