A team of researchers in school psychology made up of Ryan J. McGill of William and Mary University, Stefan Dombrowski of Rider University, and Gary Canivez of Eastern Illinois University conducted a review of the history and empirical examination of cognitive assessment in schools. Cognitive assessment by school psychologists has long been fraught with controversy, but past reviews highlighting concerns surrounding theoretical and methodological concerns are no longer current. McGill, Dombrowski, and Canivez’s review, published in the Journal of School Psychology, sheds light on some of the same concerns highlighted over two decades ago, indicating that “reliability, validity, diagnostic utility, and treatment utility [cognitive profile analysis procedures] remains less than compelling.”
“Researchers have long debated how cognitive measures should be interpreted in clinical practice with some questioning whether they should be used at all,” they write. “Further complicating the matter are the numerous interpretive systems, heuristics, and complex software programs […], that are available for practitioners to use; many of which encourage users to engage in some variant of cognitive profile analysis (i.e., making inferences about strengths and weaknesses observed in an individual’s profile of scores).”
Cognitive assessments are often conducted by school psychologists in the context of psychoeducational evaluations and are intended to inform classification and treatment trajectories for students. Psychoeducational evaluations may occur when new academic, social, emotional, or behavior concerns occur and are completed on a triennial basis for students deemed eligible for services outside of the general education curriculum. Although cognitive profiles reflected in general, composite, and subtest scores reflect features of a student’s thought process, the use of these data in practical decision making for school-based supports for students has been heavily debated.
In some US states, cognitive assessments have been formally eliminated from the test battery required to make special education and counseling determinations. Patterns of disproportionality in disability determinations reflected an overidentification of students of color for special education services, cultural biases in popular cognitive assessments, and mixed evidence for the reliability and validity of popular assessment procedures have prompted alternative approaches in some districts, but not all.
According to one survey referenced by McGill, Dombrowski, and Canivez, 49% of school psychologists surveyed indicated using composite-level analysis (examining dimensions of intelligence indicated through cognitive assessment) in clinical decision making. Despite a substantive review by Marley Watkins in 2000 that drew attention to the limitations of cognitive assessment in schools, cognitive assessment maintains a major role in the training and professional work of school psychologists.
Determinations made by school psychologists predominately inform in-school supports and accommodations. However, labels and recommendations issued by school psychologists may have a lasting impact that transcends school. Thus, precision, care, and innovation in assessment procedures are imperative.
The authors begin their review by outlining the critical works put forth in the nineties that revealed questionable psychometric properties of intelligence tests and emphasized concerns surrounding the application of cognitive assessment in schools. Findings from these articles were also synthesized in Watkins’ seminal review. A combination of new and longstanding assessment procedures (e.g., cross-battery assessment, ipsative assessment, levels-of-analysis approach, and X-BASS) applied within the past two decades paved the way for the current, updated examination.
Although strengths are touted in the manuals that accompany popular cognitive assessments applied in schools, McGill, Dombrowski, and Canivez draw attention to the poor reliability of within-individual subtest and composite performance over time demonstrated by many intelligence tests. One student may receive vastly different results in one cognitive domain at one timepoint than they would in a different room, with a different examiner, three years later upon reevaluation.
Alan Kaufman (known his work in cognitive assessment) and other big-names have defended the utility of cognitive assessments in schools, arguing that intelligence tests should never be administered or interpreted in the absence of other evaluation components and that they are an important piece of a larger puzzle. However, empirical evidence for this oft-repeated argument is limited. The question remains; are cognitive assessment results necessary in the process of establishing a comprehensive snapshot of a student’s capabilities and needs in a psychoeducational evaluation? Authors of the current review indicate that perhaps they are not.
“Put simply, most people with cognitive weaknesses are able to get through school just fine and most people with academic difficulties do not have a learning disorder,” they write. “Although it is frequently claimed in the professional and commercial literature that use of profile analytic methods such as PSW may be useful for diagnosis and treatment planning for individuals with academic weaknesses, a countering body of literature has emerged over the last five years documenting a host of psychometric and conceptual concerns about these methods.”
McGill, Dombrowski, and Canivez’s results echo those achieved approximately twenty years ago, yet cognitive assessment continues to play a major role in special education and counseling determinations made in schools. There has been a recent push to ensure that research and training better align in the field of school psychology, and with regard to cognitive assessment, this could mean a reduction in the administration of cognitive assessment in schools.
Mcgill, R. J., Dombrowski, S. C., & Canivez, G. L. (2018). Cognitive profile analysis in school psychology: History, issues, and continued concerns. Journal of School Psychology, 71, 108-121. (Link)