David Lohman presented at the 2008 ISIR conference. I've been a big fan of Lohman's work as much of it has direct application to the work of practicing school and educational psychologists. Lohman was a student of the late Richard Snow, whose work has had a significant work on my thoughts regarding non-cognitive factors important for school learning (see prior post today). Lohman is an author of the group CogAt (click here to see prior post re: study with WJ III). Aside from being an excellent applied psychometrician, Lohman has written papers on a wide variety of topics in educational psychology and intelligence. He is also very generous in making his various publications available for download at his web page.At this conference he presented a paper comparing scores and norm characteristics from the CogAT, NNAT, and Ravens. The name of the paper and abstract (italics added by me) is below. The focus was on the use of nonverbal measures of intelligence in the identifcation of gifted students. The results presented were a bit disconcerting regarding possible technical issues with the norms of two of the tests---I've featured Lowman's conclusion in the abstract below. Lohman's research raises significant issues re: the accuracy of gifted identification via the NNAT and Ravens. Of course, and appropriately so, Lohman made it clear that his findings and research needed to recognize his potential conflict of interest as author of the CogAT, a direct competitor to the other tests, esp. the NNAT. It is refreshing to see such scholarly integrity in person.
- Ethnic Differences on Fluid Reasoning Tests: Is the NNAT the Panacea? David F. Lohman, University of Iowa
- Abstract: Nonverbal, figural reasoning tests such as the Raven Progressive Matrices are often used as markers for Gf in research on intelligence. These tests are also widely used in schools to help identify academically gifted students. The Naglieri Nonverbal Ability Test (NNAT) has been particularly popular following a recent report that in a large, representative sample of U.S. school children, it was found to identify equal proportions of high-scoring White, Black, and Hispanic students. Although questions have been raised about the integrity of the data analyses used in this study, the author of the study continues to defend it as the most important research yet conducted with the NNAT. The goal of this investigation was to compare the NNAT with two other nonverbal assessments: the Raven Progressive Matrices and the Nonverbal Battery of the Cognitive Abilities Test (CogAT). All three tests were administered by trained examiners in counterbalanced order to 1,200 children in grades K to 6 in an ethnically diverse school district. Results showed provided no support for the assertion that the NNAT reduced ethnic differences – either at the mean or at the tails of the distribution. Rather, ethnic differences were actually somewhat larger on the NNAT than on the other two tests. Furthermore, it was discovered that the variance of basic normative score on the NNAT (M = 100, SD = 15) substantially exceeded the reported value of 15 at all but one test level. Re-analyses of the standardization data for the NNAT confirmed this finding. A similar normative score on the Raven (computed from the most recent U.S. national norms) was 10 points too lenient. Consequences of invalid or outdated normative scores for research and practice are discussed.
The bottom line take-away: Buyer beware. Educators need to do their due diligence when evaluating and comparing psychometric instruments (group or individual) that impact important educational decisions regarding children.
[Conflict of interest disclosure - I'm a coauthor of the WJ III mentioned above, which is an instrument that competes with a different individually administered cognitive battery of the author of the NNAT, which was a focus of Lohman's presentation and paper].
