I just stumbled across an interesting study in The Clinical Neuropsychologist that investigated the concurrent validity of the Automated Neuropsychological Assessment Metrics (ANAM) and the Woodcock-Johnson III Tests of Cognitive Ability (WJ III) [Conflict of interest disclosure - I'm a co-author of the WJ III). Below is the abstract. Following the abstract are a few select findings and comments from the blogmaster. As appropriately noted by the authors (Jones et al., 2008), the results should be viewed with caution and need replication in larger and more diverse samples. The current sample was a relatively small (n=77) and homogenous/select sample of young adults (colleage undergraduate students).
Abstract
Although the focus of the study was validating the ANAM, given the blogmaster's interests (and most readers of this blog), I'm more interested in the validity evidence from the perspective of the WJ III measures.
First, what is the ANAM? As described by the authors, the ANAM " is a clinical subset of instruments originally developed by the Office of Military Performance Assessment Technology. The ANAM battery is comprised of a series of cognitive tasks administered and scored with a computer...the common feature in the ANAM scales is highly structured, repetitive information processing tasks with well-defined stimuli and simple response modes."
The most intriguing results, from the perspective of the WJ III, are the significant and consistent moderate to moderately high correlations between the ANAM throughput scores and the WJ-III Cognitive Efficiency (Gsm+Gs) cluster (comprised of the Visual Matching and Numbers Reversed tests). Six of these seven correlation coefficients were statistically significant (ranging from approximately .30 to .52). The authors suggest that the respective ANAM measures "appear to be tapping an underlying construct of effective initial processing of input." This finding is also consistent with the WJ III authors suggested interpretation of the Cognitive Efficiency cluster (click here for a prior post that will eventually take you, if you follow some additional links, to a more detailed discussion of cognitive efficiency and some WJ III/CHC organized studies that investigated the relations between cognitive efficiency and psychometric g)
The next intriguing finding was a .58 (when corrected for range restriction) correlation between the WJ III Fluid Reasoning (Gf) and the NAM Matching to Sample measure. As described on the official ANAM web page, the MSP test "assesses spatial processing and visuo-spatial working memory. The user views a pattern produced by eight shaded cells in a 4x4 sample grid. The sample is then removed and two comparison patterns are displayed side by side. One grid is identical to the sample grid and the other grid differs by one shaded cell. The user is instructed to press a designated button to select the grid that matches the sample." Thus, the .58 correlation, which suggests approximately 34% shared variance, suggests that performance on the WJ III Fluid Reasoning (Gf) cluster may require a significant amount of visual-spatial working memory. This would be consistent with a large body of contemporary research that suggests a strong relation between working memory and higher-order cognitive functions (often Gf and or g, general intelligence).
The third major finding of interest were the range-restricted corrected correlations of .56 and .77 bewteen the WJ III GIA (g) cluster and the ANAM Matching to Sample (MSP)and Logical Reasoning (LR) scales. According to the offical ANAM web page, the LR test "assesses abstract reasoning and verbal syntax ability by asking the user to evaluate the truth of a statement (e.g., "& comes after #") describing the order of two symbols displayed on the display (e.g., "& #"). The user presses designated buttons to indicate whether the statement is true or false."
The article also presents correlations between select individual WJ III tests and ANAM measures. I was disappointed that the authors only presented select WJ III test with ANAM measure correlations (also only select WJ III cluster with ANAM measure correlations). Additional analysis focused on the prediction of WJ III variables from the ANAM variables using multiple regression. You can read this on your own, particularly if you are interested in the ANAM.
Supplementary exploratory factor analysis (EFA) of the ANAM measures were also included. I was disappointed that the authors reported only one EFA with any WJ III measures, and then, with only the Fluid Reasoning cluster. It would have been more informative to see a joint cross-battery factor analysis that included all of the WJ III individual tests that were administered. Combined with the small and select sample size, I don't see much value in discussing the limited ANAM/WJ III EFA analysis.
Technorati Tags: pscyhology, educational psychology, school psychology, neuropsychology, neuroscience, ANAM, WJ III, Woodcock-Johnson, IQ, intelligence, g, cognitive efficiency, CHC theory, Gs, Gf, Gsm, working memory
Abstract
- This study examines the relationship between a computerized neuropsychological assessment battery, the Automated Neuropsychological Assessment Metrics (ANAM) and a widely used ability measure, Woodcock-Johnson III Tests of Cognitive Ability (WJ-III). Results indicated substantial relationship between the ANAM throughput (accuracy /response time) scores and the WJ-III Cognitive Efficiency cluster. An unexpectedly strong relationship was evident between accuracy scores on the ANAM Logical Reasoning scale and the WJ-III General Intellectual Ability score, purporting to measure the g factor. The findings support the viability of the ANAM as a time- and cost-effective tool for appraisal of cognitive function.
Although the focus of the study was validating the ANAM, given the blogmaster's interests (and most readers of this blog), I'm more interested in the validity evidence from the perspective of the WJ III measures.
First, what is the ANAM? As described by the authors, the ANAM " is a clinical subset of instruments originally developed by the Office of Military Performance Assessment Technology. The ANAM battery is comprised of a series of cognitive tasks administered and scored with a computer...the common feature in the ANAM scales is highly structured, repetitive information processing tasks with well-defined stimuli and simple response modes."
The most intriguing results, from the perspective of the WJ III, are the significant and consistent moderate to moderately high correlations between the ANAM throughput scores and the WJ-III Cognitive Efficiency (Gsm+Gs) cluster (comprised of the Visual Matching and Numbers Reversed tests). Six of these seven correlation coefficients were statistically significant (ranging from approximately .30 to .52). The authors suggest that the respective ANAM measures "appear to be tapping an underlying construct of effective initial processing of input." This finding is also consistent with the WJ III authors suggested interpretation of the Cognitive Efficiency cluster (click here for a prior post that will eventually take you, if you follow some additional links, to a more detailed discussion of cognitive efficiency and some WJ III/CHC organized studies that investigated the relations between cognitive efficiency and psychometric g)
The next intriguing finding was a .58 (when corrected for range restriction) correlation between the WJ III Fluid Reasoning (Gf) and the NAM Matching to Sample measure. As described on the official ANAM web page, the MSP test "assesses spatial processing and visuo-spatial working memory. The user views a pattern produced by eight shaded cells in a 4x4 sample grid. The sample is then removed and two comparison patterns are displayed side by side. One grid is identical to the sample grid and the other grid differs by one shaded cell. The user is instructed to press a designated button to select the grid that matches the sample." Thus, the .58 correlation, which suggests approximately 34% shared variance, suggests that performance on the WJ III Fluid Reasoning (Gf) cluster may require a significant amount of visual-spatial working memory. This would be consistent with a large body of contemporary research that suggests a strong relation between working memory and higher-order cognitive functions (often Gf and or g, general intelligence).
The third major finding of interest were the range-restricted corrected correlations of .56 and .77 bewteen the WJ III GIA (g) cluster and the ANAM Matching to Sample (MSP)and Logical Reasoning (LR) scales. According to the offical ANAM web page, the LR test "assesses abstract reasoning and verbal syntax ability by asking the user to evaluate the truth of a statement (e.g., "& comes after #") describing the order of two symbols displayed on the display (e.g., "& #"). The user presses designated buttons to indicate whether the statement is true or false."
The article also presents correlations between select individual WJ III tests and ANAM measures. I was disappointed that the authors only presented select WJ III test with ANAM measure correlations (also only select WJ III cluster with ANAM measure correlations). Additional analysis focused on the prediction of WJ III variables from the ANAM variables using multiple regression. You can read this on your own, particularly if you are interested in the ANAM.
Supplementary exploratory factor analysis (EFA) of the ANAM measures were also included. I was disappointed that the authors reported only one EFA with any WJ III measures, and then, with only the Fluid Reasoning cluster. It would have been more informative to see a joint cross-battery factor analysis that included all of the WJ III individual tests that were administered. Combined with the small and select sample size, I don't see much value in discussing the limited ANAM/WJ III EFA analysis.
Technorati Tags: pscyhology, educational psychology, school psychology, neuropsychology, neuroscience, ANAM, WJ III, Woodcock-Johnson, IQ, intelligence, g, cognitive efficiency, CHC theory, Gs, Gf, Gsm, working memory
