Garruto responds to "factor analysis of IQ tests"

John Garruto, a member of IQ's Corner Virtual Community of Scholars group was weighed in on the Frazier & Youngstrom 2007 Intelligence article listed below (click here for prior posts re: this article). John's comments speak for themselves. Thanks John

Frazier, T.W. & Youngstrom, E.A. (2007). Historical increase in the number of factors measured by commercial tests of cognitive ability: Are we overfactoring? Intelligence, 35, 169-182. [article can be viewed at link at "prior post" link above]


Abstract

• A historical increase in the number of factors purportedly measured by commercial tests of cognitive ability may result from four distinct pressures including: increasingly complex models of intelligence, test publishers' desires to provide clinically useful assessment instruments with greater interpretive value, test publishers' desires to include minor factors that may be of interest to researchers (but are not clinically useful), and liberal statistical criteria for determining the factor structure of tests. The present study examined the number of factors measured by several historically relevant and currently employed commercial tests of cognitive abilities using statistical criteria derived from principal components analyses, and exploratory and confirmatory factor analyses. Two infrequently used statistical criteria,that have been shown to accurately recover the number of factors in a data set, Horn's parallel analysis (HPA) and Minimum Average Partial (MAP) analysis, served as gold-standard criteria. As expected, there were significant increases over time in the number of factors purportedly measured by cognitive ability tests (r=.56, p=.030). Results also indicated significant recent increases in the overfactoring of cognitive ability tests. Developers of future cognitive assessment batteries may wish to increase the lengths of the batteries in order to more adequately measure additional factors. Alternatively, clinicians interested in briefer assessment strategies may benefit from short batteries that reliably assess general intellectual ability.

Garruto comments

As I was catching up on my NASP listserv digest, I saw a responsible practitioner query the group about how she could brush up on interpretation with her new WJ-III kit. I noted a variety of excellent responses. One really caught my eye where the poster indicated, “Read this article.” I began to think to myself, “I wonder what this article could add incrementally to this examiner’s skills?”

So I took the time to read the article and I must say that I found this article quite helpful. It will serve me well when I need scratch paper to write a phone number or perhaps as reading material when I am suffering from insomnia. On a more serious note, however, I did not find a great deal of utility for the applied practitioner. Furthermore, the article did not in the slightest way abet the poster with her query. I can only surmise the poster had made his recommendation to voice a possible disdain toward either the WJ-III or batteries that present with multiple factors.

Nevertheless, I will do my best to offer a summary and my perspective. However, individuals with a greater degree of statistical acumen will probably be able to comment more on its empirical support. The authors reviewed a variety of cognitive ability instruments and have noted that the number of factors measured by an individual ability test has outpaced the number of subtests/length of administration of these batteries. The authors identify the various factor analysis methods that have been used, citing certain methods have enjoyed popular use in recent history (including Cattell’s scree method, Kaiser’s criterion, and chi square method.) Furthermore, the authors cite two little used methods for identifying factors which included the minimum average partial (MAP) analysis and Horn’s Parallel Analysis (HPA)).

The authors did do a nice job of differentiating exploratory factor analysis (principal components analysis) and confirmatory factor analysis. For the lay reader, factors tend to be latent variables believed to be measured by observed variables (for example-the observed subtest of block design is believed to measure a latent variable of visualization). Exploratory factor analysis is a technique where certain skills tend to cluster or “hang” together and the commonality of the tasks imposed by the observed measures would lead to a factor naming. Confirmatory factor analysis, on the other hand, specifies the model beforehand and confirmation can be done via a goodness of fit (how well do our observed variables fit our specified model?)

I am not going to go into a lengthy discourse on statistics. In short, the authors indicate that test developers have been using factor analytic methods that have led to an over-identification of factors. The authors espouse using other techniques (such as MAP and HPA) in order to identify the number of factors likely at play for a given instrument. The findings of the authors is that using MAP or HPA, the number of factors should never exceed three (and that was just the HPA analysis of the WJ-III-the most complex instrument out there-all others recommended a 2 or 1 factor solution.) The conclusion of the authors is that we are over-factoring our tests and therefore we either need to make the tests longer (which the authors counter with a cost-prohibitive notion) or start reducing our factors.

I am going to leave it to the test developers and statisticians to comment on the merit of the statistics. Instead, I am going to provide the perspective of an applied practitioner. I have been using the WJ-III since its publication. Although I have had some concerns with this test-it always remains a staple of my assessment battery because of the breadth of the interpretable results. The WJ-III does have a three-factor solution, but it is then parsed further into a seven-factor solution if you give the extended battery. Namely, that (the three factor solution) is verbal ability, thinking ability, and cognitive efficiency. As a practitioner, these constructs help somewhat-but what does that really gain for me in understanding the various skills of a child? Sure I can look at verbal ability as just that, thinking ability as intentional cognitive processing, and cognitive efficiency as how effectively we manage our information, but what does that say about a child in terms of their need? Even more important, when a child presents with very solid short-term memory but very poor processing speed (yielding an average cognitive efficiency score), how does that factor help me? Even more important, how does this limited framework help the student?

The ongoing issues with subtest/profile analysis are well understood, particularly with regard to reliability issues. However, to intentionally blunt our knives seems counterproductive-particularly since the research has already panned out many of these factors. Furthermore, some subtests that are not figured into either factor because of mixed loadings can be very valuable. One subtest I am thinking of is Rapid Picture Naming. In my experience, I cannot tell you the amount of times I have looked at a child with a reading problem and that child completely bombed RPN. However, it’s a very mixed measure-there is still ongoing debate as to whether it’s a long-term storage and retrieval task (due to naming ability as a narrow ability) or processing speed task (given the speed at which one must name the tasks.)

On a final note…let’s look back to another test-the WISC-III. The WISC-III had both a two and a four-factor solution-yes it had the verbal and performance IQ. I also remember reading (I think) that Picture Arrangement subtest that seemed to load on both verbal and performance factors (just a bit stronger for performance). Frazier & Youngstrom discuss how many of our tests should have more than two subtests to measure a factor (p. 179). They also spend a good part of their article talking about the costs of lengthier assessments (p. 180). I can think of no greater waste of time and money than to measure a construct four times (ala crystallized intelligence on the WISC-III) so I can be really sure my factor is strong. Nevertheless-I will keep this article at the ready…I have three phone calls to return tomorrow.

Technorati Tags: psychology, educational psychology, school psychology, neuropsychology, psychometrics, factor analysis, intelligence, IQ, IQ tests, WJ III, Woodcock Johnson

Powered by ScribeFire.