AP101 Brief # 13: CHC-consistent scholastic aptitude clusters: Back to the Future

This is a continuation of a set of analyses previously posted under the title  Visual-graphic tools for implementing intelligent intelligence testing in SLD contexts:  Formative concepts and tools.  It is recommended that you read the prior post to obtain the necessary background and context, which will not be repeated here.

The third  method approach to SLD identification (POSW; pattern of strengths and weaknesses) has been advanced primarily by Flanagan and colleagues, as well as Hale and colleagues and Naglieri (see Flanagan & Fiorrello, 2010 for an overview and discussion).  A central concept to these POSW third method SLD models is that an individual with a possible SLD must show cognitive deficits that have been empirically or theoretically demonstrated to be the most relevant cognitive abilities for the achievement domain where the person is deficient.  That is, the individual's cognitive deficits are consistent or concordant with the persons academic deficits, in the context of other cognitive/achievement strengths that suggest strengths in non-SLD areas.  I have often referred to this as a domain-specific constellation or complex of abilities and achievements.

Inherent in these models is the operationalization of the notion of  aptitude-achievement consistency or concordance.  It is important to note that aptitude is not the same as general intelligence or IQ.  Aptitude in this contexts draws on the historical/traditional notion of aptitude that has been around for decades.  Richard Snow and colleagues have (IMHO) written the best information regarding this particular definition of aptitude.  Aptitude includes both cognitive and conative characteristics of a person (see Beyond IQ Project).  But for this specific post, I am only focusing on the cognitive portion of aptitude--which would, in simple terms, represent the best combination of particular CHC narrow or broad cognitive abilities that are most highly correlated with success within a particular narrow or broad achievement domain.

What are the CHC narrow or broad abilities most relevant to different achievement domains?  This information has been provided in a narrative research synthesis form by Flanagan and colleagues (in their various cross-battery books and chapters) and more recently in a structured empirical research synthesis by McGrew and Wendling (2010).  These CHC-based COG--ACH relations summaries provide assessment professionals with information on the specific broad or narrow CHC abilities most associated with sudomains in reading and math, and to a lessor extent writing.  Additionally, the McGrew and Wendling's (2010) synthesis provides information on developmental considerations--that is, the relative importance of CHC abilities for different achievement domains varies as a function of age.  McGrew and Wendling (2010) presented their results for three broad age groups (6-8; 9-13; 14-18 years of age).

Given this context, I presented a series of analysis (see the first post mentioned above as recommended background reading) that took the findings of the McGrew and Wendling (2010) as an initial starting point and used logical, empirical, and theoretical considerations to identify the best set of WJ III cognitive test predictors in the same three age groups for two illustrative achievement domains.  I have since winnowed down the best set of cognitive predictors in the two achievement domains (basic reading skills-BRS; math reasoning-MR).  I then took each set of carefully selected predictor tests and ran multiple regression models for each year of age from ages 6 thru 18 in the WJ III NU norm data.  I saved the standardized regression coefficients for each predictor and  plotted them by age. The plotted raw standardized coefficients demonstrated clear systematic developmental trends, but with noticeable "bounce" due to sampling error.  I thus generated smoothed curves using a non-linear smoothing function...with the smoothed curve representing the best estimate of the population parameters.  This technique has been used previously in a variety of studies that explored the relations between WJ-R/WJ III clusters and achievement (see McGrew, 1993 and McGrew and Wrightston, 1997 for examples and description of methodology).  Below is a plot of the raw standardized coefficients and the smoothed curve two of the significant predictors (Verbal Comprehension; Visual-Auditory Learning) for the prediction of the WJ III Basic Reading Skills cluster. [click on images to enlarge]. It is clear that the relative importance of Verbal Comprehension and Visual-Auditory Learning increase/decrease (respectively) systematically with age.

The next two figures present the final smoothed results for the CHC-based aptitude clusters for the prediction of the WJ III Basic Reading Skills and Math Reasoning clusters.

There is much that could be discussed after looking at the two figures.  Below are a few comments and thoughts.

• The composition of what I am calling CHC-consistent scholastic aptitude clusters make theoretical and empirical (CHC-->ACH research synthesis) sense. For example, in both BRS and MR, Gc-LD/VL abilitiy (Verbal Comprehension) is salient at all ages and systematically increases in importance with age.  In BRS, visual-auditory associative memory (Glr-MA; Vis-Aud. Learning) is very important during the early school years (ages 6-9), but then disappears from being important in the prediction model.  This ability (test) is not found in the MR model.  Gf abilities (quantitative reasoning-RQ, Number Matrices; general sequential reasoning-RG, Analysis-Synthesis) are important throughout all ages for predicting math reasoning achievement.  In fact, both increase in relative importance with age, particularly for the measure of Gf-RQ (Number Matrices).  These two Gf tests are no where to be found in the BRS plot.  Instead, measures of  Ga abilities (Sound Blending; Sound Awareness) are important in the BRS model.  Gs and Gsm-WM (domain-general cognitive efficiency variables) are present in both the BRS and MR models.

• The amount of explained variance (multiple R squared; Tables in figures) is higher for the CHC-consistent scholastic aptitude clusters when compared to the WJ III General Intellectual Ability (GIA-Std) clusters.  This is particularly true at the oldest ages for MR.  Of course, these values capitalize on chance factors due to the nature of multiple regression and would likely shrink somewhat in independent sample cross-validation (yes...I could have split the sample in half to develop and then cross-validate the models..but I didn't). 

• These age-by-age plots provide a much more precise picture of the developmental nature of the relations between narrow CHC abilities and achievement than the McGrew & Wendling (2010) and Flanagan and colleagues reviews.  These findings suggest that when selecting tests for referral-focused selective assessment (see McGrew & Wendling, 2010) it is critical that examiners know the developmental nature of CHC--ACH relations research.  The fact that some specific narrow CHC tests show such dramatic changes across the ages suggests that those who implement a CHC-based aptitude-achievement consistency SLD model must be cautious and not use a "one size fits all" approach when determining which CHC abilities should be examined for the aptitude portion of the consistency model.  An ability that may be very important certain age levels may not be important at other age levels (e.g., Vis-Aud. Learning in the WJ III BRS aptitude cluster).  

• The above results further reinforce the conclusion of McGrew & Wendling (2010) that development of more "intelligent" referral-focused selective assessment strategies requires a recognition that this process requires an understanding of the 3-way interaction of CHC abilities X Ach domains X Age (developmental status)

These results suggest that the field of intellectual assessment, particularly in the context of educational-related assessments, should go "Back to the Future."  The 1977 WJ and 1989 WJ-R batteries both included scholastic aptitude clusters (SAPTs; click here to read relevant select text from McGrew's two WJTCA books) as part of the WJ/WJ-R pragmatic decision-making discrepancy model.  In particular, see the Type I aptitude-achievement discrepancy feature in the second figure.  

The WJ and WJ-R SAPT's were differentially weighted combinations of the four best predictive tests across the norms sample.  See the two figures below which show the weighting schemes used.  Due to the lack of computerized norm tables and scoring that is now possible, a single set of average test weights were used for all ages.

[WJ SAPT weights]

 As I wrote in 1986, "because of their differential weighting system, the WJTCA Scholastic Aptitude clusters should provide some of the best curriculum-specific expectancy information available in the field of psychoeducational assessment" (p. 217).  Woodock (1984), in a defense of the SAPTs in School Psychology Review, made it clear that the composition of these clusters was to make the best possible aptitude-achievement comparison.  He stated that "the mix of cognitive skills included in each of the four scholastic aptitude clusters represents the best match with those achievement skills that could be obtained from the WJ cognitive subtests" (p.359).  However, the value of the WJ SAPTs were not fully appreciated at the time and was largely due to the IQ-ACH discrepancy model that constrained assessment professionals from using these measures as intended (McGrew, 1994).  This, unfortunately, led to their elimination in the WJ III and their replacement with the Predicted Achievement (PA) option which provided achievement domain-specific predictions of achievement based on the age-based optimal weighting of the seven individual tests that comprised the WJ III GIA-Std cluster.  Although effective and stronger predictors of achievement than the GIA-Std, the PA option never captured the attention of many assessment professionals...for a number of reasons (not covered here).

As I reiterated in 1994, when discussing the WJ-R SAPTs (same link as before), "The purpose of the WJTCA-R differential aptitude clusters is to provide predictions of current levels of achievement.  If a person obtains low scores on individual tests that measure cognitive abilities related to a specific achievement area and these tests are included in the aptitude cluster, then the person's current achievement expectancies should also be lowered.   This expectancy information will be more accurately communicated by the narrower WJTCA-R different aptitude clusters than by any broad-based score from the WJTCA-R or other tests" (p. 223).

The original WJ and WJ-R SAPTs were not presented as part of an explicitly defined comprehensive SLD identification model based on the concepts of consistency/concordance as was eventually advanced by Flanagan et al, Hale et al and Naglieri.  They were presented as part of a more general psychoeducational pragmatic decision making model.  However, it is clear that the WJ and WJ-R SAPTs were ahead of their time as they philosophically are in line with the aptitude portion of the aptitude-achievement consistency/concordance component of contemporary third method  SLD models.  In a sense, the field has now caught up with the WJ/WJ-R operationalization of aptitude clusters and they would now serve an important role in the aptitude-consistency SLD models.  It is my opinion that they represented the best available measurement approach to operationalizing domain-specific aptitudes for different achievement domains, which is at the heart of the new SLD models.

It is time to bring the SAPT's back...Back to the Future...as the logic of their design is a nice fit with the aptitude component of the aptitude-achievement consistency/concordance SLD models.  The field is now ready for this type of conceptualized and developed measure.

However, the original concept can now be improved upon via the methods and analyses presented in this (and the prior) post.  They can be improved upon via two methods:

1.  CHC-consistent aptitude clusters (aka, CHC designer aptitudes).  Creating  4-5 test clusters that are the bests predictors of achievement subdomains should utilize the extant CHC COG-->ACH relations literature when selecting the initial pool of tests to include in the prediction models.  This extant research literature should also guide the selection of variables in the final models...the models should not allowed to be driven by the raw empiricism of prediction.  This varies from the WJ and WJ-R SAPTS which were designed primarily based on empirical criteria (which combination predicted the most achievement variance), although their composition often made considerable theoretical sense when viewed via a post-hoc CHC lens.

2.  Provide age-based developmental weighting of the tests in the different CHC SAPTs.  The authors of the WJ III provided the necessary innovation to make this possible when they implemented an approach to constructing age-based differentially-weighted GIA g-scores via the WJ III computer scoring software.  The same technology can readily be applied to the development of CHC-designed SAPTS with developmentally shifting weights (as per the smoothed curves in the models above).  The technology is available.

Finally, I fully recognize that there are significant limitations in using an incremental partitioning of variance multiple regression approach to develop CHC-based SAPT's.  In other papers (g+specific abilities research using SEM causal models) I have been critical of this method.  The method was used here in an "intelligent" manner.....the selection of the initial pool of predictors was guided by the CHC COG-ACH extant literature and variables were not allowed to enter blindly into the final models.  The purpose of this (and the prior post) is to demonstrate the feasibility of designing CHC-consistent scholastic aptitude clusters.  I am pursuing other analyses with different methods to expand and improve upon this set of formative analyses and results.

Build it and they shall come.