AP101 Brief # 15: Beyond CHC: Cognitive-Aptitude-Achievement Trait Complex Analysis: Implications for SLD Assessment and Dx

This is the final post in a series of posts clarifying the nature of cognitive, aptitude, achievement ability constructs.  Readers should consult the preceding post (which contains links to all prior background posts) that defined cognitive abilities, aptitudes, achievement abilities, and CHC cognitive-aptitude-achievement trait complexes (CATTCs).  I apologize for not including the reference list.  These posts are snippets of a manuscript in preparation and I like to post to IQs Corner for feedback that I might incorporate in the final manuscript.  References are the last thing I do.

Beyond CHC:  CHC Cognitive-Aptitude-Achievement Trait Complex Analyses

I have previously argued that alternative non-factor analysis methodological (e.g., multidimensional scaling-MDS) and theoretical lenses need to be applied to validated CHC measures to better understand “both the content and processes underlying performance on diverse cognitive tasks” (McGrew, 2005, p. 172).  When MDS “faceted” methods have been applied to data sets previously analyzed by exploratory or confirmatory factor methods, “new insights into the characteristics of tests and constructs previously obscured by the strong statistical machinery of factor analysis emerge.” (Schneider & McGrew, 2012, p. 110).[1]   

Following the methods similar to that explained and demonstrated by Beauducel, Brocke and Liepmann (2001), Beauducel and Kersting (2002), SÜß and Beauducel (2005), Tucker-Drob and Salthouse (2009; this is an awesome example of MDS analyses side-by-side with factor analsysis of the same set of variables) and Wilhelm (2005), I subjected all WJ-R standardization subjects (McGrew, Werder & Woodcock, 1991) who had complete sets of scores (i.e., listwise deletion of missing data) for the WJ-R Broad Cognitive Ability-Extend (BCA-EXT), Reading Aptitude (RAPT), Math Aptitude (MAPT), Written Language Aptitude (WLAPT), Gf-Gc cognitive factors (Gf, Gc, Glr, Gsm, Gv, Ga, Gs), and Broad Reading (BRDG), Broad Math (BMATH), and Broad Written Language (BWLANG) achievement clusters to a Guttman Radex MDS analysis (n = 4,328 subjects from early school years to late adulthood).[2]  MDS procedures have more relaxed assumptions than linear statistical models and allow for the simultaneous analysis of variables that share common variables or tests—a situation that results in non-convergence problems due to excessive multicolinearity when using linear statistical models.  This feature made it possible to explore the degree of similarity of the WJ-R operationalized measures of the constructs of cognitive abilities, general intelligence (g), scholastic aptitudes, and academic achievement, in a single analysis.  That is, it was possible to explore the relations between and among the core elements of CHC-based cognitive-aptitude-achievement trait complexes (CAATC).  The results are presented in Figure 1. [Click on images to enlarge] 

Figure 1 (Click on image to enlarge)

WJ-R MDS Analysis:  Basic Interpretation

In Guttman Radex models, variables closest to the center of the 2-D plots are the most cognitively complex. Also, the variables are located along two continua or dimensions that often have substantive/theoretical interpretations.  The two dimensions in Figure 1 are labeled A<->B and C<->D.  The following is concluded from a review of Figure 1:

--The WJ-R g-measure (BCA-EXT) is almost directly at the center of the plot and is the most cognitive complex variable.  This makes theoretical sense given that it is a composite comprised of 14 tests from 7 of the CHC Gf-Gc cognitive domains.  Proximity to the center of MDS plots is sometimes considered evidence for g.

--Reading and Writing Aptitude (GRWAPT) and MAPT are also cognitively complex.  Both the GRWAPT[3] and MAPT clusters are comprised of four equally weighted tests of four different Gf-Gc abilities—and thus, the finding that they are also among the most cognitively complex WJ-R measures is not surprising.  The CHC Gf-Gc cognitive measures of Gf and Gc are much more cognitively complex that Gv, Glr, Ga and Gsm.[4]

--The A<->B dimension appears to reflect the ordering of variables as per stimulus content, a common finding in MDS analyses.  The cognitive variables on the left-hand side of the continuum midlines (Gv, Glr, Gf, Gs, MAPT) are comprised of measures with predominant visual-figural or numeric/quantitative characteristics.  The majority of the variables on the right-hand of the continuum midline (GRWAPT, Gc, Ga, Gsm, BRDG, BWLANG) are characterized as more auditory-linguistic, language, or verbal.  This visual-figural/numeric/quantitative-to-auditory-linguistic/language/verbal content dimension is very similar to the verbal, figural, and numeric content facets of the Berlin Model of Intelligence Structure (BIS; SÜß and Beauducel, 2005).[5] 

--The C<->D dimension appears to reflect the ordering of variables as per cognitive operations or processes, another common finding in MDS analyses.  The majority of the cognitive variables above the continua midline (Gv, Glr, Ga, Gc, Gsm, BCAEXT, GRWAPT) are comprised primarily of cognitive ability tasks the involve mental processes or operations.  Conversely, although not as consistent, three of the lowest variables below the continua midline are the achievement ability clusters (BRDG BWLANG; BMATH).  Thus, the C<->D dimension is interpreted as representing a cognitive operations/process-to-acquired knowledge/product dimension.

--In contrast to factor analysis, interpretation of MDS is more is more qualitative and subjective.  Variables that may share a common dimension are typically identified as lying on relatively straight lines or planes, in separate quadrants or partitions, or tight groupings (often represented by circles or ovals or connected as a shape via lines).  Inspecting the four quadrants created by the A<->B C<->D dimensions (see Figure 1) suggests the following.  The AC quadrant is interpreted to represent (excluding BCAEXT which is near the center) cognitive operations with visual-figural content (Gv; Glr).  The CB quadrant is interpreted as representing auditory-linguistic/language/verbal content based cognitive operations.  The BC quadrant only includes the three broad achievement clusters, and is thus an achievement or an acquired knowledge dimension.  Finally, the DA quadrant can be interpreted as cognitive operations that involve quantitative operations or numeric stimuli (e.g., Gf is highly correlated with math achievement; McGrew & Wendling, 2010; one-half of the Gs-P cluster is the Visual Matching test which requires the efficient perceptual processing of numeric stimuli—Glr-N).[6]  The interpretation of these four quadrants is very consistent with the BIS content-faceted content-by-operations model research.

--The theoretical interpretation of the two continua and four quadrants provides potentially important insights into the abilities measured by the WJ-R measures.  More importantly, the conclusions provide potentially important theoretical insights into the nature of human intelligence, insights that typically fail to emerge when using factor analysis methods (see Schneider & McGrew, 2012 and SÜß and Beauducel, 2005). In other MDS analyses I have completed, similar visual-figural/numeric/quantitative-to-auditory-linguistic/language/verbal and cognitive operations/process-to-acquired knowledge/product continua dimensions have emerged (McGrew, 2005; Schneider & McGrew, 2012).  When I have investigated a handful of 3-D MDS[7] models the same two dimensions emerge along with a third automatic-to-deliberate/controlled cognitive processing dimension which is consistent with the prominent dual-process models of cognition and neurocognitive functioning (Evans, 2008, 2011; Barrouillet, 2011; Reyna & Brainerd, 2011; Rico & Overton, 2011; Stanovich, West & Toplak, 2011) that are typically distinguished as Type I/II or System I/II  (see Kahneman’s, 2011, highly acclaimed Thinking, Fast and Slow).[8] 

--These higher-order cognitive processing dimensions, which are not present in the CHC taxonomy, suggest that intermediate strata (or dimensions that cut across broad CHC abilities) might be useful additions to the current three-stratum CHC model.  These higher-order dimensions may be capturing the essence of fundamental neurocognitive processes and argue for moving beyond CHC to integrate neurocognitive research to better understand intellectual performance.

WJ-R MDS Analysis:  Cognitive-Aptitude-Achievement Trait Complex (CAATC) Interpretation

Figure 2 is an extension of the results presented in Figure 1.  Two different CAATCs are suggested.  These were identified by starting first with the BMATH and BRDG/BWLANG achievement variables and next connecting these variables to their respective SAPTs (GRWAPT; MAPT).  Next, the closest cognitive Gf-Gc measures that were in the same general linear path were connected (the goal was to find the math and reading related variables that were closest to lying on a straight line).  Ovals encompassing the entire space comprising the two circle-line-circle traces where superimposed on the figure.  A dotted line that represented the approximate bisection of each of the cognitive-aptitude-achievement trait complex vectors was drawn.  Finally, an approximate correlation (r = .55; see Figure 2) between the two multidimensional CAATC was estimated via measurement of the angle between the CAATC vector dotted lines.[9]

Figure 2 (Click on image to enlarge)

As presented in Figure 2, Math and Reading-Writing CAATCs are suggested as a viable perspective from which to view the relations between cognitive abilities, aptitudes, and achievement abilities.  The primary conclusions, insights, and questions are drawn from Figure 1 and 2 are:

--It appears that the potential exists to empirically identify CAATCs via the use of CHC-grounded theory, the extant CHC COG->ACH relations research, and multidimensional scaling.  It also appears possible to estimate the correlation between different trait complexes (see math/reading-writing trait complex r = .55 in Figure 2).  I suggest these preliminary findings may help the field of cognitive-achievement assessment and research better approximate the multidimensional nature of human cognitive abilities, aptitudes, and achievement abilities.

--Although the WJ-R battery is not as comprehensive a measure of CHC abilities as the WJ III, the cognitive abilities within the respective math and reading/writing CAATCs are very consistent with the extant CHC COG->CHC relations research (McGrew & Wendling, 2010; click here for visual-graphic summary).  The reading-writing trait complex (see Figure 2) includes Ga-PC, Gc-LD/VL, and via the GRWAPT, Gs-P, and Gsm-MS, abilities that are listed as domain-general and domain-specific abilities in Figure 3.  In the case of math, the trait complex includes indicators of Gf-RG, Gv-MV, and via the MAPT, Gs-P (Visual Matching, which might also tap Gs-N) and Gc-LD/VL, abilities that are either domain-general or domain-specific for math in Figure 3.  Working memory (Gsm-WM) is not present (as suggested by Figure 3) as the WJ-R battery did not include a working memory cluster that could enter the analysis.

Figure 3 (Click image to enlarge)

--Also of interest are the three WJ-R cognitive factors (Gsm-MS, Glr-MA, Gs-P) that are excluded from the hyperspace representations of the proposed math and reading-writing CAATCs.  Although highly speculative, it may be possible that their separation from the designated trait complexes suggest, that if known to be related to reading-writing or math achievement, their independence from the narrower trait complexes may be an indication that they represent domain-general abilities.  Glr-MA and Gs-P are both listed as domain-general abilities in Figure 3.  Additional work is needed to determine if the independence (from identified CAATCs) of CHC measures known to be significantly related to achievement indicates domain general abilities.  Alternatively, it is very possible, given the previously demonstrated developmental nuances of CHC COG->ACH relations that the results presented in Figures 1 and 2, which used the entire age range of the WJ-R measures, may mask or distort findings in unknown ways.

--Those knowledgeable of the CHC COG->ACH relations research will obviously note the prior inclusion of certain Gv abilities (Vz, SR, MV) in Figure 3 as well as the inclusion of the WJ-R Gv-MV/CS cluster as part of the proposed math CAATC (Figure 2), despite the lack of consistently reported significant CHC Gv-ACH relations.  McGrew and Wendling (2010) recognized that some Gv abilities have clearly been linked to reading and math achievement (especially the later) in non CHC-organized research.  They speculated that the “Gv Mystery” may be due to certain Gv abilities being threshold abilities or that the cognitive batteries included in their review did not include Gv measures that measured complex Gv related Vz or MV processes.  Given this context, it may be an important finding (via the methods described above) that the WJ-R Gv measure is unexpectedly included in the math CAATC.  This may support the importance of Gv abilities in explaining math and concurrently indicate a problem with the operational Gv measures. 

--The long distance from the WJ-R Gv measure to the center of the diagram (see Figure 2) indicates that the WJ-R Gv measure, which included tests classified as indicators of CS and MV, is not cognitively complex.  This conclusion is consistent with Lohman’s seminal review of Gv abilities (Lohman, 1979) where he specifically mentions CS and MV as representing low level Gv processes and “such tests and their factors consistently fall near the periphery of scaling representations, or at the bottom of a hierarchical model” (Lohman, 1979, 126-127).  I advance the hypothesis that the math CAATC in Figure 2 suggests that Gv is a math-relevant domain, but more complex Gv tests (e.g., 3-D mental “mind’s eye” rotation; complex visual working memory), which would be closer to the center of the MDS hyperspace, need to be developed and included in cognitive batteries.  This suggestion is consistent with Wittmann’s concept of Brunswick Symmetry, which, in turn, is founded on the fundamental concept of symmetry which has been central to success in most all branches of science (Wittmann & SÜß, 1999).  The Brunswick Symmetry model argues that in order to maximize prediction or explanation between predictor and criterion variables, one should match the level of cognitive complexity of the variables in both the predictor and criterion space (Hunt, 2011; Wittmann & SÜß, 1999).  The WJ-R Gv-WJ-R BRMATH relation may represent a low (WJ-R Gv)-to-high (WJ-R BMATH) predictor-criterion complexity mismatch, thus dooming any possible significant relation. 

--Researchers and practitioners in the area of SLD should recognize that when third method POSW “aptitude-achievement” discrepancies are evaluated to determine “consistency”, the combination of domain-general and domain-specific abilities that comprise an aptitude for a specific achievement domain in many ways can be considered a mini-proxy for general intelligence (g).  In Figures 1 and 2 the BCA-EXT and MAPT and GRWAPT variables are in close proximity (which also represents high correlation) and are all near the center of the MDS Radex model.  The manifest correlations between the WJ-R BCA-EXT (in the WJ-R data used to generate the CAATCs in Figure 10) and RAPT, WLAPT, and MAPT clusters are .91, .89 and .91, respectively.  This reflects the reality of the CHC COG->ACH research as in both reading and math achievement, cognitive tests or clusters with high g-loadings (viz., measures of Gc and Gf), as well as shared domain-general abilities, are always in the pool of CHC measures associated with the academic deficit.

--However, the placement of GRWAPT and MAPT in the different content/operations quadrants in Figures 1 and 2 suggests that more differentiated CHC-designed achievement domain SAPT measures might be possible to develop.   The manifest correlations between MAPT and the two GRWAPT measures were .82 to .84, suggesting approximately 69 % shared variance.  GRWAPT and MAPT are strongly related SAPTs, yet there is still unique variance in each.  Furthermore, the WJ-R SAPT measures used in this analysis were equally weighted clusters and not the differentially weighted clusters as in the original WJ.  As presented previously, research suggests that optimal SAPT prediction requires developmentally shifting weights across age.  It is my opinion that the development of developmentally-sensitive CHC-designed SAPTs will result in lower correlations between RAPT and MAPT measures.

Beyond CHC Theory:  Cognitive-Aptitude-Achievement Trait Complexes and SLD Identification Models

The possibility of measuring, mapping and quantifying CAATCs raises intriguing possibilities for re-conceptualizing approaches to the identification of SLD.  Figure 4 presents the generic representation of the prevailing third-method SLD models as well as a formative proposal for a conceptual revision.  As noted previously, the prevailing POSW model (left half of Figure 4), although useful for communication and enhancing understanding of the conceptual approach, is simplistic.   Implementation of the model requires successive calculations of simple (and often multiple) discrepancies which fails to capture the multidimensional and multivariate nature of human cognitive, aptitude, and achievement abilities.  I believe that the CAATC representations in Figure 2, although still clearly imperfect and fallible representations of the non-linear nature of reality, are a better approximation of the complex nature of cognitive-aptitude trait complex relations.  The right-side of Figure 4 is an initial attempt to conceptualize SLD within a CAATC framework.  In this formative model, the bottom two components of the current third-method models (i.e., academic and cognitive weakness) have been combined into a single multidimensional CAATC domain.

Figure 4 (Click on image to enlarge)

CAATCs better operationalize the notion of consistency among the multiple cognitive, aptitude, and achievement elements of an important academic learning domain or domain of SLD.  As noted in the operational definition of a CAATC presented earlier, the emphasis is on a constellation or combination of elements that are related and are combined together in a functional fashion.  These characteristics imply a form of a centrally inward directed force that pulls elements together much like magnetism.  Cohesion appears the most appropriate term for this form of multiple element bonding.  Cohesion is defined, as per the Shorter English Oxford Dictionary (2002), as “the action or condition of sticking together or cohering; a tendency to remain united” (p. 444).  Element bonding and stickiness are also conveyed in the APA Dictionary of Psychology (VandenBos, 2007) definition of cohesion as “the unity or solidarity of a group, as indicated by the strength of the bonds that link group members to the group as a whole” (p. 192).  Thus, in the CAATC-based SLD proposal in Figure 4, the degree of cohesion within a CAATC (as designed by circular icon shape) is considered an integral and critical step to ascertaining if a strong cohesive CAATC, which represents a particular academic domain deficit, is present.  

The stronger the within-CAATC cohesion, the more confidence one could place in the identification of a CAATC as possibly indicative of a SLD.  This focus on quantifying the CAATC cohesion is seen as a necessary, but not sufficient, first step in attempting to identify SLD based on a multivariate POSW.  If the CAATC demonstrates very weak cohesion, the hypothesis of a possible SLD should receive less consideration.  If there is significant (yet to be defined) moderate to strong CAATC cohesion, then the comparison of the CAATC to the cognitive/academic strengths portion of the conceptual model is appropriate for SLD consideration.  To simplify, POSW-based SLD identification would be based first on the identification of a weakness in a cohesive specific CAATC which is then determined to be significantly discrepant from relative strengths in other cognitive and achievement domains.  

Of course, additional variations of this model require further exploration.  For example, should discrepant/discordant comparisons be made between other empirically identified and quantified CAATCs?  Would CAATC-to-CAATC comparisons between high empirical and theoretically correlated CAATCs (e.g., basic reading skills and basic writing skills), when contrasted to less empirically and theoretically correlated CAATC-to-CAATC domains (e.g., basic reading skills and math reasoning), be diagnostically important?  I have more questions than answers at this time.

      

Yes—this proposed framework is speculative and in the formative stages of conceptualization.  It is based on exploratory data analyses, theoretical considerations, and well reasoned logic.  It is not yet ready for applied practice.  Appropriate statistical metrics and methods for operationalizing the degree of domain cohesion are required.  I do not see this as an insurmountable hurdle as methods based on Euclidean distance measures (e.g., Mahalanobis and or Minkowski distance) which can quantify the cohesion between CAATC measures as well as the distance of all the trait complex elements from the centroid of a CAATC exist.  Or, statisticians much smarter than I can might apply centroid-based multivariate statistical measures to quantify and compare CAATC domain cohesion.  I urge those with such skills and interest to pursue the development of these metrics.  Also, the current limited exploratory results with the WJ-R data should be replicated and extended in more contemporary samples with a larger range of both CHC cognitive, aptitude, and achievement tests and clusters.  I would encourage split-sample CAATC model-development and cross-validation in the WJ III norm data.

The proposed CAATC framework, and integration into SLD models is, at this time, simply that—a proposal.  It is not ready for prime-time, in-the-field implementation.  It is presented here as a formative idea that will hopefully encourage others to explore.  Additional research and development, some of which I suggested above, will either prove this to be a promising methodology or an idea with limited validity or one with too many practical constraints that render it hard to implement.  Nevertheless, the results presented here suggest promise.  The results suggest possible incremental progress toward better defining SLD and learning complexes that are more consistent with nature—with the identification of CAATC taxon’s[10] that better approximate “nature carved at the joints” (Meehl, 1973, as quoted and explained by Greenspan, 2006, in the context of MR/ID diagnosis).  Such a development would be consistent with Reynolds and Lakin’s (1987) plea, 25 years ago, for disability identification methods that better represent dispositional taxon’s rather than classes or categories based on specific cutting scores which are grounded in “administrative conveniences with boundaries created out of political and economic considerations” (p. 342). 

---

[1] See SÜß and Beauducel (2005) and Tucker-Drob and Salthouse (2009) for excellent descriptions of these methods and illustrative results.

[2] The WJ-R battery was analyzed since it was the last version of the WJ series to include scholastic aptitude clusters.

[3] As noted in Figure 1, the Reading and Written Language Aptitude clusters, which were separate variables in the analysis, shared 3 of 4 common tests and nearly overlapped in the MDS plot.  Thus, for simplicity they were combined into the single GRWAPT variable in Figure 1.  This is also consistent the factor analysis of reading and writing achievement variables that typically produce a single Grw factor and not separate reading and writing factors.

[4] The primary narrow abilities measured by each of the cognitive Gf-Gc cluster are included in the label for each cluster.  Contrary to the WJ III, the Gf-Gc clusters were not all operationally constructed as broad Gf-Gc abilities (see McGrew, 1997; McGrew & Woodcock, 2001).  Only the WJ-R Gf and Gc clusters can be interpreted as measuring broad domains as per the requirement that broad measures must include indicators of different narrow abilities (e.g., Concept Formation-I and Analysis-Synthesis-RG).  The other five WJ-R Gf-Gc clusters are now understood to be valid indicators of narrow CHC abilities (Gsm-MS; Ga-PC; Glr-MA; Gv-MV/CS; Gs-P).

[5]  The BIS model is a heuristic framework, derived from both factor analysis and MDS facet analysis, for the classification of performance on different tasks and is not to be considered a trait-like structural model of intelligence as exemplified by the factor-based CHC theory.  Nevertheless, Guttman Radex MDS models often show strong parallels to hierarchical factor based models based on the same set of variables (Kyllonen, 1996; SÜß & Beauducel, 2005; Tucker-Drob & Salthouse, 2009).

[6] The MAPT cluster also includes the two Gf tests and Visual Matching.

[7] WJ III 3-D MDS model for norms subjects aged 9-13 is available at http://www.iqscorner.com/2008/10/wj-iii-guttman-radex-mds-analysis.html

[8] A similar dimension emerged as a plausible higher-order cognitive processing dimension in the previously mentioned Carroll type analysis of 50 WJ III test variables.

[9] Using trigonometry, the cosine of the intersection of the two trait complex vectors was converted to a correlation.  I thank Dr. Joel Schneider for helping fill the gap in my long-lost expertise in basic trigonometry via an excel spreadsheet that converted the measured angle to a correlation.

[10] The Shorter Oxford English Dictionary defines a taxon as “a taxonomic group of any ran, as species, family, class, etc; an organism contained in such a group” (p. 3193) and taxonomy as “classification, esp. in relation to its general laws or principles; the branch of science, or of a particular science or subject, that deals with classification; esp. the systematic classification of living organisms” (p. 3193; italics in original)