CHC cognitive-achievement relations: Limitations in prior reviews

This is another post in a continued series of posts re: research synthesis of the CHC cognitve-achievement research literature.

Click here for other posts in this series.

The sheer number of key, review, and individual studies populating the published Flanagan et al. (2006) CHC CB COG-ACH correlates summary tables is impressive. However, the available CHC COG-ACH relations summaries suffer from a number of limitations.

First is a lack of descriptive and operational rigor in the CHC COG-ACH syntheses. The Flanagan research group’s efforts appear to fit the definition of a purposeful research synthesis which “focus on empirical studies and seek to summarize past research by drawing overall conclusions from many separate investigations that address related or identical hypotheses” (Cooper, 1998, p. 3). The available CHC COG- ACH relations research syntheses are narrative as they do not attempt to quantify significance results across studies, an approach typically defined as a meta-analysis (Cooper, 1998). It would be nearly impossible to replicate the CHC COG-ACH relations summaries in Flanagan et al. (2006) due to the lack of: (a) specified and published database search terms and date ranges, (b) significant CHC-ACH relation significance criteria, (c) study inclusion and exclusion criteria, and (d) procedures to control for possible publication bias (see Cooper, 1998; also see Pillemer & Light, 1980). The research summaries to date have largely been informal "tallies" and the continued listing of more and more studies.

Second, the use of the broad dependent variable (DV) domains of reading, math, and writing may cloud important sub-achievement domain findings. From the available CHC COG- ACH synthesis tables it is impossible to ascertain if the salient COG-ACH relations generalize across sub-achievement domains (e.g., basic reading skills vs. reading comprehension). Third, although general developmental CHC COG-ACH trends are briefly discussed by Flanagan et al. (2006), they are only based on the findings from the handful of key CHC studies. The age characteristics of the bulk of the studies included (review and individual studies) are ignored and no attempt is made to present CHC COG-ACH results by age-differentiated sub-samples.

Finally, specification error, sometimes called omitted-variable bias (Keith, 2006), occurs when potentially important variables in predictive or explanatory research are not included in a study’s research design. Specification error is a form of modeling error that can lead to biased estimates of the effects (i.e., relative importance) of individual predictor variables on the dependent variable of interest (Pedhazur, 1997).

Floyd et al. (2003) used an analogy from sports to explain specification error.

"As an analogy, a college basketball coach could construct a regression model that included season-long player statistics of starters in order to predict post-season performance. If the coach omitted the game statistics for the point guards and included only the statistics for the remaining four starters, over- or under- estimates of the importance of one or more of the other four starting positions would probably result. These biased findings might lead the coach to make erroneous decisions about the strengths of certain player positions when developing game strategies. Because of the failure to include measures of potentially important." (p. 156)

Almost all the review and individual studies listed in the contemporary CHC COG- ACH summary tables suffer from some unknown degree of specification error when the intent is to identify the relative importance of the various CHC domains within the CHC framework. Two examples are presented below.

An individual study example is Wagner et al. (1997) a study listed as evidence for the importance of Gc, Ga, and Glr in reading. Specification error is possibly present due to the lack of indicators of the more complete array of potentially important CHC abilities for reading. The CHC cognitive abilities of Ga, Gc, and Glr may have displayed slightly different degrees of relative importance (and might have demonstrated non-significance) in the reported analysis had the predictor models included measures of Gsm, Gf, Gs, and Gv.

In the CHC COG-ACH correlate summary tables published to date, Aaron (1995) is classified as a review study. Inspection of the original article finds it not to be a systematic review in the traditional sense of a literature review (Cooper, 1998). Aaron's article describes a diagnostic procedure for reading disabilities and draws upon select literature sources. This is not a criticism of the original purpose of the Aaron (1995) publication, nor is this a criticism of the categorization of the Aaron study as a review in the published summary tables. The point is that the term review typically has the connotation in the scientific literature of systematic narrative or empirical research synthesis of extant research studies.