Ability composite scores in the face of variability: Lohman's thoughts

David Lohman (University of Iowa) made a post to the CHC listserv today regarding a recurring question that surfaces in the interpretation of composite scores in the face of significant variability in the tests that comprise a composite. I have reproduced his comments below and have embedded a link to a paper to which he refers.

This post is for the quanotids who read this blog and who are dying for their RDA of quanotid-speak.

David Lohman stated in CHC listserv post on 4-6-05

• "As I see it, there are two issues in the spread or scatter of scores used to estimate a composite. The first is the dependability of the composite score and the second is whether the test measures more than one dimension. Most people worry about the latter problem, I and my students have worried about the former problem. The basic idea is that the composite or average score is most meaningful in those cases in which the scores that define it agree with one another and least meaningful when part or subtests scores vary markedly. The problem is how best to represent this variability in a way that is meaningful for test users. We have approached the problem from the standpoint of person fit in IRT, but expressed the degree of misfit in the metric of the score scale. Specifically, we use variability of subtest scores as one estimate of error of measurement in the composite score for the individual. Examinees who show much scatter have a composite scores with wide confidence intervals whereas examinees with minimal scatter have the confidence intervals typical for that score level (i.e., proportional to the conditional standard error of measurement). These procedured are actually implemented in the 6th edition of the Cognitive Abilities tests, which I co-author with Betty Hagen. For a brief description of how these individual confidence intervals are derived, see pp 62-63 in the CogAT6 Research Handbook."

For a more extensive discussion, click here to see(and/or download) a paper by Lohman that describes his approach to this issue.