Wots the word? "Whaz up" with bad stats in dyslexia study?

I recently ran across the following article in the journal Dyslexia, an article that suggested that measures of word reading (WJ III Letter-Word Identification) and pseudoword reading (WJ III Word Attack test) are measuring the same construct. As reported in the abstract below, the authors report a very high correlation of .94 between these measures in a sample of 71 subjects (ages 6 to 9). As highlighted in the abstract conclusion, the authors suggest that measures of pseudoword reading (Word Attack) are of dubious value, as they measure little different from tests of simple word reading (Letter-Word Identification).

Unfortunately, a close inspection of the research methods suggests a serious problem with the analysis. The authors correlated raw scores for achievement measures. It is well known that raw scores (as well as other developmental based scores such as the WJ III W-scores) are developmental in nature...that is..the scores increase systematically with age due to simply maturation and the influence of moving through successive grades. Developmental scores across a wide-age range cannot be correlated without first removing the confound of the age/maturation-based developmental variance shared by the correlated measures. The authors did NOT do this. They do report "standardizing" the scores from each measure in the study, but this sounds like they standardized the scores across the entire age range...a process that retains the developmental/age-based variance. [The authors should have standardized the metric for each measure BY AGE. Although the age subsamples make this a bit tricky. Ideally, the absence of standard scores, the authors should have partialled out the influence of chronological age, saved the residuals, and then correlated the residuals].

The proper way to conduct correlations with developmental scores (raw scores; W-scores) across a wide age-range, is to first remove the developmental/age variance either by using age-based standard scores or by partialling out the developmental age variance from the developmental scores, and then correlating the residual variance scores.

To make this point, I took the WJ III norm data [conflict of interest disclosure - I'm a coauthor of the WJ III) and selected all norm subjects from ages 6 yo 9. I first correlated (inappropriately) the W-scores, which is similar to the authors correlation raw scores. The correlation was .85. Although not as high as the .94 reported by the authors, it still is at the high end. Then, I eliminated the problem of the "third variable" confound (age/developmental variance shared by both Letter-Word ID and Word Attack) by correlating, for the same subjects, the age-based standard scores (Mean=100; SD=15). The correlation dropped to .72.

It is clear that the incorrect correlation of raw scores reported in the article produced a spuriously high correlation of .94, a value that suggests the two measures of reading shared 88% common variance (.94 squared). In a more nationally representative sample this inappropriate correlation is .85, which suggests .72% common variance. However, the accurate result is the .72 52 % shared variance correlation, a value that indicates . 52 % shared variance suggests a much different conclusion than does th 88% incorrect reported value. Although word reading (Letter-Word ID) and pseudoword reading (Word Attack) tests are significantly correlated (.72), they only share approximately 50% common variance, a finding that suggests that they are related measures, but are measures that still provide measurement of unique aspects of the reading process.

The rest of the results reported in the article are tainted by the same problem and should be ignored. I've posted scatterplots for the W-score and SS correlations I calculated for those who want to see these plots (click here).

• Thomson, D., Crewther, D. & Crewther, S (2006). Wots that Werd? Pseudowords (non-words) may be a Misleading Measure of Phonological Skills in Young Learner Readers. Dyslexia, 12, 289-299. (click here to view)

Abstract (italics added by blogmaster)

• Pseudoword (non-word) reading tasks are a commonly used measure of phonological processing across diverse fields of reading research. However, whether pseudoword reading gives any more information about phonological processing in young learner readers than does the reading of real words has seldom been considered. Here we show that pseudoword and real word reading are so strongly correlated (r=0.94) in the first 4 years of school as to be representative of the same construct. Two of the subskills of phonological processing, phonological awareness and rapid automatic naming also predict almost identical amounts of variance in pseudoword and real word reading. A divergence in the correlations between word and pseudoword reading and phonological awareness and rapid naming only emerges in the fourth year, while a significant correlation between phonological awareness and rapid automatic naming is evident only in the first year of schooling. Thus these results suggest that, at least in young children learning to read, care should be taken when using pseudoword reading to measure either phonological processing ability or phonological awareness as this may misinform the choice of therapy for a child showing symptoms of dyslexia.

Technorati Tags: psychology, education, educational psychology, neuropsychology, special education, dyslexia, LD, learning disabilities, reading, correlations, statistics, WJ III, Woodcock Johnson

powered by performancing firefox