WJ test reliability over time

"A test does not change from one time to another: people do. There may be considerable change on some traits, but relatively little on others. Test-retest studies evaluate the tendency for change in people, not some aspect of test quality. A test that does not reflect such changes in human traits would be an insensitive measure of those traits" (McGrew, Werder, & Woodcock, 199, p. 99).

Over on the NASP Listserv Dr. Gary Canivez asked the following question, in response to a post regarding changes in scores on the K-ABC and WJ---"Does anyone have references for long-term stability of WJ or KABC-2 scores? I'd be interested in references for such studies."

There was a very sophisticated test-retest study reported in the WJ-R Technical Manual (McGrew, Werder & Woodcock, 1991) (click here to view/download). Unfortunately it is in a test technical manual...a document that is too often ignored once a test is purchased. Additonal information can be found in the following article.

• McArdle, J. J., FerrerCaja, E., Hamagami, F., & Woodcock, R. W. (2002). Comparative longitudinal structural analyses of the growth and decline of multiple intellectual abilities over the life span. Developmental Psychology, 38(1), 115-142. (click to view)
  

I urge practicing psychologists to read these reports. Yes..they are hard to digest. They are not reporting simple test-retest correlations. Instead, they are complex research designs intended to identify and partition the sources of test score variance that change over time. Scores for some tests will change over time...and this is what you want...as it is reflecting trait variance....some traits do change over time (just as our weight changes over time...unfortunately, using in an upward direction). Other traits measured by tests are more stable across time...and thus the scores will likely change less.

And then, of course, a person's state (concentration, anxiety, fatigue, etc.) at any testing moment can impact test performance...and test's sensitive to these states (e.g., Gsm, Gs) will likely reflect these temporary state fluctuations. This reflects state variance...which is NOT a problem with the measure...the measure is accurately reflecting how the person is doing at that time. School psychologists (and others who do psychological testing), unfortunately, typically receive measurement training that only talks about simple test-retest reliability studies and results....a diservice to our profession. We need to understand our instruments better. Properly designed test-retest studies, where the retest interval is varied, can help identify and portion the difference sources of test score variance to the appropriate sources of stable or unstable change score variance.

Read the WJ-R technical manual excerpt in particular. It is the most readable and understandable of the two sources I've included in this post.

Technorati Tags: psychology, school psychology, educational psychology, neuropsychology, psychometrics, reliability, test-retest, measurement, WJ-R, WJ III, K-ABC, KABC-II, Woodcock Johnson