Journal Alert - JOURNAL OF EDUCATIONAL MEASUREMENT

> Title:
> Validating the Interpretations and Uses of Test Scores
>
> Authors:
> Kane, MT
>
> Source:
> *JOURNAL OF EDUCATIONAL MEASUREMENT*, 50 (1):1-73; SI SPR 2013
>
> Abstract:
> To validate an interpretation or use of test scores is to evaluate the
> plausibility of the claims based on the scores. An argument-based
> approach to validation suggests that the claims based on the test scores
> be outlined as an argument that specifies the inferences and supporting
> assumptions needed to get from test responses to score-based
> interpretations and uses. Validation then can be thought of as an
> evaluation of the coherence and completeness of this interpretation/use
> argument and of the plausibility of its inferences and assumptions. In
> outlining the argument-based approach to validation, this paper makes
> eight general points. First, it is the proposed score interpretations
> and uses that are validated and not the test or the test scores. Second,
> the validity of a proposed interpretation or use depends on how well the
> evidence supports the claims being made. Third, more-ambitious claims
> require more support than less-ambitious claims. Fourth, more-ambitious
> claims (e.g., construct interpretations) tend to be more useful than
> less-ambitious claims, but they are also harder to validate. Fifth,
> interpretations and uses can change over time in response to new needs
> and new understandings leading to changes in the evidence needed for
> validation. Sixth, the evaluation of score uses requires an evaluation
> of the consequences of the proposed uses; negative consequences can
> render a score use unacceptable. Seventh, the rejection of a score use
> does not necessarily invalidate a prior, underlying score
> interpretation. Eighth, the validation of the score interpretation on
> which a score use is based does not validate the score use.
>
> ========================================================================
>
>
> *Pages: 74-83 (Article)
> *View Full Record: http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=CCC&DestLinkType=FullRecord;KeyUT=CCC:000316286300002
> *Order Full Text [ ]
>
> Title:
> Commentary on "Validating the Interpretations and Uses of Test Scores"
>
> Authors:
> Brennan, RL
>
> Source:
> *JOURNAL OF EDUCATIONAL MEASUREMENT*, 50 (1):74-83; SI SPR 2013
>
> Abstract:
> Kane's paper Validating the Interpretations and Uses of Test Scores is
> the most complete and clearest discussion yet available of the
> argument-based approach to validation. At its most basic level,
> validation as formulated by Kane is fundamentally a simply-stated
> two-step enterprise: (1) specify the claims inherent in a particular
> interpretation and/or use of test scores (IUA); and (2) provide an
> evaluation of the claims (validity argument). Kane discusses four types
> of inferences that provide a scaffolding for addressing these two
> arguments: scoring, generalization, extrapolation, and decision rules.
> Decision rules, in particular, are closely related to consequences,
> which loom large in the argument-based approach to validation. The
> present commentary on Kane's paper attempts to simplify some of his
> discussions, while expanding others. The author suggests that Kane's
> argument-based approach to validation offers by far the best current
> basis for optimism about improvements in validation.
>
> ========================================================================
>
>
> *Pages: 84-90 (Article)
> *View Full Record: http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=CCC&DestLinkType=FullRecord;KeyUT=CCC:000316286300003
> *Order Full Text [ ]
>
> Title:
> Getting the Help We Need
>
> Authors:
> Haertel, E
>
> Source:
> *JOURNAL OF EDUCATIONAL MEASUREMENT*, 50 (1):84-90; SI SPR 2013
>
> Abstract:
> In validating uses of testing, it is helpful to distinguish those that
> rely directly on the information provided by scores or score
> distributions (direct uses and consequences) versus those that instead
> capitalize on the motivational effects of testing, or use testing and
> test reporting to shape public opinion (indirect uses and consequences).
> Some uses and consequences, both direct and indirect, are intended;
> others are unintended. Unintended consequences pose greater challenges
> in test validation because they must be identified before they can be
> investigated. Validation of uses and consequences can employ theories
> and methods from various social science disciplines. Educational
> measurement is most closely allied with psychology and statistics, but
> sociologists, anthropologists, economists, linguists, and others also
> could help in theorizing and investigating the consequences of test use,
> especially indirect and unintended consequences.
>
> ========================================================================
>
>
> *Pages: 91-98 (Article)
> *View Full Record: http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=CCC&DestLinkType=FullRecord;KeyUT=CCC:000316286300004
> *Order Full Text [ ]
>
> Title:
> Validity in Action: Lessons From Studies of Data Use
>
> Authors:
> Moss, PA
>
> Source:
> *JOURNAL OF EDUCATIONAL MEASUREMENT*, 50 (1):91-98; SI SPR 2013
>
> Abstract:
> Studies of data use illuminate ways in which education professionals
> have used test scores and other evidence relevant to students'
> learningin action in their own contexts of workto make decisions about
> their practice. These studies raise instructive challenges for a
> validity theory that focuses on intended interpretations and uses of
> test scores as Kane's (this issue) does. This commentary explores
> implications of data use studies for elaborating Kane's approach to
> validation to accommodate the ways test scores are used with other
> sources of evidence to address users' questions.
>
> ========================================================================
>
>
> *Pages: 99-104 (Article)
> *View Full Record: http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=CCC&DestLinkType=FullRecord;KeyUT=CCC:000316286300005
> *Order Full Text [ ]
>
> Title:
> Agreeing on Validity Arguments
>
> Authors:
> Sireci, SG
>
> Source:
> *JOURNAL OF EDUCATIONAL MEASUREMENT*, 50 (1):99-104; SI SPR 2013
>
> Abstract:
> Kane (this issue) presents a comprehensive review of validity theory and
> reminds us that the focus of validation is on test score interpretations
> and use. In reacting to his article, I support the argument-based
> approach to validity and all of the major points regarding validation
> made by Dr. Kane. In addition, I call for a simpler, three-step method
> for developing validity arguments, one that focuses on explicit testing
> purposes, as suggested by the Standards for Educational and
> Psychological Testing. If testing purposes are appropriately
> articulated, the process of developing an interpretive argument becomes
> unnecessary and validation can directly address intended interpretations
> and uses.
>
> ========================================================================
>
>
> *Pages: 105-109 (Article)
> *View Full Record: http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=CCC&DestLinkType=FullRecord;KeyUT=CCC:000316286300006
> *Order Full Text [ ]
>
> Title:
> Two Kinds of Argument?
>
> Authors:
> Newton, PE
>
> Source:
> *JOURNAL OF EDUCATIONAL MEASUREMENT*, 50 (1):105-109; SI SPR 2013
>
> Abstract:
> Kane distinguishes between two kinds of argument: the interpretation/use
> argument and the validity argument. This commentary considers whether
> there really are two kinds of argument, two arguments, or just one. It
> concludes that there is just one argument: the validity argument.
>
> ========================================================================
>
>
> *Pages: 110-114 (Article)
> *View Full Record: http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=CCC&DestLinkType=FullRecord;KeyUT=CCC:000316286300007
> *Order Full Text [ ]
>
> Title:
> Truth and Evidence in Validity Theory
>
> Authors:
> Borsboom, D; Markus, KA
>
> Source:
> *JOURNAL OF EDUCATIONAL MEASUREMENT*, 50 (1):110-114; SI SPR 2013
>
> Abstract:
> According to Kane (this issue), the validity of a proposed
> interpretation or use depends on how well the evidence supports the
> claims being made. Because truth and evidence are distinct, this means
> that the validity of a test score interpretation could be high even
> though the interpretation is false. As an illustration, we discuss the
> case of phlogiston measurement as it existed in the 18th century. At
> face value, Kane's theory would seem to imply that interpretations of
> phlogiston measurement were valid in the 18th century (because the
> evidence for them was strong), even though amounts of phlogiston do not
> exist and hence cannot be measured. We suggest that this neglects an
> important aspect of validity and suggest various ways in which Kane's
> theory could meet this challenge.
>
> ========================================================================
>
>
> *Pages: 115-122 (Article)
> *View Full Record: http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=Alerting&SrcApp=Alerting&DestApp=CCC&DestLinkType=FullRecord;KeyUT=CCC:000316286300008
> *Order Full Text [ ]
>
> Title:
> Validation as a Pragmatic, Scientific Activity
>
> Authors:
> Kane, MT
>
> Source:
> *JOURNAL OF EDUCATIONAL MEASUREMENT*, 50 (1):115-122; SI SPR 2013
>
> Abstract:
> This response to the comments contains three main sections, each
> addressing a subset of the comments. In the first section, I will
> respond to the comments by Brennan, Haertel, and Moss. All of these
> comments suggest ways in which my presentation could be extended or
> improved; I generally agree with their suggestions, so my response to
> their comments is brief. In the second section, I will respond to
> suggestions by Newton and Sireci that my framework be simplified by
> employing only one kind of argument, a validity argument, and dropping
> the interpretation/use argument (IUA); I am sympathetic to their desire
> for greater simplicity, but I see considerable value in keeping the IUA
> as a framework for the validation effort and will argue for keeping both
> the IUA and the validity argument. In the third section, I will respond
> to Borsboom and Markus, who raise a fundamental objection to my approach
> to validation, suggesting that I give too much attention to
> justification and too little to truth as a criterion for validity; I
> don't accept their proposed conception of validity, and I will indicate
> why.
>