On the Performance of the Marginal Homogeneity Test to Detect Rater Drift
| Autor: | Adrienne Sgammato, John R. Donoghue |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2017 |
| Předmět: |
Empirical data
05 social sciences 050401 social sciences methods Marginal homogeneity Articles 01 natural sciences Test (assessment) 010104 statistics & probability Inter-rater reliability 0504 sociology Constructed response Sample size determination Statistics Econometrics Psychology (miscellaneous) 0101 mathematics Psychology Social Sciences (miscellaneous) Student's t-test Type I and type II errors |
| Popis: | When constructed response items are administered repeatedly, “trend scoring” can be used to test for rater drift. In trend scoring, raters rescore responses from the previous administration. Two simulation studies evaluated the utility of Stuart’s Q measure of marginal homogeneity as a way of evaluating rater drift when monitoring trend scoring. In the first study, data were generated based on trend scoring tables obtained from an operational assessment. The second study tightly controlled table margins to disentangle certain features present in the empirical data. In addition to Q, the paired t test was included as a comparison, because of its widespread use in monitoring trend scoring. Sample size, number of score categories, interrater agreement, and symmetry/asymmetry of the margins were manipulated. For identical margins, both statistics had good Type I error control. For a unidirectional shift in margins, both statistics had good power. As expected, when shifts in the margins were balanced across categories, the t test had little power. Q demonstrated good power for all conditions and identified almost all items identified by the t test. Q shows substantial promise for monitoring of trend scoring. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|