| Popis: |
Davis-Stober and Regenwetter (2019; Psychological Review) discussed the ‘paradox’ of converging evidence, whereby, with more and more positive Cohen's d values across multiple studies, support for a theory does not accumulate. Instead, more and more people may be exceptions to the theory. Using a psychometric framework, Heck (2021; Psychological Review) argued that Davis-Stober and Regenwetter's worst case scenarios are too pessimistic. His upwards-adjusted lower bounds on the number of people who satisfy multiple predictions of a theory jointly only occur when true score distributions are equally, and maximally, negative correlated across conditions. We show that Heck's conclusions hinge on untestable auxiliary assumptions. If one drops those assumptions, then the lower bounds of Davis-Stober and Regenwetter (2019) are attainable for any combination of effect sizes and number of predictions - and can still occur even when correlations across predictions are no longer all negative. Our arguments point to larger issues in quantitative psychology where seemingly innocuous modeling assumptions can unintentionally rule out empirically possible, as well as entirely plausible, outcomes. Even with generously large Cohen's d values and generously high correlations among individuals across effects, the proportion of the population that satisfies a theory quickly becomes smaller as of the number of predictions increases, under both frameworks. Said simply, the 'paradox' does not dissipate in either framework. |
