| Popis: |
Over the last decade, psychology has started to question the validity of the statistical inferences drawn from many studies, partly due to the problem of inflated false positive rates. In the present study, we explored the influence of undisclosed flexibility in data analysis on one of the most popular dependent measures in psychology, namely, reaction times (RT). RTs are somewhat special because they entail some degrees of freedom of their own, mainly due to their non-normal distribution and the presence of outliers. Moreover, these degrees of freedom are usually not considered part of the analysis itself, but data preprocessing steps that must be contingent on the recorded data. We analyzed the impact of these degrees of freedom on the false positive rate using simulations over real and simulated data. Our results show that the false positive rate is inflated when instead of having a clearly defined a priori preprocessing pipeline, the preprocessing of RTs is modified after seeing the data or after several analyses are run. When several central-tendency measures, data transformations or outlier rejection techniques are combined in different ways, the false positive rate can rise up to 17%. This figure, worrying by itself, becomes more concerning when we consider that even more degrees of freedom are awaiting down the analysis pipeline, potentially making the final false positive rate much higher. We also propose an alternative method that reduces the need of preprocessing decisions by using a statistical approach based on permutations that do not require any particular underlying distribution. |
