Estimating the deep replicability of scientific findings using human and artificial intelligence
Autor: | Yang Yang, Brian Uzzi, Wu Youyou |
---|---|
Rok vydání: | 2020 |
Předmět: |
Persuasion
Computer science media_common.quotation_subject Social Sciences 02 engineering and technology 050105 experimental psychology Task (project management) Machine Learning 020204 information systems Replication (statistics) 0202 electrical engineering electronic engineering information engineering Humans Psychology 0501 psychology and cognitive sciences Generalizability theory Human resources Set (psychology) media_common Multidisciplinary business.industry 05 social sciences Novelty Reproducibility of Results Computational sociology Artificial intelligence Periodicals as Topic business |
Zdroj: | Proc Natl Acad Sci U S A |
ISSN: | 1091-6490 0027-8424 |
Popis: | Replicability tests of scientific papers show that the majority of papers fail replication. Moreover, failed papers circulate through the literature as quickly as replicating papers. This dynamic weakens the literature, raises research costs, and demonstrates the need for new approaches for estimating a study’s replicability. Here, we trained an artificial intelligence model to estimate a paper’s replicability using ground truth data on studies that had passed or failed manual replication tests, and then tested the model’s generalizability on an extensive set of out-of-sample studies. The model predicts replicability better than the base rate of reviewers and comparably as well as prediction markets, the best present-day method for predicting replicability. In out-of-sample tests on manually replicated papers from diverse disciplines and methods, the model had strong accuracy levels of 0.65 to 0.78. Exploring the reasons behind the model’s predictions, we found no evidence for bias based on topics, journals, disciplines, base rates of failure, persuasion words, or novelty words like “remarkable” or “unexpected.” We did find that the model’s accuracy is higher when trained on a paper’s text rather than its reported statistics and that n-grams, higher order word combinations that humans have difficulty processing, correlate with replication. We discuss how combining human and machine intelligence can raise confidence in research, provide research self-assessment techniques, and create methods that are scalable and efficient enough to review the ever-growing numbers of publications—a task that entails extensive human resources to accomplish with prediction markets and manual replication alone. |
Databáze: | OpenAIRE |
Externí odkaz: |