Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID events.

Autor: Gruen A; The Florey Institute of Neuroscience and Mental Health, Melbourne, Australia., Mattingly KR; Dysrupt Labs by SlowVoice, Melbourne, Australia., Morwitch E; The Florey Institute of Neuroscience and Mental Health, Melbourne, Australia., Bossaerts F; Dysrupt Labs by SlowVoice, Melbourne, Australia., Clifford M; Boston Consulting Group, Melbourne, Australia., Nash C; Dysrupt Labs by SlowVoice, Melbourne, Australia., Ioannidis JPA; Stanford Prevention Research Center, Department of Medicine, and Departments of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Meta-Research Innovation Center at Stanford, Stanford, CA, USA., Ponsonby AL; The Florey Institute of Neuroscience and Mental Health, Melbourne, Australia; Murdoch Children's Research Institute, Royal Children's Hospital, Melbourne, Australia; Centre of Epidemiology and Biostatistics, School of Population and Global Health, University of Melbourne, Australia. Electronic address: annelouise.ponsonby@florey.edu.au.
Jazyk: angličtina
Zdroj: EBioMedicine [EBioMedicine] 2023 Oct; Vol. 96, pp. 104783. Date of Electronic Publication: 2023 Sep 12.
DOI: 10.1016/j.ebiom.2023.104783
Abstrakt: Background: The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for likely accuracy in real-time with machine learning.
Methods: We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n = 1822) and Next Generation Social Science (NGS2) platform (n = 103) were utilised.
Findings: A 43-feature model predicted accurate forecasters, those with top quintile relative Brier accuracy, with subsequent replication in two out-of-sample datasets (p both <1 × 10 -9 ). Trades graded by this model as having higher accuracy scores than others produced a greater AUC temporal gain in the overall market after vs before trade. Accuracy score-weighted forecasts had higher accuracy than market forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial % AUC gains of 13.2%, p = 1.35 × 10 -14 and 13.8%, p = 0.003 in two out-of-sample datasets. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for market models, p = 0.007. This net classification benefit was replicated in the separate Almanis B dataset, p = 2.4 × 10 -7 .
Interpretation: Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. This could provide improved anticipation of and thus response to emerging risks.
Funding: This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources, Australia, to SlowVoice Pty Ltd. (IR 2101990) and Fellowship (GNT 1110200) and Investigator grant (GNT 1197234) to A-L Ponsonby by the National Health and Medical Research Council of Australia.
Competing Interests: Declaration of interests Dysrupt Labs, a subsidiary of SlowVoice Pty Ltd, supplied the Almanis prediction market database for this research. KM, FB, and CN are employees of Dysrupt Labs. KM, FB, CN, and ALP have stocks in Dysrupt Labs. AG, EM, MC and JPAI have no conflicts of interest.
(Copyright © 2023 The Authors. Published by Elsevier B.V. All rights reserved.)
Databáze: MEDLINE
Externí odkaz: