Influence of external peer reviewer scores for funding applications on funding board decisions: A retrospective analysis of 1561 reviews
| Autor: | Taeko Becque, Lexy Sorrell, Helen Payne, Beth Stuart, Jeremy C Wyatt, Nicola Mcardle, Sheila Turner |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2018 |
| Předmět: |
medicine.medical_specialty
National Health Programs Decision Making research funding 03 medical and health sciences 0302 clinical medicine Research Support as Topic medicine Retrospective analysis score 030212 general & internal medicine Retrospective Studies Receiver operating characteristic business.industry Research Outcome measures General Medicine United Kingdom On board Family medicine business 030217 neurology & neurosurgery |
| Zdroj: | BMJ Open |
| Popis: | ObjectivesTo evaluate the influence of external peer reviewer scores on the National Institute for Health Research (NIHR) research funding board decisions by the number of reviewers and type of reviewer expertise.DesignRetrospective analysis of external peer review scores for shortlisted full applications for funding (280 funding applications, 1236 individual reviewers, 1561 review scores).SettingFour applied health research funding programmes of NIHR, UK.Main outcome measuresBoard decision to fund or not fund research applications.ResultsThe mean score of reviewers predicted funding decisions better than individual reviewer scores (area under the receiver operating characteristic (ROC) curve 0.75, 95% CI 0.69 to 0.81 compared with 0.62, CI 0.59 to 0.65). There was no substantial improvement in how accurately mean reviewer scores predicted funding decisions when the number of reviewers increased above 4 (area under ROC curve 0.75, CI 0.59 to 0.91 for four reviewers; 0.80, CI 0.67 to 0.92 for seven or more). Reviewers with differing expertise influenced the board’s decision equally, including public and patient reviewers (area under ROC curves from 0.57, CI 0.47 to 0.66 for health economists to 0.64, CI 0.57 to 0.70 for subject-matter experts). The areas under the ROC curves were quite low when using reviewers’ scores, confirming that boards do not rely solely on those scores alone to make their funding decisions, which are best predicted by the mean board score.ConclusionsBoards value scores that originate from a diverse pool of reviewers. On the basis of independent reviewer score alone, there is no detectable benefit of using more than four reviewer scores in terms of their influence on board decisions, so to improve efficiency, it may be possible to avoid using larger numbers of reviewers. The funding decision is best predicted by the board score. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|