ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews
| Autor: | D'Arcy, Mike, Ross, Alexis, Bransom, Erin, Kuehl, Bailey, Bragg, Jonathan, Hope, Tom, Downey, Doug |
|---|---|
| Rok vydání: | 2023 |
| Předmět: | |
| Druh dokumentu: | Working Paper |
| Popis: | We introduce the task of automatically revising scientific papers based on peer feedback and release ARIES, a dataset of review comments and their corresponding paper edits. The data is drawn from real reviewer-author interactions from computer science, and we provide labels linking each reviewer comment to the specific paper edits made by the author in response. We automatically create a high-precision silver training set, as well as an expert-labeled test set that shows high inter-annotator agreement. In experiments with 10 models covering the state of the art, we find that they struggle even to identify which edits correspond to a comment -- especially when the relationship between the edit and the comment is indirect and requires reasoning to uncover. We also extensively analyze GPT-4's ability to generate edits given a comment and the original paper. We find that it often succeeds on a superficial level, but tends to rigidly follow the wording of the feedback rather than the underlying intent, and lacks technical details compared to human-written edits. Comment: ACL 2024, 10 pages, 2 figures |
| Databáze: | arXiv |
| Externí odkaz: |
|