ERASER: A Benchmark to Evaluate Rationalized NLP Models
| Autor: | Nazneen Fatema Rajani, Richard Socher, Jay DeYoung, Caiming Xiong, Byron C. Wallace, Eric Lehman, Sarthak Jain |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Computer Science - Computation and Language Computer Science - Artificial Intelligence Computer science business.industry 02 engineering and technology 010501 environmental sciences computer.software_genre 01 natural sciences Machine Learning (cs.LG) Documentation Artificial Intelligence (cs.AI) 0202 electrical engineering electronic engineering information engineering Benchmark (computing) Code (cryptography) 020201 artificial intelligence & image processing Artificial intelligence business computer Computation and Language (cs.CL) Natural language processing 0105 earth and related environmental sciences |
| Zdroj: | ACL |
| DOI: | 10.48550/arxiv.1911.03429 |
| Popis: | State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP that reveal the `reasoning' behind model outputs. But work in this direction has been conducted on different datasets and tasks with correspondingly unique aims and metrics; this makes it difficult to track progress. We propose the Evaluating Rationales And Simple English Reasoning (ERASER) benchmark to advance research on interpretable models in NLP. This benchmark comprises multiple datasets and tasks for which human annotations of "rationales" (supporting evidence) have been collected. We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i.e., the degree to which provided rationales influenced the corresponding predictions). Our hope is that releasing this benchmark facilitates progress on designing more interpretable NLP systems. The benchmark, code, and documentation are available at https://www.eraserbenchmark.com/ Comment: Accepted as a long paper to ACL2020 Website and leaderboard available at http://www.eraserbenchmark.com/ Code available at https://github.com/jayded/eraserbenchmark |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|