GO FIGURE: A Meta Evaluation of Factuality in Summarization
| Autor: | Asli Celikyilmaz, Yejin Choi, Rahul Jha, Jianfeng Gao, Saadia Gabriel |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Computation and Language Correctness business.industry Computer science InformationSystems_INFORMATIONSTORAGEANDRETRIEVAL computer.software_genre ComputingMethodologies_ARTIFICIALINTELLIGENCE Automatic summarization Field (computer science) Consistency (database systems) InformationSystems_MODELSANDPRINCIPLES Generalizability theory Artificial intelligence Source text Metric (unit) business Computation and Language (cs.CL) computer Natural language processing Reliability (statistics) |
| Zdroj: | ACL/IJCNLP (Findings) |
| Popis: | While neural language models can generate text with remarkable fluency and coherence, controlling for factual correctness in generation remains an open research question. This major discrepancy between the surface-level fluency and the content-level correctness of neural generation has motivated a new line of research that seeks automatic metrics for evaluating the factuality of machine text. In this paper, we introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. We propose five necessary and intuitive conditions to evaluate factuality metrics on diagnostic factuality data across three different summarization tasks. Our benchmark analysis on ten factuality metrics reveals that our meta-evaluation framework provides a robust and efficient evaluation that is extensible to multiple types of factual consistency and standard generation metrics, including QA metrics. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated. ACL 2021 Findings |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|