Fill in the BLANC: Human-free quality estimation of document summaries
| Autor: | Oleg V. Vasilyev, Vedant Dharnidharka, John Bohannon |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
FOS: Computer and information sciences
Estimation Measure (data warehouse) Language understanding Computer Science - Computation and Language business.industry Computer science media_common.quotation_subject computer.software_genre Task (project management) Fully automated Quality (business) Language model Artificial intelligence business Computation and Language (cs.CL) computer Document summary Natural language processing media_common |
| Zdroj: | Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems. |
| Popis: | We present BLANC, a new approach to the automatic estimation of document summary quality. Our goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. Our approach achieves this by measuring the performance boost gained by a pre-trained language model with access to a document summary while carrying out its language understanding task on the document's text. We present evidence that BLANC scores have as good correlation with human evaluations as do the ROUGE family of summary quality measurements. And unlike ROUGE, the BLANC method does not require human-written reference summaries, allowing for fully human-free summary quality estimation. Comment: 10 pages, 9 figures, 3 tables. In: Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP, Nov. 2020) p.11-20, ACL |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|