HEvaS: Headline Evaluation System
| Autor: | Marina Litvak, Itzhak Eretz Kdosha, Natalia Vanetik |
|---|---|
| Rok vydání: | 2018 |
| Předmět: |
Information retrieval
ComputingMilieux_THECOMPUTINGPROFESSION Semantics (computer science) Computer science media_common.quotation_subject Search engine indexing Headline 02 engineering and technology Automatic summarization Set (abstract data type) 020204 information systems 0202 electrical engineering electronic engineering information engineering Task analysis Quality (business) Baseline (configuration management) media_common |
| Zdroj: | WI |
| DOI: | 10.1109/wi.2018.00-16 |
| Popis: | Automatic headline generation is a sub-task of oneline summarization with many reported applications. Evaluation of systems generating headlines is a very challenging and undeveloped area. In this paper, we introduce a system that performs automatic evaluation of systems in terms of a quality of the generated headlines. The evaluation is performed using multiple metrics for comparing evaluated headlines with the gold standard ones or measuring their coverage of main document topics. Both types of metrics evaluate headline's content and informativeness, but not grammatical structure. The only input required by our system is a set of documents with gold standard and automatically generated headlines. The Headline Evaluation System (HEvaS) provides a user with a choice from multiple (10) metrics, then calculates the chosen metrics, performs statistical analysis of the evaluated systems and visualizes the results. Multiple headline generation systems can be evaluated at the same run. This paper describes all evaluation metrics and architecture, utilized by our system. As an evaluation of the HEvaS, we perform a case study with a couple of baseline systems and report the results. Although we tested the system on English content only, the multilingual content can also be supported. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|