Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction

Autor:	Tan, Qingyu, Xu, Lu, Bing, Lidong, Ng, Hwee Tou, Aljunied, Sharifah Mahani
Rok vydání:	2022
Předmět:	Computer Science - Computation and Language Computer Science - Information Retrieval
Druh dokumentu:	Working Paper
Popis:	The DocRED dataset is one of the most popular and widely used benchmarks for document-level relation extraction (RE). It adopts a recommend-revise annotation scheme so as to have a large-scale annotated dataset. However, we find that the annotation of DocRED is incomplete, i.e., false negative samples are prevalent. We analyze the causes and effects of the overwhelming false negative problem in the DocRED dataset. To address the shortcoming, we re-annotate 4,053 documents in the DocRED dataset by adding the missed relation triples back to the original DocRED. We name our revised DocRED dataset Re-DocRED. We conduct extensive experiments with state-of-the-art neural models on both datasets, and the experimental results show that the models trained and evaluated on our Re-DocRED achieve performance improvements of around 13 F1 points. Moreover, we conduct a comprehensive analysis to identify the potential areas for further improvement. Our dataset is publicly available at https://github.com/tonytan48/Re-DocRED. Comment: Accepted by EMNLP 2022
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2205.12696 Zobrazit plný text záznamu View this record from Arxiv