Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing

Autor:	Shi, Freda, Gimpel, Kevin, Livescu, Karen
Rok vydání:	2024
Předmět:	Computer Science - Computation and Language Computer Science - Data Structures and Algorithms
Druh dokumentu:	Working Paper
Popis:	We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers. STRUCT-IOU enables comparison between a constituency parse tree (over automatically recognized spoken word boundaries) with the ground-truth parse (over written words). To compute the metric, we project the ground-truth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU can address token-mismatch issues, and shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991). Comment: ACL 2024 camera-ready
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2402.13433 Zobrazit plný text záznamu View this record from Arxiv