Toward a unified approach to statistical language modeling for Chinese

Autor:	Mingjing Li, Kai-Fu Lee, Joshua T. Goodman, Jianfeng Gao
Rok vydání:	2002
Předmět:	Perplexity General Computer Science Computer science business.industry Text segmentation Pinyin computer.software_genre Lexicon Set (abstract data type) Trigram Language model Artificial intelligence business computer Word (computer architecture) Natural language processing
Zdroj:	ACM Transactions on Asian Language Information Processing. 1:3-33
ISSN:	1558-3430 1530-0226
DOI:	10.1145/595576.595578
Popis:	This article presents a unified approach to Chinese statistical language modeling (SLM). Applying SLM techniques like trigram language models to Chinese is challenging because (1) there is no standard definition of words in Chinese; (2) word boundaries are not marked by spaces; and (3) there is a dearth of training data. Our unified approach automatically and consistently gathers a high-quality training data set from the Web, creates a high-quality lexicon, segments the training data using this lexicon, and compresses the language model, all by using the maximum likelihood principle, which is consistent with trigram model training. We show that each of the methods leads to improvements over standard SLM, and that the combined method yields the best pinyin conversion result reported.
Databáze:	OpenAIRE
Externí odkaz:	https://explore.openaire.eu/search/publication?articleId=doi_________::c68999598eaae56d299c86512eaea972 https://doi.org/10.1145/595576.595578 Zobrazit plný text záznamu