A Stochastic Arabic Diacritizer Based on a Hybrid of Factorized and Unfactorized Textual Features
| Autor: | Sherif M. Abdou, Mohamed Al-Badrashiny, Mohamed S. Attia, Mohsen A. Rashwan, Ahmed Rafea |
|---|---|
| Rok vydání: | 2011 |
| Předmět: |
Acoustics and Ultrasonics
Computer science business.industry Speech recognition Phonetic transcription Word error rate Statistical model Speech synthesis computer.software_genre Arabic diacritics Morpheme Hybrid system Language model Artificial intelligence Electrical and Electronic Engineering business computer Natural language processing |
| Zdroj: | IEEE Transactions on Audio, Speech, and Language Processing. 19:166-175 |
| ISSN: | 1558-7924 1558-7916 |
| Popis: | This paper introduces a large-scale dual-mode stochastic system to automatically diacritize raw Arabic text. The first of these modes determines the most likely diacritics by choosing the sequence of full-form Arabic word diacritizations with maximum marginal probability via A^ lattice search and long-horizon n-grams probability estimation. When full-form words are OOV, the system switches to the second mode which factorizes each Arabic word into all its possible morphological constituents, then uses also the same techniques used by the first mode to get the most likely sequence of morphemes, hence the most likely diacritization. While the second mode achieves a far better coverage of the highly derivative and inflective Arabic language, the first mode is faster to learn, i.e., yields better disambiguation results for the same size of training corpora, especially for inferring syntactical (case-ending) diacritics. Our presented hybrid system that benefits from the advantages of both modes has experimentally been found superior to the best performing reported systems of Habash and Rambow, and of Zitouni, using the same training and test corpus for the sake of fair comparison. The word error rates of (morphological diacritization, overall diacritization including the case endings) for the three systems are, respectively, as follows (3.1%, 12.5%), (5.5%, 14.9%), and (7.9%, 18%). The hybrid architecture of language factorizing and unfactorizing components may be inspiring to other NLP/HLT problems in analogous situations. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|