From characters to words: the turning point of BPE merges
| Autor: | Tanja Samardzic, Ximena Gutierrez-Vasques, Christian Bentz, Olga Sozinova |
|---|---|
| Rok vydání: | 2021 |
| Předmět: |
Computer science
business.industry Lexical analysis Orthographic projection computer.software_genre Feature (linguistics) Range (mathematics) Entropy (information theory) Probability distribution Turning point Artificial intelligence business computer Natural language processing Word (computer architecture) |
| Zdroj: | EACL Scopus-Elsevier |
| DOI: | 10.18653/v1/2021.eacl-main.302 |
| Popis: | The distributions of orthographic word types are very different across languages due to typological characteristics, different writing traditions and potentially other factors. The wide range of cross-linguistic diversity is still a major challenge for NLP and the study of language. We use BPE and information-theoretic measures to investigate if distributions become similar under specific levels of subword tokenization. We perform a cross-linguistic comparison, following incremental merges of BPE (we go from characters to words) for 47 diverse languages. We show that text entropy values (a feature of probability distributions) tend to converge at specific subword levels: relatively few BPE merges (around 350) lead to the most similar distributions across languages. Additionally, we analyze the interaction between subword and word-level distributions and show that our findings can be interpreted in light of the ongoing discussion regarding different types of morphological complexity. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|