Mapping Languages: The Corpus of Global Language Use
| Autor: | Jonathan Dunn |
|---|---|
| Jazyk: | angličtina |
| Rok vydání: | 2020 |
| Předmět: |
FOS: Computer and information sciences
050101 languages & linguistics Linguistics and Language Language identification Arabic Computer science 02 engineering and technology Representation (arts) Library and Information Sciences computer.software_genre Language and Linguistics Education 0202 electrical engineering electronic engineering information engineering 0501 psychology and cognitive sciences Computer Science - Computation and Language business.industry 05 social sciences Variety (linguistics) language.human_language Focus (linguistics) language 020201 artificial intelligence & image processing Artificial intelligence Computational linguistics business computer On Language Computation and Language (cs.CL) Natural language processing |
| Popis: | This paper describes a web-based corpus of global language use with a focus on how this corpus can be used for data-driven language mapping. First, the corpus provides a representation of where national varieties of major languages are used (e.g., English, Arabic, Russian) together with consistently collected data for each variety. Second, the paper evaluates a language identification model that supports more local languages with smaller sample sizes than alternative off-the-shelf models. Improved language identification is essential for moving beyond majority languages. Given the focus on language mapping, the paper analyzes how well this digital language data represents actual populations by (i) systematically comparing the corpus with demographic ground-truth data and (ii) triangulating the corpus with an alternate Twitter-based dataset. In total, the corpus contains 423 billion words representing 148 languages (with over 1 million words from each language) and 158 countries (again with over 1 million words from each country), all distilled from Common Crawl web data. The main contribution of this paper, in addition to describing this publicly-available corpus, is to provide a comprehensive analysis of the relationship between two sources of digital data (the web and Twitter) as well as their connection to underlying populations. This is a pre-print of an article published in Language Resources and Evaluation. The final authenticated version is available online at: https://doi.org/10.1007/s10579-020-09489-2 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|