An Empirical Analysis of BERT Embedding for Automated Essay Scoring
| Autor: | Saleh Alzahrani, Majdi Beseiso |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
General Computer Science
Artificial neural network Computer science business.industry Deep learning 05 social sciences 050301 education 02 engineering and technology Automated essay scoring computer.software_genre Field (computer science) Spelling Cohen's kappa 0202 electrical engineering electronic engineering information engineering Feature (machine learning) 020201 artificial intelligence & image processing Language model Artificial intelligence Representation (mathematics) business 0503 education computer Natural language processing |
| Zdroj: | International Journal of Advanced Computer Science and Applications. 11 |
| ISSN: | 2156-5570 2158-107X |
| DOI: | 10.14569/ijacsa.2020.0111027 |
| Popis: | Automated Essay Scoring (AES) is one of the most challenging problems in Natural Language Processing (NLP). The significant challenges include the length of the essay, the presence of spelling mistakes affecting the quality of the essay and representing essay in terms of relevant features for the efficient scoring of essays. In this work, we present a comparative empirical analysis of Automatic Essay Scoring (AES) models based on combinations of various feature sets. We use 30-manually extracted features, 300-word2vec representation, and 768-word embedding features using BERT model and forms different combinations for evaluating the performance of AES models. We formulate an automated essay scoring problem as a rescaled regression problem and quantized classification problem. We analyzed the performance of AES models for different combinations. We compared them against the existing ensemble approaches in terms of Kappa Statistics and Accuracy for rescaled regression problem and quantized classification problem respectively. A combination of 30-manually extracted features, 300-word2vec representation, and 768-word embedding features using BERT model results up to 77.2 ± 1.7 of Kappa statistics for rescaled regression problem and 75.2 ± 1.0 of accuracy value for Quantized Classification problem using a benchmark dataset consisting of about 12,000 essays divided into eight groups. The reporting results provide directions to the researchers in the field to use manually extracted features along with deep encoded features for developing a more reliable AES model. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|