An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction
| Autor: | Jason Mars, Jonathan K. Kummerfeld, Joseph Peper, Anish Mahendran, Andrew Lee, Lingjia Tang, Kevin Leach, Michael A. Laurenzano, Stefan Larson, Christopher Clarke, Parker Hill |
|---|---|
| Rok vydání: | 2019 |
| Předmět: |
FOS: Computer and information sciences
Computer Science - Machine Learning Information retrieval Computer Science - Computation and Language Computer science Computer Science - Artificial Intelligence Inference 020206 networking & telecommunications 02 engineering and technology Class (biology) Field (computer science) Machine Learning (cs.LG) Identification (information) Artificial Intelligence (cs.AI) 0202 electrical engineering electronic engineering information engineering Benchmark (computing) 020201 artificial intelligence & image processing Dialog box Computation and Language (cs.CL) |
| Zdroj: | EMNLP/IJCNLP (1) |
| DOI: | 10.48550/arxiv.1909.02027 |
| Popis: | Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope---i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems. Comment: Accepted to EMNLP-IJCNLP 2019 |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|