Evaluating spoken dialogue agents with PARADISE: Two case studies
| Autor: | Alicia Abella, Diane J. Litman, Marilyn A. Walker, Candace A. Kamm |
|---|---|
| Rok vydání: | 1998 |
| Předmět: |
business.industry
Computer science media_common.quotation_subject System evaluation computer.software_genre ComputingMethodologies_ARTIFICIALINTELLIGENCE Theoretical Computer Science Task (project management) Human-Computer Interaction Paradise Performance function Artificial intelligence business computer Software Natural language processing media_common |
| Zdroj: | Computer Speech & Language. 12:317-347 |
| ISSN: | 0885-2308 |
| DOI: | 10.1006/csla.1998.0110 |
| Popis: | This paper presents PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating and comparing the performance of spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviours, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to performance, and makes it possible to compare agents performing different taks by normalizing for task complexity. After presenting PARADISE, we illustrate its application to two different spoken dialogue agents. We show how to derive a performance function for each agent and how to generalize results across agents. We then show that once such a performance function has been derived, it can be used both for making predictions about future versions of an agent, and as feedback to the agent so that the agent can learn to optimize its behaviour based on its experiences with users over time. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|
Full Text from ScienceDirect