InteractEva: A Simulation-Based Evaluation Framework for Interactive AI Systems
| Autor: | Yannis Katsis, Maeda F. Hanafi, Martín Santillán Cooper, Yunyao Li |
|---|---|
| Rok vydání: | 2022 |
| Předmět: | |
| Zdroj: | Proceedings of the AAAI Conference on Artificial Intelligence. 36:13182-13184 |
| ISSN: | 2374-3468 2159-5399 |
| DOI: | 10.1609/aaai.v36i11.21721 |
| Popis: | Evaluating interactive AI (IAI) systems is a challenging task, as their output highly depends on the performed user actions. As a result, developers often depend on limited and mostly qualitative data derived from user testing to improve their systems. In this paper, we present InteractEva; a systematic evaluation framework for IAI systems. InteractEva employs (a) a user simulation backend to test the system against different use cases and user interactions at scale with (b) an interactive frontend allowing developers to perform important quantitative evaluation tasks, including acquiring a performance overview, performing error analysis, and conducting what-if studies. The framework has supported the evaluation and improvement of an industrial IAI text extraction system, results of which will be presented during our demonstration. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|