ChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning
| Autor: | Shao, Jie-Jing, Yang, Xiao-Wen, Zhang, Bo-Wen, Chen, Baizhi, Wei, Wen-Da, Cai, Guohao, Dong, Zhenhua, Guo, Lan-Zhe, Li, Yu-feng |
|---|---|
| Rok vydání: | 2024 |
| Předmět: | |
| Druh dokumentu: | Working Paper |
| Popis: | Recent advances in LLMs, particularly in language reasoning and tool integration, have rapidly sparked the real-world development of Language Agents. Among these, travel planning represents a prominent domain, combining academic challenges with practical value due to its complexity and market demand. However, existing benchmarks fail to reflect the diverse, real-world requirements crucial for deployment. To address this gap, we introduce ChinaTravel, a benchmark specifically designed for authentic Chinese travel planning scenarios. We collect the travel requirements from questionnaires and propose a compositionally generalizable domain-specific language that enables a scalable evaluation process, covering feasibility, constraint satisfaction, and preference comparison. Empirical studies reveal the potential of neuro-symbolic agents in travel planning, achieving a constraint satisfaction rate of 27.9%, significantly surpassing purely neural models at 2.6%. Moreover, we identify key challenges in real-world travel planning deployments, including open language reasoning and unseen concept composition. These findings highlight the significance of ChinaTravel as a pivotal milestone for advancing language agents in complex, real-world planning scenarios. Comment: Webpage: https://www.lamda.nju.edu.cn/shaojj/chinatravel |
| Databáze: | arXiv |
| Externí odkaz: |
|