| Autor: |
Zheng, Yuanhang, Tan, Zhixing, Li, Peng, Liu, Yang |
| Rok vydání: |
2023 |
| Předmět: |
|
| Zdroj: |
IEEE/ACM Transactions on Audio, Speech, and Language Processing. 32 (2024) 3002-3013 |
| Druh dokumentu: |
Working Paper |
| DOI: |
10.1109/TASLP.2024.3407519 |
| Popis: |
Black-box prompt tuning employs derivative-free optimization algorithms to learn prompts within low-dimensional subspaces rather than back-propagating through the network of Large Language Models (LLMs). Recent studies reveal that black-box prompt tuning lacks versatility across tasks and LLMs, which we believe is related to the suboptimal choice of subspaces. In this paper, we introduce Black-box prompt tuning with Subspace Learning (BSL) to enhance the versatility of black-box prompt tuning. Based on the assumption that nearly optimal prompts for similar tasks reside in a common subspace, we propose identifying such subspaces through meta-learning on a collection of similar source tasks. Consequently, for a target task that shares similarities with the source tasks, we expect that optimizing within the identified subspace can yield a prompt that performs well on the target task. Experimental results confirm that our BSL framework consistently achieves competitive performance across various downstream tasks and LLMs. |
| Databáze: |
arXiv |
| Externí odkaz: |
|
