Model Merging by Uncertainty-Based Gradient Matching

Autor:	Daheim, Nico, Möllenhoff, Thomas, Ponti, Edoardo Maria, Gurevych, Iryna, Khan, Mohammad Emtiyaz
Rok vydání:	2023
Předmět:	Computer Science - Machine Learning Computer Science - Artificial Intelligence Computer Science - Computation and Language
Druh dokumentu:	Working Paper
Popis:	Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging. Our new method gives consistent improvements for large language models and vision transformers, both in terms of performance and robustness to hyperparameters. Code available here. Comment: ICLR 2024; Code: https://github.com/UKPLab/iclr2024-model-merging
Databáze:	arXiv
Externí odkaz:	http://arxiv.org/abs/2310.12808 Zobrazit plný text záznamu View this record from Arxiv