Mini but Mighty: Finetuning ViTs with Mini Adapters - Télécom Paris Accéder directement au contenu
Proceedings/Recueil Des Communications Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops Année : 2024

Mini but Mighty: Finetuning ViTs with Mini Adapters

Résumé

Vision Transformers (ViTs) have become one of the dominant architectures in computer vision, and pre-trained ViT models are commonly adapted to new tasks via fine-tuning. Recent works proposed several parameter-efficient transfer learning methods, such as adapters, to avoid the prohibitive training and storage cost of finetuning. In this work, we observe that adapters perform poorly when the dimension of adapters is small, and we propose MiMi, a training framework that addresses this issue. We start with large adapters which can reach high performance, and iteratively reduce their size. To enable automatic estimation of the hidden dimension of every adapter, we also introduce a new scoring function, specifically designed for adapters, that compares the neuron importance across layers. Our method outperforms existing methods in finding the best trade-off between accuracy and trained parameters across the three dataset benchmarks DomainNet, VTAB, and Multi-task, for a total of 29 datasets.

Dates et versions

hal-04365041 , version 1 (27-12-2023)

Identifiants

Citer

Imad Eddine Marouf, Enzo Tartaglione, Stéphane Lathuilière. Mini but Mighty: Finetuning ViTs with Mini Adapters. 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, IEEE, pp.1721-1730, 2024. ⟨hal-04365041⟩
10 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More