Gå til søgefeltetSpring over til navigationSpring til hovedindhold

A Comprehensive Analysis of Adapter Efficiency.

  • Nandini Mundra
    ,
  • Sumanth Doddapaneni
    ,
  • Raj Dabre
    ,
  • Anoop Kunchukuttan
    ,
  • ,
  • Mitesh M. Khapra
  • Indian Institute of Technology Madras
    ,
  • National Institute Of Information And Communications Technology, Japan
    ,
  • Microsoft India
    ,
  • Agency for Science, Technology and Research (A*Star)
Publikation:
Konference artikel i Proceeding eller bog/rapport kapitel
Konferencebidrag i proceedings
Forskning
Peer-review

Open Access

Resume

Adapters have been positioned as a parameter-efficient fine-tuning (PEFT) approach, whereby a minimal number of parameters are added to the model and fine-tuned. However, adapters have not been sufficiently analyzed to understand if PEFT translates to benefits in training/deployment efficiency and maintainability/extensibility. Through extensive experiments on many adapters, tasks, and languages in supervised and cross-lingual zero-shot settings, we clearly show that for Natural Language Understanding (NLU) tasks, the parameter efficiency in adapters does not translate to efficiency gains compared to full fine-tuning of models. More precisely, adapters are relatively expensive to train and have slightly higher deployment latency. Furthermore, the maintainability/extensibility benefits of adapters can be achieved with simpler approaches like multi-task training via full fine-tuning, which also provide relatively faster training times. We, therefore, recommend that for moderately sized models for NLU tasks, practitioners should rely on full fine-tuning or multi-task training rather than using adapters. Our code is available at https://github.com/AI4Bharat/adapter-efficiency.

Publikation information

Produktionstype

Publikation:
Konference artikel i Proceeding eller bog/rapport kapitel
Konferencebidrag i proceedings
Forskning
Peer-review

Originalsprog

Engelsk

Sider fra-til (Antal sider)

Sider 136-154 (19 sider)

Publikationsmilepæle

  • Udgivet - 2024

Publikationsstatus

Udgivet - 2024

Forlag

Association for Computing Machinery, USA

Publikations IDer

  • Scopus: 85183581297

Titel på værtspublikation

CODS-COMAD '24: Proceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD)

Metrikker

PlumX, åbner i en ny fane

Hentninger
17
Antal citationer
13

Relateret event

Titel

International Conference on Data Science & Management of Data

Begivenhedstype

Konference

Grad af anerkendelse

International begivenhed

Dato

04/01/2024 - 07/01/2024

Lokation

BangaloreIndien