BERT Busters: Outlier Dimensions That Disrupt Transformers
- Olga Kovaleva,
- Saurabh Kulshreshtha,
- ,
- Anna Rumshisky
- University of Massachusetts
Research Output:
Conference Article in Proceeding or Book/Report chapter
Article in proceedings
Peer-reviewPublication Information
Output type
Research Output:
Conference Article in Proceeding or Book/Report chapter
Article in proceedings
Peer-reviewOriginal language
EnglishPages from-to (Number of pages)
Pages 3392-3405 (14 pages)Publication milestones
- Published - 01/08/2021
Publication status
Published - 01/08/2021
Place of publication
OnlinePublisher
Association for Computational Linguistics, United StatesPublication IDs
- Scopus: 85115709718
Host publication title
Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021Abstract
Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (
Publication metrics
PlumX
Citations
56
Captures
85
Related Event
Title
Findings of the Association for Computational Linguistics
Event type
ConferenceDate
01/08/2021 - 06/08/2021Location
VIRTUAL