A Primer in BERTology: What We Know About How BERT Works
- ,
- Olga Kovaleva,
- Anna Rumshisky
- University of Massachusetts
Research Output:
Journal Article or Conference Article in Journal
Journal article
Peer-reviewPublication Information
Output type
Research Output:
Journal Article or Conference Article in Journal
Journal article
Peer-reviewOriginal language
EnglishPages from-to (Number of pages)
Pages 842-866 (25 pages)Journal (Volume, Issue Number)
Transactions of the Association for Computational Linguistics (Volume 8)Publication milestones
- Published - 01/12/2020
Publication status
Published - 01/12/2020
ISSN
2307-387XPublication IDs
- Scopus: 85098839172
Abstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.
Publication metrics
PlumX
Mentions
15
Citations
1331
Captures
2030