Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings
Research Output:
Conference Article in Proceeding or Book/Report chapter
Article in proceedings
Research
Peer-reviewOpen access
Abstract
As input representation for each sub-word, the original BERT architecture proposes the sum of the sub-word embedding,<br/>position embedding and a segment embedding. Sub-word and position embeddings are well-known and studied, and<br/>encode lexical information and word position, respectively. In contrast, segment embeddings are less known and have so<br/>far received no attention. The key idea of segment embeddings is to encode to which of the two sentences (segments)<br/>a word belongs—the intuition is to inform the model about the separation of sentences for the next sentence prediction<br/>pre-training task. However, little is known on whether the choice of segment impacts downstream prediction performance.<br/>In this work, we try to fill this gap and empirically study the impact of alternating the segment embedding during inference<br/>time for a variety of pre-trained embeddings and target tasks. We hypothesize that for single-sentence prediction tasks<br/>performance is not affected—neither in mono- nor multilingual setups—while it matters when changing the segment IDs<br/>in paired-sentence tasks. To our surprise, this is not the case. Although for classification tasks and monolingual BERT<br/>models no large differences are observed, particularly word-level multilingual prediction tasks are heavily impacted. For<br/>low-resource syntactic tasks, we observe impacts of segment embedding and multilingual BERT choice. We find that<br/>the default setting for the most used multilingual BERT model underperforms heavily, and a simple swap of the segment<br/>embeddings yields an average improvement of 2.5 points absolute LAS score for dependency parsing over 9 different treebanks.
Publication Information
Output type
Research Output:
Conference Article in Proceeding or Book/Report chapter
Article in proceedings
Research
Peer-reviewOriginal language
EnglishPages from-to (Number of pages)
Pages 1418-1427Publication milestones
- Published - 2022
Publication status
Published - 2022
Publication IDs
- Scopus: 85144366424
Host publication title
Proceedings of the Language Resources and Evaluation ConferencePublication metrics
PlumX
Citation count
4
Access to documents
Final published version
License:CC BY-NC-SA, opens in new tab
Related Event
Title
Conference on Linguistic Resources and Evaluation
Event type
ConferenceDegree of recognition
International eventDate
20/06/2022 - 25/06/2022Location
Palais du PharoMarseilleFrance
