Skip to search boxSkip to navigationSkip to main content

Frustratingly Easy Performance Improvements for Low-resource Setups: A Tale on BERT and Segment Embeddings

Research Output:
Conference Article in Proceeding or Book/Report chapter
Article in proceedings
Research
Peer-review

Open access

Abstract

As input representation for each sub-word, the original BERT architecture proposes the sum of the sub-word embedding,<br/>position embedding and a segment embedding. Sub-word and position embeddings are well-known and studied, and<br/>encode lexical information and word position, respectively. In contrast, segment embeddings are less known and have so<br/>far received no attention. The key idea of segment embeddings is to encode to which of the two sentences (segments)<br/>a word belongs—the intuition is to inform the model about the separation of sentences for the next sentence prediction<br/>pre-training task. However, little is known on whether the choice of segment impacts downstream prediction performance.<br/>In this work, we try to fill this gap and empirically study the impact of alternating the segment embedding during inference<br/>time for a variety of pre-trained embeddings and target tasks. We hypothesize that for single-sentence prediction tasks<br/>performance is not affected—neither in mono- nor multilingual setups—while it matters when changing the segment IDs<br/>in paired-sentence tasks. To our surprise, this is not the case. Although for classification tasks and monolingual BERT<br/>models no large differences are observed, particularly word-level multilingual prediction tasks are heavily impacted. For<br/>low-resource syntactic tasks, we observe impacts of segment embedding and multilingual BERT choice. We find that<br/>the default setting for the most used multilingual BERT model underperforms heavily, and a simple swap of the segment<br/>embeddings yields an average improvement of 2.5 points absolute LAS score for dependency parsing over 9 different treebanks.

Publication Information

Output type

Research Output:
Conference Article in Proceeding or Book/Report chapter
Article in proceedings
Research
Peer-review

Original language

English

Pages from-to (Number of pages)

Pages 1418-1427

Publication milestones

  • Published - 2022

Publication status

Published - 2022

Publication IDs

  • Scopus: 85144366424

Host publication title

Proceedings of the Language Resources and Evaluation Conference

Publication metrics

PlumX

Citation count
4

Related Event

Title

Conference on Linguistic Resources and Evaluation

Event type

Conference

Degree of recognition

International event

Date

20/06/2022 - 25/06/2022

Location

Palais du PharoMarseilleFrance