Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Prajjwal Bhargava; Aleksandr Drozd; Anna Rogers

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Prajjwal Bhargava
, Aleksandr Drozd
, Anna Rogers

University of Texas
RIKEN Center for Computational Science
Tokyo Institute of Technology

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

Abstract

Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

Original language	English
Title of host publication	Proceedings of the Second Workshop on Insights from Negative Results in NLP
Number of pages	11
Place of Publication	Online and Punta Cana, Dominican Republic
Publisher	Association for Computational Linguistics
Publication date	1 Nov 2021
Pages	125-135
Publication status	Published - 1 Nov 2021
Event	Insights from Negative Results in NLP - VIRTUAL, Dominican Republic Duration: 1 Nov 2021 → … Conference number: 2

Conference

Conference	Insights from Negative Results in NLP
Number	2
Country/Territory	Dominican Republic
City	VIRTUAL
Period	01/11/2021 → …

Keywords

Natural Language Understanding
Generalization
BERT-based architectures
Adversarial robustness
Transformer models

Access to Document

https://aclanthology.org/2021.insights-1.18

Cite this

@inproceedings{40cdd8350fe947eba133d72a41aacf01,

title = "Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics",

abstract = "Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.",

keywords = "Natural Language Understanding, Generalization, BERT-based architectures, Adversarial robustness, Transformer models, Natural Language Understanding, Generalization, BERT-based architectures, Adversarial robustness, Transformer models",

author = "Prajjwal Bhargava and Aleksandr Drozd and Anna Rogers",

year = "2021",

month = nov,

day = "1",

language = "English",

pages = "125--135",

booktitle = "Proceedings of the Second Workshop on Insights from Negative Results in NLP",

publisher = "Association for Computational Linguistics",

address = "United States",

note = "Insights from Negative Results in NLP ; Conference date: 01-11-2021",

}

Bhargava, P, Drozd, A & Rogers, A 2021, Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics. in Proceedings of the Second Workshop on Insights from Negative Results in NLP. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, pp. 125-135, Insights from Negative Results in NLP, VIRTUAL, Dominican Republic, 01/11/2021. <https://aclanthology.org/2021.insights-1.18>

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics. / Bhargava, Prajjwal; Drozd, Aleksandr; Rogers, Anna.
Proceedings of the Second Workshop on Insights from Negative Results in NLP. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021. p. 125-135.

Research output: Conference Article in Proceeding or Book/Report chapter › Article in proceedings › Research › peer-review

TY - GEN

T1 - Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

AU - Bhargava, Prajjwal

AU - Drozd, Aleksandr

AU - Rogers, Anna

N1 - Conference code: 2

PY - 2021/11/1

Y1 - 2021/11/1

N2 - Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

AB - Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.

KW - Natural Language Understanding

KW - Generalization

KW - BERT-based architectures

KW - Adversarial robustness

KW - Transformer models

KW - Natural Language Understanding

KW - Generalization

KW - BERT-based architectures

KW - Adversarial robustness

KW - Transformer models

M3 - Article in proceedings

SP - 125

EP - 135

BT - Proceedings of the Second Workshop on Insights from Negative Results in NLP

PB - Association for Computational Linguistics

CY - Online and Punta Cana, Dominican Republic

T2 - Insights from Negative Results in NLP

Y2 - 1 November 2021

ER -

Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this