What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability

In Natural Language Generation (NLG) tasks,
for any input, multiple communicative goals are
plausible, and any goal can be put into words,
or produced, in multiple ways. We characterise
the extent to which human production varies
lexically, syntactically, and semantically across
four NLG tasks, connecting human production
variability to aleatoric or data uncertainty. We
then inspect the space of output strings shaped
by a generation system’s predicted probability
distribution and decoding algorithm to probe
its uncertainty. For each test input, we measure
the generator’s calibration to human production
variability. Following this instance-level ap-
proach, we analyse NLG models and decoding
strategies, demonstrating that probing a genera-
tor with multiple samples and, when possible,
multiple references, provides the level of detail
necessary to gain understanding of a model’s
representation of uncertainty.