This work explores an important dimension of variation in the language used by dialogue participants: their age. While previous work showed differences at various linguistic levels between age groups when experimenting with written discourse data (e.g., blog posts), previous work on dialogue has largely been limited to acoustic information related to voice and prosody. Detecting fine-grained linguistic properties of human dialogues is of crucial importance for developing AIbased conversational systems which are able to adapt to their human interlocutors. We therefore investigate whether, and to what extent, current text-based NLP models can detect such linguistic differences, and what the features driving their predictions are. We show that models achieve a fairly good performance on agegroup prediction, though the task appears to be more challenging compared to discourse. Through in-depth analysis of the best models' errors and the most predictive cues, we show that, in dialogue, differences among age groups mostly concern stylistic and lexical choices. We believe these findings can inform future work on developing controlled generation models for adaptive conversational systems.
Research on developing conversational agents has
experienced impressive progress, particularly in
recent years
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). age 19-29 A: oh that’s coolaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
B: different sights and stuff A: oh age 50+ A: well quite and I’d have to come back as wellaaaaaaaa B: that’s of course
A: and make up for you know
of a particular individual or group of users
continue to pose more of a challenge. Recent
examples of this line of research include adaptation at
style level
Personalised interaction is of crucial importance
to obtain systems that can be trusted by users and
perceived as natural
In this work, we focus on one particular
aspect that may influence conversational agent
success: user age profile. We investigate whether
the linguistic behaviour of conversational
participants differs across age groups using
state-of-theart NLP models on purely textual data, without
considering vocal cues. We aim to detect age from
characteristics of language use and adapt to this
signal, rather than work from ground-truth
metadata about user demographics. This is in the
interest of preserving privacy, and from the perspective
that while age and language use may have a
relationship, this will not be linear
Previous work on age detection in dialogue has focused on speech features, which are known to systematically vary across age groups. For example, Wolters et al. (2009) learn logistic regression age classifiers from a small dialogue dataset using different acoustic cues supplemented with a small set of hand-crafted lexical features, while Li et al. (2013) develop SVM classifiers using acoustic and prosodic features extracted from scripted utterances spoken by participants interacting with an artificial system. In contrast to this line of work, we investigate whether different age groups can be detected from textual linguistic information rather than voice-related cues. We explore whether, and to what extent, various state-of-the-art NLP models are able to capture such differences in dialogue data as a preliminary step to age-group adaptation by conversational agents.
We build on the work of Schler et al. (2006), who focus on age detection in written discourse using a corpus of blog posts. The authors learn a Multi-Class Real Winnow classifier leveraging a set of pre-determined style- and content-based features, including part-of-speech categories, function words, and the 1000 unigrams with the highest information gain in the training set. They ifnd that content features (lexical unigrams) yield higher accuracy (74%) than style features (72%), while their best results (76.2%) are obtained with their combination. We extend this investigation in several key ways: (1) we leverage state-of-the-art NLP models that allow us to learn representations end-to-end, without the need to specify concrete features in advance; (2) we apply this approach to dialogue data, using a large-scale dataset of transcribed, spontaneous open-domain dialogues, and also use this approach to replicate the experiments of Schler et al. (2006) on disccourse; (3) we show that text-based models can indeed detect age-related differences, even in the case of very sparse signal at the level of dialogue utterances; and finally (4) we carry out an in-depth analysis of the models’ predictions to gain insight on which elements of language use are most informative.1
Our work can be considered a first step toward
the modeling of age-related linguistic adaptation
by AI conversational systems. In particular, our
results can inform future work on controlled text
generation for dialogue agents
1Code and data available at: https://github.com/
lennertjansen/detecting-age-in-dialogue
age #samples #tokens mean L (± sd) min-max L
19-29 33,641 381,195 1-423
50+ 33,641 406,157 1-1246
We use a dataset of dialogue data where
information about the age of the speakers involved in the
conversation is available (see the dialogue
snippets in Figure 1), i.e., the spoken partition of the
British National Corpus
We focus on conversations that took place between two interlocutors, and only consider dialogues between people of the same age group. We then restrict our investigation to a binary opposition: younger vs. older age group. We split the dialogues into their constituent utterances (e.g., from each dialogue snippet in Figure 1 we extract three utterances), and further pre-process them by removing non-alphabetical characters. Only samples which are not empty after pre-processing are kept. For the younger group, we consider the 19-29 bracket, which contains 138,662 utterances. For the older, we group conversations from vfie brackets: 50-59, 60-69, 70-79, 80-89, and 90-99 (hence, 50+), which sums up to a total of 33,641 utterances. The choice of grouping these brackets is a trade-off between experimenting with fairly distinct age groups (the age difference between them is at least 20 years) and obtaining largeenough data for each of them.
We randomly sample 33,614 utterances from the 19-29 group in order to experiment with a balanced number of samples per group. The resulting dataset, that we use for our experiments, includes around 67K utterances with an average length of 11.7 tokens. Descriptive statistics are in Table 1.
We frame the problem as a binary classification task: given some text, we seek to predict whether the age class of its speaker is younger or older. 3.1
We experiment with various models, that we
briefly describe below. Details on model training
and evaluation are given at the end of the section.
n-gram Our simplest models are based on
ngrams, which have the advantage of being highly
interpretable. Each data entry (i.e., a dialogue
utterance) is split into chunks of all possible
contiguous sequences of n tokens. The resulting
vectorized features are used by a logistic regression
model to estimate the odds of a text sample
belonging to a certain age group. We experiment
with unigram, bigram and trigram models. A
bigram model uses unigrams and bigrams, and a
trigram model unigrams, bigrams, and trigrams.
LSTM and BiLSTM We use a standard Long
Short-Term Memory network (LSTM)
Accuracy ↑ better ent random initializations. All models are trained on an NVIDIA TitanRTX GPU.
The n-gram models are trained in a One-vs-Rest
(OvR) fashion, and optimized using the
Limitedmemory Broyden–Fletcher–Goldfarb–Shanno
(LBFGS) algorithm
LSTMs and BERT models are optimized using
Adam
We report accuracy and F1 for each age group in Table 2. As can be seen, the performance of all models is well beyond chance level, which indicates that age-related linguistic differences can be detected, to some extent, even by a simple model based on unigrams. At the same time, BERT fine-tuned on the task turns out to be the best-performing model both in terms of accuracy (0.729) and F1 scores, which confirms the effectiveness of Transformer-based representations to encode fine-grained linguistic differences. However, it can be noted that the model based on tri% cases avg. length (± std)* both correct both wrong only trigram correct only BERT correct grams is basically on par with BERT in terms of accuracy (0.722), and well above both the LSTM and BiLSTM models (0.693 and 0.691, respectively). A similar pattern is observed for F1 scores, where BERTF T and the trigram model achieve comparable performance, with LSTMs being overall behind.
Overall, our results indicate that text-based
models are effective, to some extent, in
predicting the age group to which a speaker involved
in a dialogue belongs. This complements
previous evidence that age-related features can be
detected in discourse
There can be several reasons why age group detection is more challenging in dialogue than in discourse. For example, in dialogue there may be dimensions of variation, such as turn-taking patterns, that are not captured by our models and experimental setup. Yet, the present results do reveal a few interesting insights. In particular, 2The corpus contains blog posts appeared on https:// www.blogger.com, gathered in or before August 2004. the very good performance of the trigram model suggests that leveraging ‘local’ linguistic features captured by n-grams is extremely effective in dialogue. This could indicate that differences among various age groups are at the level of local lexical constructions. This deserves further analysis, that we carry out in the next section. 5
We compare the two best-performing models, i.e., BERTF T and the one using trigrams, and aim to shed light on what cues they use to solve the task. We first compare the prediction patterns of the two models, which allows us to detect easy and hard examples. Second, we focus on the trigram model and report the n-grams that turn out to be most informative to distinguish between age groups. 5.1
We split the data for analysis by whether or not both models make the same correct or incorrect prediction, or whether they differ. Table 3 shows the breakdown of these results. As can be seen, a quite large fraction of samples are correctly classified by both models (63.17%), while in 19.78% cases neither of the models make a correct prediction. The remaining cases are almost evenly split between cases where only one of the two is correct. As shown in Figure 2, the 19-29 age group appears to be be slightly easier compared to the 50+ group, where models make more errors.
To qualitatively inspect what the utterances falling into these classes look like, in Table 4 we show a few cherry-picked cases for each age group. We notice that, not surprisingly, both models have trouble with backchanneling utterances consisting of a single word, such as yeah, mm, or really?, which are used by both age groups. For example, both models seem to consider yeah as a ‘young’ cue, which leads to wrong predictions when yeah is used by a speaker in the 50+ group. As for the utterance really?, BERTF T assigns it to the 50+ group, while the trigram model makes the opposite prediction. This indicates that certain utterances simply do not contain sufficient distinguishing information, and model predictions that are based on them should therefore not be considered reliable. This seems to be particularly the case for short utterances. Indeed, through comparing the average length of the utterances incorrectly classified by both models (rightmost column of Table 3), we notice that they are much shorter than those belonging to the other cases. This is interesting, and indicates a key challenge in the analysis of dialogue data: on average, shorter utterances contain less signal. On the other hand, short utterances can provide rich conversational signal in dialogue; for example, backchanneling, exclamations, or other acknowledging acts. As a consequence, using length alone as a filter is not an appropriate approach, as it can remove aspects of language use key to differentiating speaker groups. Analyzing the most informative n-grams used by the trigram model allows us to qualitatively compare the linguistic differences inherent to each age group. In Table 5 we report the top 15 ngrams per group. We find, firstly and intuitively, that colloquial language seems somewhat generational, with unigrams particularly indicative of younger speakers consisting of words such as cool and massive, and for older speakers, words like wonderful. These unigrams are both informative to the model and indicative of differences in both formality and ‘slang’ use across age groups.
These most informative n-grams also indicate differences in back-channeling use between age groups; younger speaker’s language is more characterized by the use of um, hmm, while the top n-grams in the older category will more likely use yes, right, right right. A feature of younger language also apparent from these examples is in their use of more informal language, which also extends to the use of foul language, making up a percent of the most informative unigrams shown in Table 5.
Interestingly, while topic words make up many of the most informative n-grams for older speakers in Table 5, younger speakers are more defined by their use of slang words such as wanna, foul language, or adjectives such as cute, cool, and massive. A key finding from Schler et al. (2006) is in the sentiment of language playing an important role, something which some of the most informative n-grams suggest may also be true for the dialogue dataset. As Table 5 demonstrates, younger speakers use more dramatic language such as negative foul words, and positive love, cute, cool; all words with a strong connotative meaning. We believe that further inspection is needed to determine whether the same sentiment pattern will be true of dialogue as it has been reported to be in discourse.
We investigated whether, and to what extent, NLP models can detect age-related linguistic features in dialogue data. We showed that, in line with what we observed for discourse, state-of-the-art models are capable of doing so with a reasonable accuracy, in particular when the dialogue fragment is long enough to contain discriminative signal. At the same time, we found that much simpler models based on n-grams achieve comparable performance, which suggests that, in dialogue, ‘local’ features can be indicative of the language of speakers from different age groups. We showed this to be the case, with both lexical and stylistic cues being informative to these models in this task.
While we performed the classification task at the level of single dialogue utterances, future work may take into account larger dialogue fragments, such as the entire dialogue or a fixed number of turns. This would make the setup more comparable to discourse, but would require making experimental choices and dealing with extra computational challenges. Moreover, it could be tested whether the language used by a speaker is equally discriminative when talking to a same-age (this work) or a different-age interlocutor.
Finally, we believe our findings could inform future work on developing adaptive conversational systems. Since consistent language style differences were found between age groups (for example, at the level of exclamatives and acknowledgments), systems whose language generation capabilities aim to be consistent with a given age group should therefore reproduce these patterns. This could be achieved, for example, by embedding one or more discriminative modules that control the generation of a system’s output, which could lead to better, more natural interactions between human speakers and a conversational system.
This work received funding from the University of Amsterdam’s Research Priority Area Human(e) AI and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 819455).
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations.