This paper explores the relationship between Neural Language Model (NLM) perplexity and sentence readability. Starting from the evidence that NLMs implicitly acquire sophisticated linguistic knowledge from a huge amount of training data, our goal is to investigate whether perplexity is affected by linguistic features used to automatically assess sentence readability and if there is a correlation between the two metrics. Our findings suggest that this correlation is actually quite weak and the two metrics are affected by different linguistic phenomena.1 1 Introduction and Motivation
Standard Neural Language Models (NLMs) are
trained to predict the next token given a context of
previous tokens. The metric commonly used for
assessing the performance of a language model is
perplexity, which corresponds to the inverse
geometric mean of the joint probability of words
w1; :::; wn in a held-out test corpus C. While
being primarily an intrinsic metric of NLM quality,
perplexity has been used in a variety of scenarios,
such as to classify between formal and colloquial
tweets
In this paper we focus on a less investigated
perspective addressing the connection between
perplexity and readability. Since by definition
perplexity gives a good approximation of how well
a model recognises an unseen piece of text as a
plausible one, our intuition is that lower model
perplexity should be assigned to easy-to-read
sentences, while difficult-to-read ones should obtain
higher perplexity. On the other hand,
state-ofthe-art NLMs trained on huge data have shown
to implicitly learn a sophisticated knowledge of
language phenomena, also with respect to
complex syntactic properties of sentences
In order to test the validity of all these hypotheses, we rely on the perplexity score given by a state-of-the-art NLM for the Italian language to several datasets representative of different textual genres containing both easy– and complex– to–read sentences: ideally, such datasets should emphasise the correlation between perplexity and readability (if present) since the corpora are explicitly designed to contain both simple and difficult examples.
Contributions We inspect whether and to which extent it is possible to find a relationship between a readability score and the perplexity of a NLM. To this aim we investigate (i) if the perplexity of a NLM and the readability score of a set of sentences show a significant correlation and (ii) whether the two metrics are equally affected by the same set of linguistic phenomena that occur in the sentence. 2
According to our research questions, we devised a set of experiments to study whether NLMs perplexity reflects the level of readability of a sentence and which are the linguistic phenomena mostly involved in each metric. For this purpose, we firstly investigated whether sentence-level perplexity scores computed with one of the most prominent NLM model correlate with the scores assigned to the same sentences by a supervised readability assessment tool. Secondly, we investigated which are the linguistic features of the considered sentences that correlate in a statistically significant way with the perplexity and readability score respectively. In order to verify whether correlations hold across different typology of texts, we tested our approach on five Italian datasets. 2.1
READ-IT. Automatic readability (henceforth ARA) was assessed using READ-IT (Dell’Orletta et al., 2011) the first readability assessment tool for Italian which combines traditional raw text features with lexical, morpho-syntactic and syntactic information extracted from automatically parsed documents. In READ-IT, analysis of readability is modelled as a binary classification task, based on Support Vector Machines using LIBSVM (Chang and Lin, 2001). Training corpora are representative of two classes of texts, i.e. difficult– vs. easy– to-read ones, both containing newspaper articles. The set of features exploited for predicting readability has been proved to capture different aspects of sentence complexity. Thus, the assigned readability score ranges between 0 (easy-to-read) and 1 (difficult-to-read) referring to the percentage probability for unseen documents or sentences to belong to the class of difficult-to- read documents. For the purposes of our work, we carried out readability assessment at sentence level, making the analysis reliable for the comparison with sentencebased perplexity of a NLM.
GePpeTto. Sentence-level perplexity scores were
computed relying on GePpeTto (De Mattei et al.,
2020). GePpeTto is a generative language model
trained on the Italian language and built using the
GPT-2 architecture
P P L = e( NNLL ) where N N L and N correspond respectively to the negative log-likelihood and to the length of each sentence w1:n = [w1; :::; wn] in the datasets. 2.2
In order to test the reliability of our initial hypothesis, we chose four corpora containing different typologies of texts, i.e. web pages, educational materials, narrative texts, newspaper and scientific articles. Each corpus includes a balanced amount of difficult- and easy-to-read sentence. In addition, we also considered in the analysis the Italian Universal Dependency treebank. This is meant to verify whether the connection between sentencelevel readability and perplexity also holds in a well-acknowledged benchmark corpus. For each of them, we excluded from our analysis short sentences, i.e. having less than 5 tokens.
PACCSS-IT2 (Brunato et al., 2016): we took into account 125,977 sentences belonging to PACCSSIT, a corpus of complex-simple aligned sentences extracted from the ItWaC corpus. The resource was build using an automatic approach for acquiring large corpora of paired sentences able to intercept structural transformations (such as deletion, reordering, etc.). For example, the two following sentences represent a pair in the corpus, where a reordering operation occurs at phrase level (i.e. the subordinate clause proceeds vs. follows the main clause):
Complex: Ringraziandola per la sua cortese attenzione, resto in attesa di risposta. [Lit: 2http://www.italianlp.it/resources/paccss-it-parallelcorpus-of-complex-simple-sentences-for-italian/ Thanking you for your kind attention, I look forward to your answer.] Simple: Resto in attesa di una risposta e ringrazio vivamente per l’attenzione. [Lit: I look forward to your answer and I thank you greatly for your attention.] Terence and Teacher3 (Brunato et al., 2015): two corpora of original and manually simplified texts aligned at sentence level. Terence contains short Italian novels for children and their manually simplified version carried out by linguists and psycholinguists targeting children with text comprehension difficulties. Teacher is a corpus of pairs of documents belonging to different genres (e.g. literature, handbooks) used in educational settings manually simplified by teachers. We exploited 1,644 sentences belonging to these corpora. Multi–Genre Multi–Type Italian corpus: a collection of Italian texts representative of three traditional textual genres: Journalism, Scientific prose and Narrative. Each genre has been internally subdivided into two sub-corpora representative of an easy- vs difficult-to-read variety, which was defined according to the intended target audience for a given genre. The journalistic prose corpus includes articles automatically downloaded from the online versions of two general-purpose newspapers4, while the “easy” sub-corpus contains articles from two easy-to-read newspapers5 addressed to adults with low literacy skills or mild intellectual disabilities. The scientific prose collection consists of scholarly publications on linguistics and computational linguistics and Wikipedia pages downloaded from the portal “Linguistics”, representative of the complex and easy variety respectively. For the narrative genre, we included long novels written by novelists of the last century and contemporary writers in the corpora of complex variety, while for the easy variety we collected short novels for children. The complete corpus contains 56,685 sentences.
cludes different sections of the Italian Universal
Dependency Treebank (IUDT), version 2.5
Our analysis starts from a comparison between the average perplexity and readability scores obtained for each sentence of the five considered datasets. As shown in Table 1, readability values (column ARA) are quite homogeneous across the datasets, with low standard deviation values. On the contrary, the range of perplexity scores is wider (column PPL), going from an average score of 3,905.83 of PACCSS-IT to 436.75 of the IUDT miscellaneous portion (Italian UD). These differences seem to provide a first evidence that perplexity and readability are not correlate to each other.
This intuition has been proved computing the Spearman’s rank correlation coefficient between the perplexity and readability scores for each dataset. Results are reported in Table 2, column PPL-ARA. As it can be seen, all correlation rates are significant, except for the result obtained on the Terence and Teacher corpus, possibly due to the fact that the size of the corpus is too small to allow a significant comparison. Contrary to our expectations, no correlation was detected between the two metrics for all corpora, suggesting that perplexity and and readability are independent from each other.
To further investigate the reasons behind these scores and to deepen the analysis about the relationship between the two metrics, we investigated whether they capture the same (or similar) linguistic properties of the sentences. To this aim, we tested the presence and strength of the correlation between each of the two metrics and a set of 176 linguistic features, which have been shown to capture properties of sentence complex6https://github.com/UniversalDependencies/UD ItalianISDT
7https://github.com/UniversalDependencies/UD ItalianPoSTWITA 8https://universaldependencies.org/treebanks/it twittiro
PACCSS-IT Terence-Teacher Multi-Genre Multi-Type Italian-UD Twitter-UD
PPL
3,905.83 ( 21,306.07)
790.85 ( 5,002.62)
570.85 ( 4,820.12)
436.75 ( 3,633.64)
986.28 ( 2,479.64)
ity (Brunato et al., 2018). In particular, this
analysis is based on the set of features described in
Brunato et al. (2020), which are acquired from
raw, morpho-syntactic and syntactic levels of
annotation. They range from basic information on
the average sentence and word length, to
lexical information about the internal composition of
the vocabulary of the text (e.g. the distribution of
lemmas belonging to the Basic Italian Vocabulary
(De Mauro, 2000)). They also include morpho–
syntactic information (e.g. POS distribution and
of inflectional properties of verbs) and more
complex aspects of sentence structure derived from
syntactic annotation and modeling global and
local properties of parsed tree structure, e.g. the
relative order of subjects and objects with respect
to the verb, the use of subordination. In order
to extract these features, the considered corpora
were morpho-syntactically annotated and
dependency parsed by the UDPipe pipeline
Column Feats of Table 2 illustrates the results of this analysis: we report the Spearman’s correlation coefficients between the two rankings of linguistic features, each ordered by strength of correlation between feature value and perplexity score and readability score respectively. Once again we observe rather weak correlation values, with the only exception of Italian-UD which is the only one reporting a medium correlation (.332). Overall, these results corroborate our previous findings that the two metrics are not particularly related with each other, and they further suggest that the linguistic phenomena affecting the perplexity of NLM and the readability level of a sentence are very different. Consider for example the two following sentences: (1) Il furto e` avvenuto gioved`ı notte.
The theft has taken place Thursday night. (2) Il comitato di bioetica: no all’eutanasia.
The bioethics committee: no to euthanasia.
While (1) is very easy-to-read, with a readability score of 0.25, but it has a quite high perplexity score, i.e. 40,737.81, (2) is quite difficult-to-read (ARA=1) but is has a very low perplexity score (PPL=11.24).
To better explore the motivation behind these results, we performed an in-depth investigation aimed at understating the relationship between our set of linguistic features and the two metrics taken into consideration. Since we noticed that for all datasets a higher number of features correlates with ARA than with PPL, we selected those that are significantly correlated with the two metrics. The number of shared features varies for each dataset, depending on their size. For example, for the two smallest ones, i.e. Terence and Teacher and the UD Twitter Treebank, we could only consider 34.65% (61) and 44.88% (79) of the whole set of features respectively, while for the larger corpora the sub-set is wider: 81.81% (144) in PACCSS-IT, 78.97% (139) for Multi-Genre MultiType and 84.65% (149) for the IUD Treebank.
Table 3 shows the top ten features for each dataset, i.e. those that obtained the strongest correlation with both PPL and ARA. As expected, correlations are generally stronger between linguistic features and readability scores, although they Terence and Teacher Corr 0,53 0,51 0,50 0,50 0,50 0,49 0,48 0,48 0,48 Corr 0,25 0,23 0,22 0,21 0,21 -0,16 0,16 -0,16 0,14 Feats Corr xpos dist FF 0,34 dep dist punct 0,32 upos dist PUNCT 0,32 ttr form 0,29 aux mood dist Cnd 0,25 upos dist DET 0,25 dep dist det 0,25 ttr lemma 0,22 upos dist NOUN 0,21
ARA Feats Corr dep dist det -0,39 upos dist DET -0,38 upos dist NOUN -0,37 xpos dist S -0,37 xpos dist RD -0,29 upos dist ADV 0,27 dep dist advmod 0,25 xpos dist FF 0,25 avg sub chain len 0,24 Multi-Genre Multi-Type are lower than expected. This could be due to the fact that, even if the READ–IT classifier is trained with a similar set of features, the non-linear feature space makes it difficult to identify clear correlations with individual features. Similarly, our set of features seem to play only a marginal role on perplexity. However, this is not the case of the PACCSS-IT corpus, for which the set of considered linguistic features have an higher correlation with PPL. This can be possibly related to the partial overlap between the GePpeTto training data and the PACCSS-IT sentences, since the latter is drawn from the ItWac corpus which is included in the GePpeTto’s training.
Inspecting these results, we can also observe that correlations between features and PPL seem to be more affected by genre–specific characteristics.
This is particularly clear if we consider the Italian UD Twitter treebank, for which among the top ten most correlated features we find some of them characterising social media language, e.g. symbols (upos-xpos dist SYM) or the vocative relation, which marks a dialogue participant addressed in a text along with the specifi(dep dist vocative:mention).
The paper presented a study aimed at investigating the relationship between two metrics computed at sentence-level, i.e. perplexity of a state-of-the-art NLM for the Italian language and readability score automatically assigned to a sentence by a supervised classifier. We carried out our analysis considering several datasets differing at the level of textual genre and language variety. Specifically, we observed that comparing the rankings obtained using the two metrics we cannot find any significant correlation, either between the scores of the two metrics or with respect to the set of linguistic features that mostly impact their values. Further investigation within this line of research will explore whether we can draw the same observations when a different NLM is exploited to compute sentence perplexity. Feats aux num pers dist Sing+3 dep dist cop avg max depth upos dist ADP xpos dist E dep dist case n tokens dep dist root xpos dist FS
PPL Feats xpos dist B verbs num pers dist Sing+3 lexical density dep dist advmod upos dist ADV verbs num pers dist Plur+3 xpos dist V avg token per clause upos dist VERB
PPL Feats Corr n tokens -0,19 dep dist root 0,19 dep dist advmod 0,19 upos dist ADV 0,18 n prepositional chains -0,18 xpos dist B 0,18 upos dist ADP -0,17 xpos dist E -0,17 ttr lemma 0,16
PPL web: a collection of very large linguistically proverting italian treebanks: Towards an italian stanford dependency treebank. In Proceedings of the ACL ARA Feats Corr principal prop dist -0,42 ttr form -0,34 xpos dist FF 0,34 dep dist det -0,33 upos dist DET -0,33 upos dist PUNCT 0,33 dep dist punct 0,33 xpos dist FB 0,31 sub prop dist 0,27
ARA Feats principal prop dist sub proposition dist n tokens dep dist root ttr form avg max depth avg links len max links len avg max links len
Corr -0,53 0,40 0,39 -0,39 -0,37 0,36 0,35 0,34 0,34 Italian UD Twitter Treebank
Corr 0,38 -0,28 0,28 -0,24 0,23 -0,22 0,21 -0,21 -0,19
ARA Feats upos dist PUNCT dep dist punct dep dist det upos dist DET verbal root perc xpos dist RD avg token per clause subj pre obj post
Corr 0,30 0,30 -0,29 -0,29 -0,27 -0,27 -0,27 -0,27 -0,24
Italian UD Treebank tion scores between perplexity and readability. cessed web-crawled corpora. Language resources and evaluation, 43(3):209–226.
Marc Benzahra and Franc¸ois Yvon. 2019. Measuring text readability with machine comprehension: a pilot study. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 412–422, Florence, Italy, August. Association for Computational Linguistics. Linguistic Annotation Workshop & Interoperability with Discourse, Sofia, Bulgaria, August.
Dominique Brunato, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni. 2015. Design and annotation of the first italian corpus for text simplification. In Proceedings of The 9th Linguistic Annotation Workshop, pages 31–41.
Dominique Brunato, Andrea Cimino, Felice Dell’Orletta, and Giulia Venturi. 2016. PaCCSSIT: A parallel corpus of complex-simple sentences for automatic text simplification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 351–361, Austin, Texas, November. Association for Computational Linguistics.
Dominique Brunato, Lorenzo De Mattei, Felice Dell’Orletta, Benedetta Iavarone, and Giulia Venturi. 2018. Is this sentence difficult? do you agree? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2690–2699, Brussels, Belgium, October-November.
Association for Computational Linguistics.
Dominique Brunato, Andrea Cimino, Felice Dell’Orletta, Giulia Venturi, and Simonetta Montemagni. 2020. Profiling-UD: a tool for linguistic profiling of texts. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 7145–7151, Marseille, France, May. European Language Resources Association. Chih-Chung Chang and Chih-Jen Lin. 2001. LIB
SVM: a library for support vector machines. Alessandra Teresa Cignarella, Cristina Bosco, and Paolo Rosso. 2019. Presenting TWITTIR O`-UD: An italian twitter treebank in universal dependencies. In Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019).
Trevor Cohen and Serguei Pakhomov. 2020. A tale of two perplexities: Sensitivity of neural language models to lexical retrieval deficits in dementia of the Alzheimer’s type. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1946–1957, Online, July. Association for Computational Linguistics.
Lorenzo De Mattei, Michele Cafagna, Felice Dell’Orletta, Malvina Nissim, and Marco Guerini. 2020. Geppetto carves italian into a language model. arXiv preprint arXiv:2004.14253.
Tullio De Mauro. 2000. Il dizionario della lingua italiana, volume 1. Paravia.
Felice Dell’Orletta, Simonetta Montemagni, and Giulia Venturi. 2011. READ–IT: Assessing readability of Italian texts with a view to text simplification. In Proceedings of the Second Workshop on Speech and Language Processing for Assistive Technologies, pages 73–83, Edinburgh, Scotland, UK, July. Association for Computational Linguistics.
Rodolfo Delmonte, Antonella Bristot, and Sara Tonelli. 2007. VIT - Venice Italian Treebank: Syntactic and quantitative features. In Proceedings of the Sixth International Workshop on Treebanks and Linguistic Theories.
V. Demberg and Frank Keller. 2008. Data from eyetracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109:193–210. Keyur Gabani, Melissa Sherman, Thamar Solorio, Yang Liu, Lisa Bedore, and Elizabeth Pen˜a. 2009. A corpus-based approach for the prediction of language impairment in monolingual English and Spanish-English bilingual children. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 46–55, Boulder, Colorado, June. Association for Computational Linguistics.