Within English second language acquisition there is an enthusiasm for using authentic text as learning materials in classroom and online settings. This enthusiasm, however, is tempered by the di culty in nding authentic texts at suitable levels of comprehension di culty for speci c groups of learners. An automated way to rate the comprehension di culty of a text would make nding suitable texts a much more manageable task. While readability metrics have been in use for over 50 years now they only capture a small amount of what constitutes comprehension di culty. In this paper we examine other features of texts that are related to comprehension di culty and assess their usefulness in building automated prediction models. We investigate readability metrics, vocabulary-based features, and syntax-based features, and show that the best prediction accuracies are possible with a combination of all three.
Within English second language acquisition there is a fundamental di culty in de ning what is meant by authentic as opposed to non-authentic or arti cial language usage. For example, is authentic usage only the remit of geographical countries where English is their rst language or is an o cial language of communication? Within language teaching the opposition can be made between language usage that is fabricated for the teaching of English as a second language (ESL), and language usage which is not fabricated. This shift between forms of usages can be seen in text books which are used in the learning of ESL where fabricated sentences are often used to highlight speci c forms of language or adapted material is incorporated into the reading and listening material.
The proponents of authentic usage tend to highlight the authentic as
capturing what language is as socio-linguistic utterance in context [
There are, however, some disadvantages to using authentic material in English language teaching. Foremost amongst these is that the language is not primarily designed for learning but for communication between native speakers. This can mean that that level of language used in authentic material can be too di cult, in terms of the complexity of sentences, and, more importantly, the use of unfamiliar words or idiomatic expressions. Authentic, but di cult, texts can make the gap between the presumed level of the student or the class and the di culty of the text too big leading students to quickly lose their motivation. Reliable methods to automatically determine the comprehension di culty of a text could greatly mitigate these disadvantages by making it easy for teachers or learners to source authentic materials of an appropriate level. Readability metrics are one long-standing approach to doing this.
Readability is a term used to refer to the overall understandability or comprehension level of a text. There are a number of established, widely used readability metrics in the literature, such as Flesch and FOG4. Whilst these metrics go some way towards determining the comprehension di culty of text, in general they all tend to focus on speci c, narrow features of the language used|most readability metrics are de ned as functions over counts of word syllables and/or sentence length. As W.H. DuBay points out: \The variables used in the readability formulas show us the skeleton of a text. It is up to us to esh out that skeleton with tone, content, organization, coherence, and design [6, p. 56]. DuBay's analysis highlights that there are many more features of the language used in a text, beyond those modelled by traditional readability metrics, that impinge on the comprehension di culty of that text. Investigating these features is the motivation behind the work described in this paper. We analyse how useful di erent sets of features of the language used in a text are in modelling comprehension di culty of a text. In this work we consider readability metrics, syntax-based features, and vocabulary-based features.
The paper is structured as follows: Section 2 describes how we designed and created a dataset of texts annotated by comprehension di culty; Section 3 describes the features we created and used in our models; Section 4 describes the di erent models we trained and presented the results of our evaluation experiments; in Section 5 we conclude the paper by discussing our results and highlighting some areas of future research. 2
In order to build models to assess the usefulness of di erent features of the
language used in text to predict comprehension di culty we needed a dataset of
texts annotated by comprehension di culty. The rst design decision in creating
this dataset was to decide on the comprehension di culty levels that we would
use for annotation. One option would have been to use the Common European
Framework of Reference for Languages (CEFR) [
Next we collected a corpus of texts whose original purpose was not ESL. The corpus contained 948 texts from a range of international English language online news sources that we expected to include texts at di erent comprehension di culty levels. The average length of these texts in words is 457:5 (with a standard deviation of 379:7). We hired a number of ESL teachers to annotate these texts with di culty levels through a bespoke annotation tool that presented texts to annotators in random order. Our review of the annotations revealed that there were no low level beginner texts in the corpus, this is to be expected as authentic texts at this level are rare. This left us with ve levels of di culty (ESL levels Elementary to Advanced). The distribution of di culty levels within the corpus is shown in Figure 1. 3
There are a wide range of potential descriptive features that could be used in building a predictive model of text comprehension di culty. The high-level domain concepts identi ed as important in this problem were: existing readability measures and related features, features based on the vocabulary in a text, and features based on a syntactic analysis of the text. In the following sections we describe the sets of features we developed and used from each of these domains. 3.1
There are a number of well-known readability metrics, for example: FOG [
F lesch = 206:835 1:015
total words total sentences 84:6 total syllables total words (1)
Figure 2 presents a scatter plot matrix (SPLOM) illustrating the linear
relationships between a number of standard readability metrics: Flesch,
AutomatedReadability Index (ARI) [
As noted in the introduction, readability metrics do not provide a measure of the comprehension di culty of a text. For example, text that includes many idiomatic phrases, or novel turns of phrase may get a good readability score but this does not indicate that it will be easy to understand or comprehend such text. That said, readability metrics do provide an objective standard and do provide some information regarding comprehension di culty. In developing our predictive models we considered all of the readability metrics shown in Figure 2 as input features. 3.2
The words used in a text can have a direct impact on the comprehension difculty of the text. The use of complex words is likely to make a text more di cult to read and to comprehend. This is why so many readability metrics use some measure of word length (syllable or character count) as a proxy for word complexity in the calculation of readability. A striking example of this is the FOG , also known as Gunning-Fog, readability metric which explicitly takes the number of complex words into account in its calculation, see Equation 2. FOG de nes complex words as those words with three or more syllables (where common su xes are not counted as syllables; e.g., -ed, -ing, etc.) and which are not proper nouns, familiar words, or compound nouns.
F OG = 0:4 + 100
total words total sentences complex words total words (2)
The FOG metric is an example of a readability metric that relies on word categories (e.g., familiar words, proper nouns, etc.) as operationalised by prespeci ed lists. A challenge for these models is how best to de ne these word lists.
While the occurrence of speci c words might work as a predictor of text di culty, intuitively documents that include larger numbers of words that are generally rare are likely to be more di cult to understand than documents that primarily use more common words. To achieve this we used what we refer to as rare-word features which capture the predominance of rare words within a document in a generalisable way.
We based our rare-word features on word frequencies from the British
National Corpus (BNC). We chose to use the BNC as our background corpus
because it is a balanced sampled corpus so it is reasonable to extrapolate from
the frequencies found in BNC to general English [
We created two other vocabulary based features, one measured the lexical
diversity of a document and the other the frequency of named entities in a
document. Lexical diversity measures the range of di erent words used in a
document, with a greater range indicating a higher diversity [
count of unique words T T R = (3)
total words
The nal vocabulary feature we used was the percentage of words within a
document that are part of named entity expressions. We identi ed the named
entities in the text using the named entity recognition module of the Stanford
CoreNLP software [
The occurrence of particular parts of speech and/or syntactic structures may
a ect the di culty of a text from an ESL perspective. For example,
prepositions and prepositional clauses, conjunctions, subjunctions, adverbs and
adverbial clauses can all pose di culties to ESL students. To capture these syntactic
phenomena within our models we generated a set of features by rst parsing the
texts and then generating features from the parse tree annotations. We parsed
the texts using the Stanford CoreNLP software [
The rst set of features we created from the parse trees were the percentages of each word-level part-of-speech (POS) tag in each text. These POS percentages were generated by simply dividing the count of occurrences of each POS tag in a text by the total number of POS tags in the text. The second set of syntactic features generated from the parse trees measured the distribution of syntactic tags in each text (e.g. tags such as ADJP adjective phrase, SBAR subordinate clause, etc.). These features were de ned in a very similar manner to the POS percentage features: we simply counted the number of occurrences of each syntactic tag in parse trees generated from a text and divide these counts by the total number of syntactic tags in this parse tree set.
Inspired somewhat by the relationship between lexical diversity and text di culty we created two features to capture the diversity of POS and syntactic tags within a text: the rst feature simply counted the number of di erent parts of speech tags that occurred at least once in the trees generated from a text; similarly, the second diversity feature counted the number of di erent syntactic tags that occurred at least once in the trees generated from a text. This was based on an intuition that a greater range of POS tags or syntactic tags within a single document could cause comprehension di culties.
The last two features we generated from the parse trees were designed to capture the complexity of the sentences in a text. In 1979 Flesch motivated the inclusion of a parameter based on the length of a sentence within his readability scores as follows:
The longer the sentence, the more ideas your mind has to hold in suspense until its nal decision on what all the words mean together. Longer sentences are more likely to be complex|more subordinate clauses, more prepositional phrases and so on. That means more mental work for the reader. So the longer a sentence, the harder it is to read. [9, p. 22] Certainly Flesch's argument is a good motivation for including sentence length in a measure of readability, and also in a measure of text di culty. However, sentence length alone does not do justice to the potential di erences in di culty between sentences of the same length. For example, a sentence that includes multiple clause embeddings is likely to be more di cult to comprehend than a sentence of a similar length that is composed of a simple (if long) list. To capture these aspects of complexity we created three other features based on the set of parse trees generated from the text. These were: 1. Max Embeddedness : that maximal phrasal parse tree depth for sentences within a text (implemented by iterating through the parse trees for a text, linearising each parse tree, then reading each linearised parse tree from left to right and during this iterative process keeping track of the maximum number of open brackets encountered at any point) 2. Average Embeddedness : the average phrasal parse tree depth for sentences within a text (implemented in a similar way to Max Embeddedness. 3. Average Phrasal Parse Tree Nodes : simply the average number of phrasal parse tree nodes in sentences within a text. 4
To evaluate the di ering power of the di erent feature sets described in Section 3 to predict the comprehension di culty of a document we built and evaluated multi-variate prediction models using each feature subset: readability metrics, vocabulary-based features, and syntax-based features. We also consider the performance of a model using the full combined set of features generated, and a model where only a subset of the what appear to be the most useful features are used.
To address the class imbalance in our dataset (we have many more documents at the intermediate level than at any other level, see Figure 1) we have converted from a categorical classi cation problem across the ve di erent levels to a numeric prediction problem which each level is associated within a numeric score. The mappings are as follows elementary : 10, lower-intermediate: 30, intermediate: 50, upper-intermediate: 70, and advanced : 90. for prediction problems with ordinal targets this is a sensible approach to handling class imbalance.
To select a subset of the most useful features from the set available we use a
simple rank and prune feature selection approach [
In all cases the models used are support vector regression (SVR) models [
To evaluate the performance of each model we perform a 10-fold cross validation experiment measuring model performance using mean absoloute prediction
error. The performances of the models built using the ve di erent features sets are shown in Table 2. We can see that the model built using the selected feature subset performs best out of the models test, although it is only very marginally better than the model trained using the full feature set.
We can illustrate the ability of these models to distinguish between the document of di erent comprehension ability through boxplots that illustrate the distribution of predictions for each di cult level. These are shown in Figure 3. We include the boxplot for the FOG readability metric as a baseline as well as the other feature sets. These boxplots clearly show a progression in the ability of models trained using di erent feature sets to separate texts into the di erent comprehension di culty levels labelled in the text. 5
The ability to automatically rate the comprehension di culty of texts would greatly reduce the challenge of using authentic text in ESL classrooms and online services. While, the use of readability metrics has been demonstrated as a very useful determination of the general level of a text, this is not su cient for rating comprehension di culty. Comprehension di culty is in uenced by more features than just the simple measures of word and sentence complexity incorporated into readability metrics. In this paper we describe an analysis into di erent types of features of texts that are useful for predicting readability. We consider three di erent groups of features: readability metrics, syntax-based features, and vocabulary-based features. We base this analysis on a corpus of 948 texts collected from a range of international English language online news sources that were expertly annotated into ve of the traditional English as Second Language levels: Beginner, Elementary, Lower Intermediate, Intermediate, Upper Intermediate and Advanced. We perform our analysis by building an evaluating predictive models using di erent feature subsets extracted from the document corpus.
The rst thing this analysis illustrates is a con rmation that, although readability metrics can provide some indications of the of the di culty of comprehension of a text for ESL, they are not su cient to do the job of automatic rating accurately. This result highlights that it is necessary to make a distinction between the level of di culty of comprehension of a text, in particular for English as Foreign language, and the standard readability scores. The second thing we show is that none of the di erent groups of features lead to models that are better than one trained with features from the di erent groups combined.
There are many extensions that could be made to the analysis described in
this paper. For example, the level of comprehension di culty of a text can be
linked to the cultural context, the use of idiosyncrasies, or the use of idioms, and
novel turns of phrase. While the identi cation of these aspects of text can be
done through the use of computational models (see for example see [