The article focuses on identifying, extracting and evaluating syntactic parameters in uencing the complexity of Russian academic texts. Our ultimate goal is to select a set of text features e ectively measuring text complexity and build an automatic tool able to rank Russian academic texts according to grade levels. models based on the most promising features by using machine learning methods The innovative algorithm of designing a predictive model of text complexity is based on a training text corpus and a set of previously proposed and new syntactic features (average sentence length, average number of syllables per word, the number of adjectives, average number of participial constructions, average number of coordinating chains, path number, i.e. average number of sub-trees). Our best model achieves an MSE of 1.15. Our experiments indicate that by adding the abovementioned syntactic features, namely the average number of participial constructions, average number of coordinating chains, and the average number of sub-trees, the text complexity model performance will increase substantially.
E ective reading comprehension implies that reading materials correspond readers' cognitive and language abilities. The idea behind the existing practice in education is to ensure that students are exposed to the age-appropriate materials which are neither too complicated nor too simple for a reader. The \ageappropriateness" has been traditionally measured by the Grade level which is viewed as \what all students need to know and be able to do at each grade level" to progress through their education1.
Grade level descriptors identify the speci c content (knowledge, skills, abilities) and the language of a particular course which students at a particular education stage (or grade) are exposed to2. There are also a number of English text complexity analyzers available online for any educator selecting a text for students34. Eliminating the gap between critically the texts and students' abilities scholars have been developing tools to pro le texts that students would be able to and want to read56. The existing automatic analyzers use hundreds of parameters ranging from quantitative, i.e. word length and sentence length only, to qualitative (levels of meaning or purpose; structure; language conventionality and clarity; and knowledge demands) to match a particular reader and a text78.
T.E.R.A., for instance, is an engine developed in 2012 by SoLET Lab which
analyzes ve textual components, such as narrativity, syntactic simplicity, word
concreteness, referential cohesion and deep cohesion9. T.E.R.A. also predicts the
grade level of the text using the Flesch-Kincaid Grade Level readability formula
(FKGL) [
The situation in Russia is di erent. Signi cant gaps have been reported between the complexity levels of texts that students are asked to read in high school as well as at tertiary levels and students' abilities: the books are either too simple or too complicated for students10.
Researchers also have evidence of students' lack of wish to read11 who in
many cases are caused by the inappropriate selection of a book by an educator.
Unfortunately, Russian text complexity analyzers so far apply no other variables
but quantitative, i.e. word length and sentence length[
Though the terms text complexity', text di culty' and text readability' are still
sometimes used interchangeably, the notions of the concepts began being
separated in Russian academic literature as early as in 1970-s [
F RE = 206:836 (1:52 ASL) (65; 14 ASW ) Here, ASL is the average sentence length, i.e., the number of words divided by the number of sentences; ASW is the average number of syllables per word, i.e., the number of syllables divided by the number of words in a text. The constants were calculated based on the similar English formulas as well as the comparison of 100 parallel English/Russian literary texts and words in two academic dictionaries: Explanatory Dictionary of the Russian Language by Ozhegov S.I. with 39174 words registered12 and English-Russian Muller's dictionary with 41977 words13. 3
Two collections of texts were assembled for the research. The rst collection of 7 texts from textbooks on Social Studies by L. N. Bogolubov marked \BOG" was selected to teach the predictive model and de ne independent variables of the text variation in the range of 5 { 11 Grade Levels. The second collection of 7 texts from textbooks on Social Studies by A.F. Nikitin marked \NIK" also aimed for 5 { 11 Grade Levels. Further we refer to the two collection as a Russian Readability Corpus (RRC). Both sets of textbooks are from the \Federal List of Textbooks Recommended by the Ministry of Education and Science of the Russian Federation to Use in Secondary and High Schools"14. 12 https://slovarozhegova.ru/ 13 https://slovar-vocab.com/english-russian/muller-vocab.html 14 http://www.fpu.edu.ru/fpu/
To ensure reproducibility of results, we uploaded the corpus on a website thus providing its availability online15. Note, however, that the published texts contain shu ed order of sentences. The sizes of BOG and NIK collections of texts are presented in Table 1.
Tokens Sentences Words per sentence Syllables per word Grade BOG NIK BOG NIK BOG NIK BOG NIK 5-th { 17,221 { 1,499 { 11.49 { 2,35 6-th 16,467 16,475 1,273 1,197 12.94 13.76 2.56 2,71 7-th 23,069 22,924 1,671 1,675 13.81 13.69 2.84 2,70 8-th 49,796 40,053 3,181 2,889 15.65 13.86 2.96 2,88 9-th 42,305 43,404 2,584 2,792 16.37 15.55 3.04 3,00 10-th 75,182 39,183 4,468 2,468 16.83 15.88 3.07 3,12 10-th* 98,034 { 5,798 { 16.91 { 3.05 { 11-th { 38,869 { 2,270 { 17.12 { 3,11 11-th* 100,800 { 6,004 { 16.79 { 3.19 {
For the sake of convenience, we have preprocessed all texts from the corpus in the same way. Common preprocessing included tokenization, splitting text into sentences and part-of-speech tagging (using the TreeTagger for Russian16). During the preprocessing step we excluded all extremely long sentences (longer than 120 words) as well as too short sentences (shorter than 5 words) which we consider outliers. Clearly, such sentences can be not outliers at all in another domain, but in case of school textbooks on Social Studies sentences shorter than 5 words are outliers.
Extremely short sentences mostly appear as names of chapters and sections of the books or as a result of incorrect sentence splitting. We omit those sentences, because the average sentence length is a very important feature in text complexity assessment and hence should not be biased due to splitting errors. At the same time sentences with ve to seven words in Russian can still be viewed as short sentences. 4.2
We have explored an extended feature set for text complexity modeling: 15 https://kpfu.ru/slozhnost-tekstov-304364.html 16 http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/ { frequency of content words (FREQ), { average words per sentence (ASL), { average syllables per word (ASW), and { features based on POS-tags: number of nouns per sentence (NOUNS), number of verbs per sentence (VERBS), number of adjectives per sentence (ADJ), number of pronouns per sentence (PRONOUNS), number of personal pronouns per sentence (PERS. PRONOUNS), number of negations per sentence (NEG), number of connectives per sentence (CONN).
We have tested the features for their importance in linear regression model. The two features have shown better performance than others: FREQ and ADJ. To assess the quality of proposed models the mean squared error (MSE) and the coe cient of determination R2 were used. For the results of tting the parameter values in the corpus, see Table 2. The modern tools for processing of Russian texts are able to extract several syntactic features of texts thus permitting to include a number of syntactic structures as features in readability formulas. To this end we use a syntactic analyzer \ETAP-3" that employs a very detailed description of Russian grammar.
The parser of the multipurpose linguistic processor ETAP-3 is a program
that performs parsing. In linguistic terms, parsing results in a dependency tree
structure, the nodes of which are word tokens of the input sentence, and `the
edges' are the established syntactic dependencies. Thus, every tree node
corresponds to a word token in the sentence processed, whilst the directed arcs are
labeled with names of syntactic relations [
The list of features extracted by the ETAP syntax parser was preprocessed to group similar features. We provide the results of correlation analysis in the Table 3. In general, some syntactic feature are similar to others and correlate with the target variable (readability, measured as a class number). However, it is evident that all the syntactic features have lower correlation coe cient with the target feature (`Grade Level'), than the two `classical' lexical features (ASL and ASW) do.
Nevertheless, syntactic features have high correlation with the target variable. This information could be useful for readability and text complexity prediction in Russian. Our goal is to evaluate syntactic features in the next subsection with respect to the capability to serve as predictors in a linear regression model. As shown in Table 3 few of the syntactic features have low correlation with the target variable (`Grade Level'). In the rest of the paper we consider only those syntactic features with correlation coe cient above 0:8 (as depicted in Table 3). 4.4
For evaluation of the syntactic features we carried out the two experiments. In the rst experiment, we clustered the syntactic features with respect to their similarity to each other. By similarity we treat the correlation of the features' values derived in RRC. The derived groups of features are the following: { Group 1, (G-1): Features related to the structure of the syntax tree leaves number (LN) average path (AP) longest path (LP) path number (PN) { Group 2, (G-2): Features related to noun and participial linear sequences prich rate (PR) nouns dep (ND) { Group 3, (G-3): Features related to coordinating constructions average sochin length (SL) sochin number (SN)
The selected syntactic features could serve better predictors in a linear regression model due to their high correlation with the target variable. We measured the performance of the resulting model in the following way. A linear regression model was trained on the `BOG' collection and tested on the `NIK' collection and vice versa. In both cases, we use the MSE as a measure of model's performance 4. First, we evaluate three features with the highest correlation: SN, PR and ND. Next, three rows of the Table 4 correspond to the groups of syntactic we found. Finally, we evaluate three features, one coming from a certain group. From each group, we pick a feature with the highest correlation: PN from G-1, PR from G-2 and SN from G-3. It can be seen from the Table 4 that syntactic features from groups G-2 and G-3 are better than those from G-1. In the second experiment, we compared syntactic features to the lexical level features (ASW, ASL, FREQ and ADJ) with respect to their performance. Finally, we have built linear models using combinations of both syntactic and lexical features (Table 5). Using only syntactic features without any additional features leads to worst values of MSE. It is shown in the 5th row of Table 5. However, combination of lexical level features (ASL and ASW) with syntactic features improves performance of linear model for readability assessment. 5
Earlier, in [
The main challenge to cope with is the selection of optimal values for the
constants in the formulas. We show that syntactic features could be useful in
readability assessment. However, the in uence of syntactic features was not as
great as one might expect. This indicates the future direction of research, related
to the expansion of a set of syntactic properties. In future research, we plan to
apply semantic features, such as features based on syntactic n-grams [
The research indicates that syntactic features have a high correlation with the target variable. The ndings received could be useful for readability and text complexity prediction of Russian texts. Since this is a preliminary study, there is ample space to improve all major stages in the future process of syntactic feature extraction. The potential working direction is viewed by the authors as improving the accuracy of sentence and clause boundary detectors and parsers. We will further experiment with syntactic complexity measures to balance the text complexity construct multiplicity and model's simplicity. Furthermore, we can also experiment with additional types of machine learning models and tune parameters to derive text complexity assessment models with better performance.
This research was nancially supported by the Russian Science Foundation, grant # 18-18-00436, the Russian Government Program of Competitive Growth of Kazan Federal University, state assignment of Ministry of Education and Science, grant agreement # 34.5517.2017/6.7. The Russian Academic Corpus (section 3 in the paper) was created without support from the Russian Science Foundation.