Text Complexity and Abstractness: Tools for the Russian Language

Abstract

The article focuses on two parallel studies aimed at validating an original automatic tool (RusAC) designed to define the level of abstractness of Russian texts. The studies were conducted on: (a) the Russian Academic Corpus (RAC) compiled of the textbooks used in middle and high schools of the Russian Federation and (b) students' recalls of academic texts. The design of RusAC is based on the Russian Dictionary of abstractness / concreteness compiled by the authors in previous studies, which enlists abstractness ratings of over 88.000 tokens. The pilot studies pursued on the Russian Academic Corpus (circa 3 mln tokens) proved that the ratio of abstract words grows in textbooks of all disciplines across grades from 5 to 11. We also confirmed that the share of abstract words in Science textbooks is lower than that in the Humanities textbooks and that abstractness of readers' recalls is typically lower than that of the original text as the respondents tend to omit more abstract words than concrete. The findings of the research may be applied in a wide range of spheres including education, business, PR, medicine etc. as RusAC facilitates leveling texts for different categories of readers.

Keywords

• Text Complexity
• Abstractness
• Concreteness
• Textbooks

Introduction

In modern Education leveling and profiling texts is viewed profoundly significant as graduated reading levels of text books build students' confidence and increase comprehension. The latter can be achieved only with the help of automated tools able to discriminate texts for readers of various reading literacy levels. “A computational approach to distinguishing texts offers researchers and educators a number of exciting avenues of interest” [ 1 ]. It is especially true about distinguishing abstractness/concreteness ratings of different texts which may serve as good predictors of text complexity [see 2, 3]. However, an automated tool able to compute texts abstractness and correlate it with text complexity has recently been a research niche. In this article we present the study aimed at validating an innovative automated tool RusAC designed and developed to assess a number of linguistic metrics

Copyright ©2020 for this paper by its authors.

Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). of Russian texts. The study was organized into three major parts: (1) design and development of a an automated tool (tagging program) that identifies abstract words in the texts; (2) validation of the tool through a computerized abstractness analysis based on the Russian Dictionary of abstractness/ concreteness and the tagging program.

Until now, in the absence of a dictionary of abstract / concrete words, quantitative studies of Russian texts complexity including assessment of abstractness of words have been either limited or impossible. In our previous work we presented the first version of a computer-generated Russian dictionary of concrete/abstract words (RDCA) [ 33 ]. The present study is the first research in which the authors apply the Dictionary to assess the complexity of texts. We also view a battery of school textbooks of a particular subject as a good Corpus, since the complexity of textbooks is expected to grow from class to class. The hypothesis of the current study is the following: if the number of abstract words grows from class to class, then the number of abstract words as a metric can be used in assessments of complexity of other text thus extending the sphere of applying RDCA.

Literature review

Psycholinguistic approach to concrete / abstract words

The notion of abstractness/ concreteness (hereinafter A / C) has been a focus of numerous studies [see 4] as the problem of discriminating concrete and abstract words is considered relevant in linguistics, psychology, education, etc. In the modern paradigm, the discrimination of abstractness / concreteness rests on the idea that concrete words denote referents experienced, primarily, through senses, whereas referents nominated with abstract words refer to ideas or concepts [ 5 ]. Psycholinguistic studies suggest a number of differences in processing concrete and abstract words [ 6, 7, 8, 9, 10 ]. Perception and acquisition of abstract words is hindered by lack of ‘word to world’ mapping, i.e. when comprehending an abstract concept a person may fail to make correspondences to real word phenomena (c.f. learning words ‘a car’ and ‘good’) [ 11, 12 ]. The latter argument was also supported in the study of specifics of acquisition and processing of abstract / concrete words by school children [ 9, 13 ]. The research shows that children take longer to acquire abstract words as compared to concrete ones even when it comes to high frequency words [ 14 ]. Due to this fact, P. Schwanenflugel infers that abstract words are harder for children to understand [ 14 ]. Moreover, when tested in a variety of lexical tasks, abstract words are found to exhibit slower reaction time and less accurate responses [ 15, 16, 17 ]. Similar conclusions are found in V. Marian’s (2009) studies who claims that concreteness is found to be a property facilitating words acquisition as concrete words are recognised and processed more rapidly [ 18 ]. Psycholinguistic experiments also indicate ‘that 75% of the words most frequently produced by school-aged children (6–12 years of age) are concrete and it is not until adolescence that children master the majority of abstract words used by adults’ [ 13 ].

The rating of abstract/concrete words as a text complexity parameter

1.3

Methods and tools measuring the degree of word abstractness / concreteness

Many worldwide research aimed at rating words as concrete or abstract involve native speakers who are asked to use a numerical scale as an effective instrument to measure A / C [ 24, 22, 5, 25 ]. A well known dictionary of English words registering A / C ratings of 4000 English words, used in the MRC Psycholinguistic Datase, was compiled based on a 7-point bipolar scale [ 24 ]. The respondents participating in the study tagged each word with an A / C rating from seven (the highest) to one (the lowest). In such a way every word received a rating from 100 to 700. This dictionary is still used in much research on the English language and in cross-linguistic studies [ 26, 5, 27, 28, 29 ].

In another study aimed at defining the A / C ratings of 60,099 English words and 2,940 two word expressions (such as “zebra crossing” and “zoom in”) Brysbaert et. al (2014) asked respondents to assess the abstractness/concreteness the meaning of each word is by using a 5-point rating scale designed from abstract to concrete [ 5 ]. Using the A/C numerical scale, Wang et. al. (2018) computed the degree of abstractness of Chinese words from the context-sensitive model of word embedding in rich contextual information. Word vectors for word distribution study were trained on Reader Corpus (Chinese Corpus). The authors ‘built paradigms of A/C words’ in two steps: (1) respondents’ evaluation of 200 Chinese words as concrete or abstract using ‘– 1 / 0 / 1’ scale, with ‘– 1’ being the most concrete, ‘1’ – the most abstract. (2) Extending obtained results by classification algorithm based on the corpus [ 25 ].

A similar online study was pursued for the Russian language in which respondents were asked to evaluate the C / A ratings of 500 most frequent Russian nouns on a 5point scale. The C /A ratings of each Russian word were computed as an average of all the assessments received in the range from 1 to 5.

As the Dictionary data [ 24 ] and our estimates were computed based on different scales we also processed our estimates with the following formula: f (x) = 100 * (1.5 * ((6-x) - 1)) +1), where x is the value obtained in our survey. After this conversion, the index values range between 100 (the most abstract words) to 700 (the most concrete).

The findings, i.e. lists of words tagged with ratings of abstractness/concreteness are uploaded at https://kpfu.ru/tehnologiya-sozdaniya-semanticheskih-elektronnyh.html and a fragment of the intra-language comparative analysis of the ratings (based on the abovementioned scale) is presented in Figure 1 below [ 30 ]. Researchers designed and developed a number of text complexity software able to match texts with lists of abstract words [ 31, 32 ]. E.g., Coh-Metrix provides the average A / C ratings for content words in a text thus offering. However, replicating large scale studies aimed at assessing the level of A / C for the Russian language was lately a challenge as there was no automated tool defining rank of abstractness of Russian words. In our latest study we identified it, designed and compiled the Russian Dictionary of abstractness/ concreteness [see 33]. 1.4

Russian Dictionary of Abstractness/concreteness

Creating a large dictionary of abstractness by computing interviewees’ assessments is time and energy consuming. Therefore, the dictionary was compiled automatically based on a large corpus of texts, i.e. the Google Books Ngram package (https://books.google.com/ngrams). The fundamental ideas of the dictionary are as follows: (A) Abstract words are more often found along with abstract words, while concrete words are used more frequently with concrete words [ 37 ]. (B) We define the core comprising a certain set of words that are obviously abstract and another set which is obviously concrete and then expand it to the size of the dictionary selecting the entries based on (A). A detailed description of the method is provided in [ 33 ]. As a result we compiled a dictionary of 88.000 words available at https://kpfu.ru/tehnologiya-sozdaniya-semanticheskih-elektronnyh.html. The values of the concreteness / abstract index are in the range from -4.91 to 4.56 for nouns and from -4.01 to 5.33 for adjectives. The A / C index for verbs was not calculated in accordance with the tradition in Russian linguistics not to consider this semantic category for verbs. Fig.2 below shows a fragment of the dictionary. The Dictionary provides researchers and testers with an instrument facilitating not only assessment of texts complexity but leveling and profiling texts for different categories of readers as well. 2

Analysis

The current study was pursued to answer three main research questions:

RQ1: How does the rating of abstractness change across the grades from elementary to high schools?

RQ2: How different or similar are the ratings of abstractness of textbooks on Humanities and textbooks on Science?

RQ3: How does the rating of abstractness of recalls differ from the ratings of abstractness of the original texts?

To answer the research questions we used the Russian Academic corpus, the Corpus of Recalls and designed an automatic tool defining abstractness of Russian texts. 2.1

Materials and methods

In this study we used the Russian Academic Corpus (RAC), a corpus of text-books used in elementary, middle and high schools of the Russian Federation [ 33 ]. As the corpus builders aim at collecting the best possible representative corpus and the list of school textbooks is non-exhaustive, RAC has been a work in progress for over four years and by now reached the size of nearly 3 mln. tokens1 (see Table 1 below). The 1

A token is viewed in the work as an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing. In this article it books included are published between 2006 and 2020 and the body of the Corpus is divided into two sub-corpora: Science Sub-corpus (628920 tokens) and Humanities Sub-corpus (2105058 tokens). Both sub-corpora comprise textbooks specified in the “Federal List of Textbooks Recommended by the Ministry of Education and Science of the Russian Federation to Use in Secondary and High Schools”. The choice of these particular textbooks was caused by a number of reasons: (a) the fact that the texts under study use minimum of non alphabetical symbols, graphs, figures etc., (b) the availability on the textbooks on the Internet (School textbooks and manuals, 2017). The detailed information on the size of the corpus is presented in Table 1 (below). RAC contains 74 documents (textbooks) of all grades and disciplines and as such is considered a representative sample of the population of Russian school textbooks. 2.2

Corpus of Readers’ Recalls

The Corpus of Students’ Recalls was compiled as a side result of the study aimed at evaluating the impact of cohesion on readers’ comprehension [ 35 ].

Of 289 respondents participating in the study we selected 65 with the General Knowledge index2 ranging between 13 and 16. Those were 11-12 year old native Russian speakers. The subjects were individually asked to read one of the informational texts, MT53 (modified text for the 5th Grade #3) and OT53 (original text for the 5th Grade #3), both of about 200 words with which they had no previous experience. The texts were fragments of a Chapter from a textbook on Social Science 5 by Bogolyubov N.F. [ 36 ]. The recalls of the respondents were recorded by experts refers to the total number of words in a text, corpus etc, regardless of how often they are repeated. A type is the class of all tokens containing the same character sequence. 2 Wechsler “General knowledge” Subtest for Children (WISC GK) as it is widely used to assess

IQ to predict or explain school performance. and assessed holistically on its relevance to the task and statistically: we computed the number of tokens and propositions in each recall.

The total size of the corpus is 6473 tokens. As the Corpus is presented in 65 separate texts with the average number of words in recalls being 106.4 (MT53) and 92 (OT53) tokens we view the Corpus is representative enough. The statistics on the Corpus of Readers’ Recalls and selected samples of recalls are uploaded at the site Technologies of electronic dictionaries’ compilation, at https://kpfu.ru/tehnologiyasozdaniya-semanticheskih-elektronnyh. html, last accessed 2020/17/05. 2.3

RusAC as the automatic tool defining abstractness of Russian texts

Text preprocessing (tokenization, etc.) is carried out with Russian TreeTagger (http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/). RusAC processes texts for abstractness/concreteness and readability, which together allow the tool to estimate which of the texts processed is more difficult for comprehension. RusAC ensures the following functions: 1) automatic assessment of text complexity based on two descriptive parameters, i.e. length of words and length of sentences; text complexity is calculated based on the formula proposed in [ 38 ]; 2) assigning words in a text with an A/C rating from the dictionary; 3) saving the results of the analysis. RusAC performs the text analysis of texts saved as doc, txt, rtf files.

Results

Abstractness of school textbooks

In this study we performed a systemic study of abstractness of all the text-books in RAC grouped into the following sets: Primary school textbooks (30, grades 1-4), Middle school textbooks (19, grades 5- 8), High school textbooks (25, grades 9-11). The complete set of textbooks for secondary and high schools comprises 21 books on Humanities and 11 books on Science (Biology). The procedure for computing the mean index of concreteness is as follows: (1) we search the texts for the tokens registered in Russian Dictionary of Abstractness; (2) tag each token with the corresponding index from the Dictionary; (3) compute the average either for the book or for a set of textbooks, (a) in the first case the sum of the indices is divided by the number of tagged tokens in the book and (b) for the set of books the sum of the indices is divided by the total number of tagged tokens in those books. The mean abstractness index (see column 4, Table 2) indicates the following: a) the highest index of concreteness is demonstrated by texts in Biology and primary school books: the concreteness of Biology textbooks for middle school is the highest with +0,49 which is even higher than that of primary school texts which is +0,34 abstractness; b) The index of Social studies textbooks marks the highest level of abstractness of those texts; c) History books are located in the middle of the scale with the “0” score probably due to the fact of an equal incidence of concrete and abstract words. It can be explained by the fact that a pattern of History texts contains descriptions of some artefacts and narration of events which bear a high degree of concreteness.

In general, there is a statistically significant (p-value <0.001) dependence of the abstract index on the Grade level both for the entire collection of textbooks and separately for subcollections of Biology and Literature textbooks. In Social Sciences and History textbooks regularity is not significant. 3.2 The texts offered to the participants of the study for recalls, OT53 and MT53, bear similar average indices of A / C (see Tables 3). It was computed in the same way as the index of textbooks: all the words in the texts registered in Russian Dictionary of Abstractness received a corresponding tag with a rating, all the total sum of the ratings was divided into the number of tagged tokens in the text. The same procedure was implemented for every recall. The results are presented in Table 4 and for the complete data visit the website of Technologies of electronic dictionaries’ compilation, at [ 34 ].

As the table above demonstrates the average index of A/C for recalls is more than that of the source texts which confirms that respondents tend to omit more abstract words and keep the concrete ones in their recalls. As expected, 5th Grade students’ recalls are simpler in terms of traditional metrics and the A / C index. The comparison of A / C indexes of the recalls and the source texts based on Student criterion confirms the hypothesis that the difference is statistically significant as the p-value equals 0.0003.

Conclusion

Abstract words as carriers of the notion of abstractness present a special interest for linguistics, psychology and pedagogy. In Natural Language Processing studies the problem is narrowed to designing and developing tools able to tag words in a text with the corresponding ratings of abstractness/concreteness. The tool evaluating the level of abstractness of Russian texts was a research and an engineering niche. The authors of the article created an automated tool, RusAC, performing computation of the index of concreteness/abstractness. The functions of RusAC are supported by the Russian Dictionary of Concrete and Abstract words with its total size of 88000 tokens compiled in our previous study. Implementation of RusAC on two representative corpora, i.e. Russian Academic Corpus and Corpus of Readers’ Recalls, verified the hypothesis that the incidence of abstract words in a text impacts its complexity as they are taking longer to be processed by readers.

School textbooks were selected to test the proposed approach, since they are graded by levels of complexity from elementary to advanced. Collections of school textbooks are used in studies of various techniques for assessing text complexity in different languages in a number of works [ 39–43 ]. One of the most important issues is to select a battery of classroom books of the same author. This eliminates the influence of the author’s style, concept or pedagogical attitudes on the texts of textbooks of different classes and allows to analyze textbooks of the same author for different grades focusing only in complexity.

The study also confirmed the highest index of concreteness of Science books and primary school books. The Humanities textbooks demonstrate the highest level of abstractness. The index of abstractness grows across grades one through 11. The findings are consistent with the earlier published hypothesis on the impact of abstract terms on text complexity and validate the designed tool. RusAC is freely available for all categories of users.

Currently, the index of abstractness is typically interpreted as a separate parameter calculated for texts but not included in the existing formulas of text complexity. In this way the index of abstractness compares various texts in this aspect without marking the level of text complexity.

The perspective of the study is viewed as extending the number of entries in RDCA and improving its quality. In the next stage of research, we plan to pursue a survey and text recall experiments with students of Grades 9-10 (15-17 years old), thus expanding the database and providing foundation to compare the level of abstractness of texts generated by schoolchildren of different grades. Acknowledgements. The research on the Russian Dictionary of Concrete and Abstract words was supported by the Russian Fund of Basic Research, Grant 19-07-00807. The Survey and Analysis of the present study was supported by the Russian Science Foundation, Grant 18-1800436. The authors also express sincere gratitude to Dr. Artem Zaikin for his assistance in processing the statistical data.

References

1. McCarthy , P. M. , Lewis , G. A. , Dufty , D. F. , McNamara , D. S.: Analyzing writing styles with Coh-Metrix . In Proceedings of the Florida Artificial Intelligence Research Society International Conference. Menlo Park, CA: AAAI Press, 764 - 769 ( 2006 ).

2. Mikk , Ya. A.: Optimization of educational text complexity: for authors and editors [Optimizatsiya slozhnosti uchebnogo teksta: v pomosch avtoram i redaktoram] . Prosveschenie ( 1981 ).

3. Krioni N.K. , Nikin A. D. , Fillipova A.V. : Automated system of academic texts complexity analysis [Avtomatizirovannaya sistema analiza slozhnosti ucebnyh tekstov] . Bulletin of Ufa State Aviation Technical University , Volume 11 , 1 ( 28 ), 101 - 107 ( 2008 ).

4. Reuter K. , Werning , M. , Kuchinke , L. , Cosentino , E.: Reading words hurts: the impact of pain sensitivity on people's ratings of pain-related words . Language and Cognition , 9 ( 3 ), 553 - 567 ( 2017 ).

5. Brysbaert , M. , Warriner , A. B. , Kuperman , V. : Concreteness ratings for 40 thousand generally known English word lemmas . Behavior research methods , 46 ( 3 ), 904 - 911 ( 2014 ).

6. Crutch , S. J. , & Ridgway , G. R. : On the semantic elements of abstract words . Cortex , 48 ( 10 ), 1376 - 1378 ( 2012 ).

7. Kousta , S.-T., Vigliocco , G. , Vinson , D. P. , Andrews , M. , & Del Campo , E. : The representation of abstract words: Why emotion matters . Journal of Experimental Psychology: General , 140 ( 1 ), 14 - 34 ( 2011 ).

8. Paivio , A. : Mind and its evolution: A dual coding theoretical approach . Mahwah, NJ: Erlbaum ( 2007 ).

9. Schwanenflugel , P. J.: Why are abstract concepts hard to understand? The psychology of word meanings , 223 - 250 ( 1991 ).

10. Borghi , A. , Binkofski , F. , Castelfranchi , C. , Cimatti , F. , Scorolli , C. , & Tummolini , L. : The challenge of abstract concepts . Psychological Bullettin , 143 , 263 - 292 ( 2017 ).

11. Gleitman , L. R. , Cassidy , K. , Nappa , R. , Papafragou , A. , Trueswell , J. C. : Hard words . Language Learning and Development , 1 ( 1 ), 23 - 64 ( 2005 ).

12. Yu , C. , Smith , L. : Rapid word learning under uncertainty via cross-situational statistics . Psychological Science , 18 ( 5 ), 414 - 420 ( 2007 ).

13. Vigliocco , G. , Ponari , M. , Norbury , C. : Learning and processing abstract words and concepts: Insights from typical and atypical development . Topics in cognitive science 10 ( 3 ), 533 - 549 ( 2018 ).

14. Schwanenflugel , P. J. (Ed.). The psychology of word meanings . Psychology Press ( 2013 ).

15. Paivio , A. : Dual coding theory: Retrospect and current status . Canadian Journal of Psychology , 45 ( 3 ), 255 - 287 ( 1991 ).

16. Nickels , L. , Howard , D. : Aphasic naming: What matters? Neuropsychologia , 33 ( 10 ), 1281 - 1303 ( 1995 ).

17. Barry , C. , Gerhand , S. : Both concreteness and age-of-acquisition affect reading accuracy but only concreteness affects comprehension in a deep dyslexic patient . Brain and Language , 84 , 84 - 104 ( 2003 ).

18. Marian , V. : Language interaction as a window into bilingual cognitive architecture . Multidisciplinary approaches to code switching, 161 - 185 ( 2009 ).

19. Taylor , L., Weir , C. J.: IELTS collected papers 2: Research in reading and listening assessment (Vol. 2 ). Cambridge University Press ( 2012 ).

20. Fisher , D. , Frey , N. , Lapp , D. : Text complexity: Stretching readers with texts and tasks . Corwin Press. ( 2016 ).

21. Holleman , B. The forbid/allow asymmetry: On the cognitive mechanisms underlying wording effects in surveys (Vol. 16 ). Rodopi. ( 2000 ).

22. Sadoski , M. , Goetz , E. T. , Rodriguez M. : Engaging texts: Effects of concreteness on comprehensibility, interest, and recall in four text types . Journal of Educational Psychology 92.1 , 85 ( 2000 ).

23. Mayer , R. V. : Assessment of the Level of Abstractness of Material Statement of in Natural Sciences School Textbooks . Standards and Monitoring in Education. 1 , 58 - 63 ( 2017 ).

24. Coltheart , M.: The MRC Psycholinguistic Database , Quarterly Journal of Experimental Psychology , 33A , 497 - 505 ( 1981 )

25. Wang , X. , Su , C. , Chen , Y. : A Method of Abstractness Ratings for Chinese Concepts . In UK Workshop on Computational Intelligence , 217 - 226 ( 2018 ).

26. Crossley , S. , Salsbury , T. , McNamara , D. S.: Validating lexical measures using human scores of lexical proficiency. Vocabulary knowledge: Human ratings and automated measures , Amsterdam: John Benjamins, 105 - 134 ( 2013 ).

27. Dellantonio , S. , Mulatti , C. , Pastore , L. , Job , R. : Measuring inconsistencies can lead you forward. the case of imageability and concreteness ratings . Language Sciences , 5 , 708 . ( 2014 ).

28. Troche , J. , Crutch , S. , Reilly , J.: Clustering, hierarchical organization, and the topography of abstract and concrete nouns . Frontiers in psychology, 5 , 360 ( 2014 ).

29. Pastore , L. , Dellantonio , S. , Mulatti , C. , Job , R.: On the nature and composition of abstract (theoretical) concepts: the X-ception theory and methods for its assessment . In Philosophy and Cognitive Science II , 35 - 58 ( 2015 ).

30. Solovyev , V. , Andreeva , M. , Solnyshkina , M. , Zamaletdinov , R. , Danilov A. , and Gaynutdinova , D. : Computing Concreteness Ratings of Russian and English Most Frequent Words: Contrastive Approach . 2019 12th International Conference on Developments in eSystems Engineering (DeSE), Kazan, Russia, 403 - 408 ( 2019 ).

31. Pitler , E. , Nenkova , A. Revisiting readability: a unified framework for predicting text quality . In Kentaro Inui, Jing Jiang, Vincent Ng, Xiaojun Wan (eds). Conference on empirical methods in natural language processing (EMNLP '08) . Association for Computational Linguistics , Stroudsburg, PA, USA, 186 - 195 ( 2008 ).

32. Laposhina , A. : Relevant features selection for the automatic text complexity measurement for Russian as a foreign language. [Analiz relevantnyh priznakov dlya avtomaticheskogo opredeleniya slozhnosti russkogo teksta kak inostrannogo ] In V.P. Selegey (eds). Computational linguistics and intellectual technologies: papers from the annual international conference 'Dialogue' , Issue 17 , 1 - 7 ( 2017 ).

33. Solovyev V. D. , Ivanov V. V. , Akhtiamov R . B.: Dictionary of Abstract and Concrete Words of the Russian Language: A Methodology for Creation and Application . Journal of Research in Applied Linguistics . vol. 10 , 215 - 227 ( 2019 ).

34. Technologies of electronic dictionaries' compilation, https://kpfu.ru/tehnologiyasozdaniya-semanticheskih-elektronnyh.html, last accessed 2020 /19/02.

35. McCarthy , K.S. , McNamara , D.S. , Solnyshkina , M.I. , Tarasova , F. Kh ., Kupriyanov , R.V. : The Russian language test: towards assessing text comprehension . Vestnik Volgogradskogo Gosudarstvennogo Universiteta. Seri︠ia︡ 2 , Iazykoznanie ; Volgograd, 18 ( 4 ), 231 - 247 ( 2019 ).

36. Bogolyubov N.F. : Social Studies Grade 5 [Obschestvoznanie 5 klass]. A textbook for secondary schools . 3rd Edition . Prosveschenie, 127 ( 2013 ).

37. Frassinelli , D. , Schulte im Walde, S.: Distributional interaction of concreteness and abstractness in verb-noun subcategorisation . Proceedings of the 13th International Conference on Computational Semantics - Short Papers. Association for Computational Linguistics , 38 - 43 ( 2019 ).

38. Solovyev , V. , Ivanov , V. , Solnyshkina , M. : Assessment of reading difficulty levels in Russian academic texts: Approaches and metrics , Journal of Intelligent & Fuzzy Systems . 34 ( 5 ), 3049 - 3058 ( 2018 ).

39. Al-Tamimi , A.K. , et al.: AARI: Automatic Arabic readability index . International Arab Journal of Information Technology . 11 ( 4 ), 370 - 378 ( 2014 ).

40. Chen , Y.-T., Chen , Y.-H. , and Cheng, Y.-C.: Assessing Chinese Readability using Term Frequency and Lexical Chain . IJCLCLP . 18 ( 2 ), 1 - 18 ( 2013 ).

41. Chen , Y.-H. and Daowadung , P. : Assessing readability of Thai text using support vector machines . Maejo International Journal of Science and Technology 09 ( 3 ), 355 - 369 ( 2015 ).

42. Si , I. and Callan , J.: A statistical model for scientific readability . In CIKM , 574 - 576 ( 2001 ).

43. Tanaka-Ishii , K. , Tezuka , S. and Terada , H.: Sorting Texts by Readability . Comput. Linguist . 36 ( 2 ), 203 - 227 ( 2010 ).