In this paper, we report a corpus study aimed at testing the hypothesis that bigram distributional information is related to text complexity. We explored a corpus of Russian textbooks on Social studies for middle and high school to examine how the number of bigrams of the core vocabulary correlates with reading levels of texts within the grade range 5 { 11. The corpus contains 45380 sentences from 14 textbooks, written by two independent groups of authors. Each word in the corpus has a part-of-speech tag derived by TreeTagger. Due to the nature of the domain, we focus our study on a single, but high-frequency pattern: `chelovek' (a man) + verb. The ndings are particularly relevant for text complexity theory as they are consistent with the previous results of corpus investigations on a correlation of text complexity with a number of text features.
The research presented in this article is a part of Russian Academic Text
Complexity (RATC) project, which aims at de ning roles of di erent metrics in
academic text complexity analysis and has been carried out at Kazan Federal
University for over a year. The ultimate goal of the project is to provide
cognitive and linguistic pro les of Russian academic texts for middle and high schools
based on the complex linguistic analyses of the latter, describe and determine
correlations between text complexity (or grade levels) and academic text
features [
Related works: Lexical Features in Text Complexity
Studies
In modern text complexity studies, lexico-semantic features of reading texts are
considered valuable and essential metrics in assessing text complexity (see [
At present, lexical metrics are used in over one hundred readability
formulas including those of Spache [
Later on, TTR was acknowledged to be sensitive to the text length, and
numerous revised indices such as Root TTR and Corrected TTR, which take the
logarithm and square root of the text length instead of the direct word count as
denominator were suggested and proved to produce better results [
For the purpose of this study, Russian Academic Corpus (RAC) was compiled of two batteries of textbooks for Russian students: edited by Bogolyubov and by Nikitin. In the Russian Federation the course on Social Studies is taught for 7 years: it starts in Grade 5 when children are typically aged 11 and nishes in Grade 11 where the predominant majority of students is 17 years old. The course nishes with a matriculation exam in the 11th Grade when a certain number of students select the subject for a high-stake exam to continue their education at universities.
To ensure reproducibility of results, we uploaded the corpus on a website thus providing its availability online3. Note, however, that the published texts contain shu ed order of sentences. The sizes of BOG and NIK collections of texts are presented in Table 1.
Tokens Sentences ASL ASW Grade BOG NIK BOG NIK BOG NIK BOG NIK 5-th { 17,221 { 1,499 { 11.49 { 2,35 6-th 16,467 16,475 1,273 1,197 12.94 13.76 2.56 2.71 7-th 23,069 22,924 1,671 1,675 13.81 13.69 2.84 2.70 8-th 49,796 40,053 3,181 2,889 15.65 13.86 2.96 2.88 9-th 42,305 43,404 2,584 2,792 16.37 15.55 3.04 3.00 10-th 75,182 39,183 4,468 2,468 16.83 15.88 3.07 3.12 10-th* 98,034 { 5,798 { 16.91 { 3.05 { 11-th { 38,869 { 2,270 { 17.12 { 3.11 11-th* 100,800 { 6,004 { 16.79 { 3.19 { 3.1
For the convenience, we have preprocessed all texts from the corpus in the same way. Common preprocessing included tokenization and splitting text into sentences. During the preprocessing step we excluded all extremely long sentences (longer than 120 words4) as well as too short sentences (shorter than 5 words) which we consider outliers. Clearly, such sentences can be not outliers at all in another domain, but for the case of school textbooks on Social Studies sentences shorter than 5 words are outliers. Sentence and word-level properties of the preprocessed dataset are presented in Table 1.
Extremely short sentences mostly appear as names of chapters and sections of the books or as a result of incorrect sentence splitting. We omit those sentences, because the average sentence length is a very important feature in text complexity assessment and hence should not be biased due to splitting errors. At the same time sentences with ve to seven words in Russian can still be viewed as short sentences, because the average sentence length (in our corpus) is higher than ten.
The last two columns in Table 1 present well-known features that have been widely exploited for assessment of readability of English texts. Average sentence length (or average words per sentence, ASL) and average syllables per word5, 3 http://kpfu.ru/portal/docs/F1554781210/shu ed.zip 4 Indeed, very long sentences appear as long citations from o cial documents which styles are completely di er from school textbooks 5 Number of syllables in a Russian word can be computed as a number of vowels in the word.
ASW, are the parameters in Flesch and Flesch-Kincaid formulas [
All annotations in the corpus are performed on three levels: text-level, sentencelevel and word-level. At the text-level meta-annotations refer to a number of sentences and a set of tokens6, an author and a grade-level of a given text. 3.2
At the word-level we have part-of-speech tag for each word. POS-tagging has
been performed with the use of the TreeTagger for Russian7. The tagset is
available from the website of the project. As reported in [
The contrastive analysis proved high frequency of content words, mostly
nouns and adjectives thus con rming two texts characteristics: (1) all the texts
studied are quali ed as informative; (2) their narrativity level is much lower
than that of ction texts [
Corpus studies reveal patterns of word use in natural languages [Hanks 2004].
These patterns can be analyzed and applied in text complexity research as a
metric which shows semantic distinctions of words in contexts of di erent
complexity. We computed the Corpus with the aim to retrieve the list of verbs with
which the word `chelovek' (a man) collocates and nd out how the semantic range
and variety of these verbs extend across the textbooks of grade levels 5 { 11.
The semantic range of verbs was analysed based on based on "Russian semantic
dictionary" [
The third class of verbs is a widely versatile system, represented in the following sets: a) verbs denoting mental and emotional activity, as well as the activity of thought and spirit; b) verbs referring to actions related to indivisible spiritual and physical sphere: naming actions - behaviors and contacts, information, as well as actions denoting work, various physical actions and movements. Each of these sets combines cross-branched sections. This class also includes verbs denoting inactive procedural states { physical and physiological. In Tables 2 and 3 we present the classes of verbs which co-occur in all the grade-level texts and, thus, form high-frequency bigrams with the word `chelovek' (a man) in the Corpus.
As we see, the core, i.e. words that most frequently collocate with 'chelovek (a man) is made by the semantic classes of functional verbs (Russian byt' (to be), stat' (become)), modal verbs (moch' (can, be able to)), verbs of possession (imet' (to have, possess). The verbs of other semantic classes are less frequent, though there are two more verbs characterized with above average frequency, i.e. Russian zhit'(to live) and stremit'sya (to seek). Semantic classes of verbs that collocate with the word `chelovek' (a man) in texts of Grade 5 textbook are presented below. The most lexically diversi ed in texts of grade 5 is the class of verbs denoting di erent types of actions. They are represented by the core vocabulary of the Russian language, the words which posses a high frequency in the National language: live, make, develop, sleep, produce, try, turn, socialize, communicate, answer, etc.
The lexico-grammatical constructions or bigrams of verbs co-occurring with the word `chelovek' (a man) functioning as a subject in texts of the 11th grade demonstrated a wider range of the verbs used. The twenty highest ranking verbs are the following: can (25), be (15), be (13), have (12), live (5), become (5), strive (4), understand (4), de ne (4), speak (3), engage in (3), acquire (3), manifest (3), follow (3), anticipate (3), call (3), see (3), create (3), join (3). The contrastive analysis of the bigrams retrieved from the subcorpora of texts of di erent classes (5 { 11) demonstrate that the variety of the semantic groups of the verbs are the same but the number of the verbs with which the word 'chelovek' (a man) collocates in each group is much higher thus the groups are 'densely populated'. E.g. the groups of verbs denoting existence and status increase dramatically acquiring a wider range of verbs. As it is seen in Fig. 4.1 the number of functional verbs increase over the grade level line from 0.28 in Grade 5 to 0.37 in Grade 12 (which in the Graph marks textbooks of the 11th Grade of the Advanced level). 5
In this paper we report a corpus study aimed at testing the hypothesis that bigram distributional information could be a function of text complexity. In a 623782-word corpus of Russian textbooks on Social studies for middle and high school we explore how the number of bigrams of the core vocabulary correlate with reading levels of texts within the grade range 5 { 11. The applied methods and techniques are exempli ed with the bigrams (Noun + Verb) of the most frequent content noun in the corpus, i.e. `chelovek' ( a man). As it is unanimously accepted that (a) texts with low frequency words are more di cult to read and (b) frequency of separate words and bigram has proven high discriminative power among other readability metrics in many languages, in this study we have put our focus on two features providing rich text representation and readability prediction: { the number of bigrams `chelovek' (a man) + VERB and { and the semantic range of the verbs in the bigrams `chelovek' (a man) +
VERB.
The ndings reveal that the distributional patterns of the identi ed core bigrams construct particular semantic classes which tend to increase from grade to grade. The research results may contribute in the following areas of professional knowledge: 1. Textbook writers on Social sciences may be supplied with a better understanding of the prevalence and type of verbs to be used to generate texts of a certain reading pro le. 2. Researchers may be better able to identify type markers and rank reading texts. 3. Identifying the ratio of functional, event (action) and entity (existance) verbs as a marker of text type, genre and complexity can be bene cial for the development of better reading formulas.
The ndings are particularly relevant for text complexity theory as they are consistent with the previous results of corpus investigations on correlation of text complexity and a number of text features. The results of the research may also have major implications for natural language processing in text complexity research. The issue which we decided not to address in the present work is granularity of verb classes. It is obvious that the `appropriate' level of class and subclass granularity may vary from one research to another. In the present study we provided a general purpose classi cation suitable for various purposes, and in the future we intend to re ne and organize semantic classes of verbs into taxonomies of higher degrees of granularity.
This research was nancially supported by the Russian Science Foundation, grant # 18-18-00436, the Russian Government Program of Competitive Growth of Kazan Federal University, and the subsidy for the state assignment in the sphere of scienti c activity, grant agreement # 34.5517.2017/6.7. The Russian Academic Corpus (section 3 up to subsection 3.2 in the paper) was created without supporting by the Russian Science Foundation.