=Paper=
{{Paper
|id=Vol-2790/paper27
|storemode=property
|title=
Exploring Book Themes in the Russian Age Rating System: a Topic Modeling Approach

|pdfUrl=https://ceur-ws.org/Vol-2790/paper27.pdf
|volume=Vol-2790
|authors=Anna Glazkova 
|dblpUrl=https://dblp.org/rec/conf/rcdl/Glazkova20
}}
==
Exploring Book Themes in the Russian Age Rating System: a Topic Modeling Approach
==
<pdf width="1500px">https://ceur-ws.org/Vol-2790/paper27.pdf</pdf>
<pre>
      Exploring Book Themes in the Russian Age
      Rating System: а Topic Modeling Approach*

                                                       6457]
                          Anna Glazkova[oooo-0001-8409-

                     Univeгsity of Tyumen, Тyumen 625003, Russia
                                 a.v.glazkova@utmn.ru



         Abstract. Age rating systems аге created to indicate target ages of
         potential content users based оп information security and text semantics.
         Age ratings are usually given as numbers, which tell us the youngest
         age the content is suitaЫe for. А book or film with а 12+ rating has
         content which is suitaЫe only fог people aged 12 уеагs and over, and а
         book or film with an 18+ rating is suitaЫe fог adults only. Cuпently,
         content assessment in terms of information security is caпied out Ьу
         experts. ln this paper, we empirically compare book abstracts assigned
         to different age ratings using unsupervised topic modeling. We use an
         LDA model to discover topics from а collection of book abstracts. We
         then use statistical methods to study relations between the age rating
         categories assigned to books Ьу experts and the topics oЬtained. We
         believe that our comparisons show interesting and useful findings for age
         rating automation.

         Keywords: Topic modeling • Age restrictions• Age rating • Text classi­
         fication • Statistical methods.


1      lntroduction

Age-based ratings serve as а warning that the content may Ье unsuitaЫe to
children. Moreover, age ratings are used to ensure that entertainment content,
such as books, but also films, games or moЬile apps, is clearly labelled with а
minimum age recommendation.
    While some books are suitaЫe for readers of all ages, others are only suit­
aЫe for older children and young teenagers. А specific portion of books contain
information that is only appropriate for an adult audience.
    Age-based rating systems in different countries differ. Whereas the classifica­
tion systems in Russia, Europe and Germany are based purely on age, the rating
systems in the USA and Australia might Ье interpreted with consideration of
factors other than age. For example, in Australia there are two different 18+
ratings applied to either adult content or pornographic materials [6].
    The Russian Age Rating System (RARS) includes 5 categories of content:

* Supported Ьу the grant of the President of the Russian Federation по. МК-
  637.2020.9.


    Copyright © 2020 for this paper Ьу its authors. Use permitted under Creative
    Commons License Attribution 4.0 lntemational (СС ВУ 4.0).




                                           304
      for children under the age of six (0+);
      for children over the age of six (6+);
      for children over the age of twelve (12+);
      for children over the age of sixteen (16+ );
      prohiЬited for children (18+).
    The RARS was introduced in 2012 when the Federal law of Russian Fed­
eration no. 436-FZ of 2010-12-23 «On Protection of Children from Information
Harmful to Their Health and Development>> was passed [3]. The law prohiЬits
the distribution of <<harmful>> material that depicts violence, unlawful activities,
substance abuse, or self-harm.
    The aim of this article is to compare the topics of the texts assigned to dif­
ferent age rating categories (according to the RARS). Our findings can potential
benefit many text classification applications, such as recommender systems and
text filtering systems. First, through topic analysis, we can gain а deeper under­
standing of the structure of age rating systems. Since the age rating of а book is
currently being assessed empirically, the topic analysis will Ье an important step
towards formalizing this task. Based on the results of the analysis, it will become
more clear which books on which topics most often contain information that is
unsuitaЫe for children or addressed to а particular age audience. In addition,
the results of topic modeling will help to highlight specific topics for different
age groups. This is the reason why topic distributions can Ье used as additional
features for automatic age rating classifiers in our further research.
    The paper is divided into six sections. The first section is introduction. The
second section is concerned with the data preprocessing used for this study and
the description of our topic model. The third part is related to empirical analysis
of topics. The last section is conclusion.


2     Methodology

2.1     Data Pгepaгation

We use а collection of abstracts for books in Russian. These abstracts was col­
lected on the basis of puЬlic online libraries.
    Text preprocessing included the following actions:
      standard steps, such as conversion into lower case letters, removing punctu­
      ations and digits, lemmatization, removing extra white spaces;
      excluding all the stop words using Natural Language Toolkit (NLTK) Python
      library [14] and words with fewer than 3 symbols;
      removing words with TF-IDF weights less than 0.15. TF-IDF (term fre­
      quency-inverse document frequency) is а statistical measure that shows how
      important а word is to а document in а text collection. The TF-IDF value
      increases proportionally to the number of times а word appears in the doc­
      ument and is offset Ьу the number of documents in the text collection that
      contain the word [9]. So, this action allowed us to exclude words typical




                                         305
   of book abstracts (these are usually the words <<book>>, <<author», «reader»,
   etc.). The threshold 0.15 was chosen empirically. During the study, we tested
   values in the range [0.1, 0.3] with an increment of 0.05 and compared the co­
   herence of the models;
 - excluding personal names to delete the mentions of authors and characters.
   This allowed our topic model to form themes according to the semantic prox­
   imity of abstracts, and not according to the belonging of books to one author
   or the coincidence of the characters names. То recognize named entities, we
   used the Natasha Python library [13];
 - we have comblned common phrases (with а frequency of mutual occurrence
   of more than 5) into Ьigrams using the Gensim library [12].
      Some statistics of the data are summarized in ТаЫе 1.

                       ТаЫе 1. Some characteristics of the data.


               Category Number of texts Avg number of words per text
                  о+            53                   45.33
                  6+           3107                  72.03
                 12+           3110                  72.03
                 16+           3989                  91.76
                 18+           3986                  76.36




2.2     LDA
То discover topics from the collection of abstracts, we choose to apply standard
Latent Dirichlet Allocation (LDA) [15]. Topic modeling is а type of statistical
modeling for recognizing main topics in а collection of documents. As а rule,
topic modeling is based on LDA, а hierarchical network that relates words and
documents through latent topics [1]. Topics are characterized Ьу diverse fre­
quency of words. The document is presented as а bag-of-words approach, and
the topic looks like а set of words ranked in decreasing order of their probabll­
ities. LDA topic model were applied to analyze various subject areas, such as
social media analysis [17,19,18], analysis of emails [5], news [7,16], fiction texts
[10], and others.
    We designed а topic model for 100 topics, which reflect the main content
of the collection of abstracts. Our approach to choosing the optimal number of
topics was to build topic models with different values of number of topics and
pick the number that gives the highest coherence value (Fig. 1).

2.3     Topic Distribution Estimation
We calculated the topic distribution for each document in the collection. It is
а vector of length equal to the number of topics in the LDA model. The topic




                                         306
                       О,б�---------<1,-51--------

                       0,5

                       0,4 1--

                  j!   0, 3

                       0,2

                       О, 1 _,__ _,__,____,___,   •---■--..--..-..---■--..-




                                            Торiс(Щ




                                 Fig. 1. Coherence values.




distribution vector shows how а document extracted from the collection of ab­
stracts corresponds to each topic. Each element of the topic distribution vector
is а number from О to 1, where О is а complete non-шatch and 1 is an abso­
lute match. Then, we got the averaged topic distribution vector for each age
rating category to oЬtain the average values of topic distribution for а group of
documents:
                                                                                (1)
where п is а number of age categories, k is а number of topics, Wx is а severity
value of а particular topic for the age category, х Е [1, k].
   For the oЬtained averaged topic distribution vectors а1, а2, ..., an , we deter-
mined:
   the most typical topics for each age category. We calculated the standard
   deviations for each vector а1, а2, ..., an . Next, we marked values that ex­
   ceeded the three standard deviations as corresponding to the most typical
   topics for the category (the three sigma rule).
   age-specific topics that are typical mainly for one age rating category. For
   each topic, we have а vector
                                                                                (2)
   where Vz (z Е [1, m]) from the vector ti corresponds to the value щ from
   the vector a z .
   In the case when the number of observations (in our case, the number of
   categories) is rather small, we cannot use the three-sigma rule to search for
   outliers. In this case, it is necessary to use other statistical techniques for
   small-sized samples [4]. We then applied the Dixon's Q-test [2] to determine
   age-specific topics. The Dixon's Q-test is the simpler test that allows us to
   examine if one observation from а small set of replicate observations (typi­
   cally the number of observations is шоrе than 3 and no шоrе than 10) is an
   outlier or not [11].
   First, we arrange the values v1,v2,---,Vm for each vector ti in ascending
   order (from the lowest to the highest value):
                                                                                (3)




                                           307
        Therefore, we estimate the experimental Q-value:

                                                                                        (4)

        In the next step, we compare the calculated Q exp value to the tabulated
        critical Q crit value for а chosen confidence level.


3       Empirical Analysis of Topics
As mentioned earlier, the purpose of this study is to compare the topics of texts
assigned to different age rating categories (according to the RARS). Understand­
ing the differences between categories will help us highlight topics specific to the
categories. Thus, in the future, we will Ье аЫе to use the empirical results of
topic modeling as а set of additional features for automatic text classification
based on the age of the addressee.

3.1       Distribution of Topics
According to the Russian law, books of any genre in printed and electronic
versions are subject to labeling in accordance with age restrictions. The law
distinguishes five age categories: 0+, 6+, 12+, 16+ and 18+ (prohiЬited for
children). The difference between the books of these age categories is determined
Ьу the presence of scenes of violence, cruelty, descriptions of antisocial actions,
diseases, the mention of narcotic substances, alcoholic beverages, tobacco and
swear words. ТаЫе 2 shows the most common topics for age categories based on
our topic model.

                           ТаЫе 2: Тор-5 topics per category.

    №                                      Keywords
                                             о+
    1 <<Сказка>>,       <<волшебный>>,       <<сказочный>>,   <<серия>>,    <<ЯПОНСКИЙ>>,
      «украинский», «индийский», «арабский» ( «tale», «magic>>, «fairy
      tale>>, <<series>>, <<Japanese», <<Ukrainian>>, <<lndian>>, <<AraЬic>>)
    2 <<Рассказ>>, <<сказка>>, <<тетрадь>>, <<маленькие>>, <<занятие>>, <<пособие>>,
      «интеллект», «серия» ( <<story», «fairy tale», «notebook», «small», «les-
      son>>, <<manual>>, <<intelligence>>, <<series>>)
    3 <<Любовь», <<рассказ>>, <<СТИХ>>, «город>>, «сборнию>, <<сказка>>,
      <<Пересказ>>, <<перевод>> ( <<love>>, <<story>>, <<verse>>, <<City>>, <<collection>>,
      <<fairy tale>>, <<retelling>>, <<translation>>)
    4 «Ребенок»,           «планета»,        «написать»,     «стихотворная - форма>>,
      <<рассказывать>>, <<сказка>>, <<русский>>, <<приключение>> ( <<child,>,
      <<planet>>, <<to write>>, <<poetic form>>, <<to tell>>, <<fairy tale>>, <<Russian>>,
      <<adventure>>)




                                            308
5   <<Рассказ», <<ребенок», «поучительный», <<чтение», «церковный>>,
    <<любить>>, <<Богородица>>, <<учебнию> ( <<story>>, <<child>>, <<instructive>>,
    <<reading>>, <<church>>, <<to love>>, «Mother of God>>, <<textbook>>)
                                          6+
1 <<Школьный_ возраст>>,                  <<рассказ>>,        <<младший_ ШКОЛЬНИК>>,
    <<детский_ писатель>>, <<ребенок>>, <<известный>>, <<сборник», <<ПОВеСТЬ>>
    ( <<school age>>, <<story>>, <<younger school student>>, <<children's writer>>,
    <<child>>, <<famous>>, <<collection>>, <<story>>)
2   <<Сказка>>,       <<волшебный>>,       <<сказочный>>,     <<серия>>,   <<ЯПОНСКИЙ>>,
    <<украинский», <<ИНДИЙСКИЙ>>, <<арабский>> ( <<tale>>, <<magic>>, <<fairy
    tale>>, <<series>>, <<Japanese», <<Ukrainian>>, <<lndian>>, <<AraЬic>>)
3   <<Упражнение>>, <<ЯЗЫК>>, <<закрепление>>, <<брошюра>>, <<английский>>,
    <<самоучитель>>, <<лингвистический>>, <<испанский>> ( <<exercise», <<lan-
    guage>>, <<fastening», <<brochure>>, <<English», <<checkbook>>, <<linguistic>>,
    <<Spanish>>)
4   <<Ребенок»,          <<планета>>,     <<написать>>,      «стихотворная_ форма>>,
    <<рассказывать>>, <<сказка>>, <<русский>>, <<приключение>> ( <<child>>,
    <<planet>>, <<to write>>, «poetic form», «to tell», <<fairy tale», «Russian>>,
    <<adventure>>)
5   <<Стихотворная_ форма>>, «планета>>, <<рассказывать>>, <<растение>>,
    <<алфавит>>, <<иллюстрация», <<стихотворение» ( <<Poetic form», «planet>>,
    <<to tell», <<plant», <<alphabet>>, <<illustration>>, <<poem>>)
                                         12+
1 <<Школьный_ возраст>>,                  <<рассказ>>,         «младший - ШКОЛЬНИК>>,
    <<детский_ писателЬ>>, <<ребенок>>, <<известный>>, <<сборник», <<ПОВеСТЬ>>
    ( «school age», «story», «younger school student», «children's writer»,
    <<child>>, <<famous>>, <<collection>>, <<story>>)
2   <<Сказка>>,       <<волшебный>>,       <<сказочный>>,      <<серия>>,  <<ЯПОНСКИЙ>>,
    «украинский», «индийский», «арабский» ( «tale», «magic», «fairy
    tale>>, <<series>>, <<Japanese», <<Ukrainian>>, <<lndian>>, <<AraЬic>>)
3   <<Упражнение>>, <<ЯЗЫК>>, <<закрепление>>, <<брошюра>>, <<английский>>,
    «самоучитель», «лингвистический», «испанский>> ( «exercise», «lan-
    guage>>, <<consolidation>>, <<brochure>>, <<English>>, <<checkbook>>, <<linguistic>>,
    <<Spanish>>)
4   <<Ребенок»,          <<планета>>,     <<написаты>,       <<стихотворная - форма>>,
    <<рассказывать>>, <<сказка>>, <<русский>>, <<приключение>> ( <<child>>,
    <<planet>>, <<to write>>, <<poetic form>>, <<to tell>>, <<fairy tale>>, <<Russian>>,
    <<adventure>>)
5   <<Стихотворная_ форма>>, «планета>>, <<рассказывать>>, <<растение>>,
    <<алфавит>>, <<иллюстрациЯ>>, <<стихотворение>> ( <<Poetic form>>, <<planet>>,
    <<to tell», <<plant», <<alphabet>>, <<illustration>>, <<poem>>)
                                         16+




                                         309
  1   <<Жадный>>, <<погон>>, <<замучить», «милость», «заметить>> » счастье>>,
      <<свадебный>>, «глотать>> ( <<greedy>>, <<epaulet>>, <<to torture>>, <<mercy>>, <<to
      notice» «happiness», «wedding», «to swallow»)
  2   Фронт», «мужчина», «трагедия>>, «гипотетический>>, «догнать>>,
      <<былой>>, <<предшественник>>, <<достижение>> ( <<forefront>>, <<man>>,
      <<tragedy>>, «hypothetical», «to catch up», <<past», «predecessor>>,
      <<achievement>>)
  3   <<Метод>>,         <<французский>>,         «упрощение>>,          <<повторяемость>>,
      <<заучивание», «уникальность», «текст>>, «лексический>> ( «method>>,
      <<French», <<simplification>>, <<repeataЬility>>, <<memorization>>, <<unique-
      ness>>, <<text>>, <<lexical>>)
  4   <<Детский>>,        <<ребенок>>,     <<отношение»,      <<педагог>>,      <<искусство>>,
      <<воспитание>>, <<зависимость>>, <<женщина>> ( <<children's>>, <<child>>,
      <<relation>>, <<teacher>>, <<art>>, <<upbringing>>, <<addiction>>, <<woman>>)
  5   <<Книга>>, <<прошлое>>, <<человек>>, <<любовь>>, <<ГОрЬКИЙ>>, <<терять>>,
      <<ПРОЙТИ>>, <<МИР>> ( <<book>>, <<past>>, <<man>>, <<love>>, <<Ьitter>>, <<to lose>>, <<to
      pass>>, <<реасе>>)
                                               18+
  1   <<Чувство>>, <<отношение>>, <<новелла», <<представлять>>, <<реальность>>,
      <<интрига>>, <<удовольствие>>, <<фантазиЯ>> ( <<feeling>>, <<relation>>, <<short
      story», <<to imagine>>, <<reality», <<intrigue», «pleasure», <<fantasy»)
  2   <<Зодиак>>, <<гороскоп>>, <<светить>>, <<знак», <<поведение>>, <<ТИП>>,
      <<предопределять>>, <<сексуальный>> ( <<Zodiac>>, <<horoscope>>, <<to shine>>,
      <<sign>>, <<behavior», <<type>>, «to determine», <<sexual»)
  3   <<Книга>>, <<прошлое>>, <<человек>>, <<любовь>>, <<горький>>, <<теряты>,
      <<пройти>>, <<мир>> ( <<book>>, <<past>>, <<man>>, <<love>>, <<Ьitter>>, <<to lose>>, <<to
      pass>>, «реасе>>)
  4   <<История», «друг>>, «жанр», «смешной», «личность», «диалог>>,
      <<герой>>, <<весёлый>> ( <<history>>, <<friend>>, <<genre>>, <<funny>>, <<personality>>,
      <<dialogue>>, <<hero», <<funny»)
  5   <<Знание>>, «отличие>>, «мужчина>>, <<женщина», <<мир», «здоровье>>,
      <<действие>>, <<тело» ( <<knowledge>>, <<difference>>, <<man>>, <<woman>>,
      <<world>>, <<health>>, <<action>>, <<body»)

    The books for children under 6 years old (О+) may contain episodic unnatu­
ralistic images justified Ьу the genre or descriptions of physical or psychological
violence, provided that the victim is compassionate and happy ending. The pre­
vailing topics in the О+ category are fairy tales of the world ( topic 1), developing
children's benefits (2), poems about the world around us (3-4) and Christian lit­
erary works for children (5).
    ln the books for children over the age of 6 (6+), non-naturalistic images or
descriptions of human diseases, accidents, catastrophes or violent death without
demonstrating their consequences are permissiЫe. The books for children over
12 (12+) may contain scenes of violence or murder, descriptions of illnesses,
disasters, but without details. Alcohol, tobacco and drug use may Ье present,




                                             310
but should Ье condemned. А schematic description of the hugs and kisses of men
and women may Ье present. These two categories are described Ьу similar topics
in our topic model. These are short stories and tales for primary and secondary
school age (topic 1), fairy tales of the world (2), study guides (3) and poems for
children (4-5).
    The books for children over 16 (16+) may contain scenes of illnesses, disasters
without detailed descriptions. Violence, alcohol and drug use can Ье described,
but should Ье condemned. Rough words may Ье present, with the exception
of swear words. Scenes of sexual relations cannot Ье described with anatomical
details. In our example, this category is represented Ьу military (topics 1-2) and
human condition (5) fiction, teaching aids (3), psychological and pedagogical
literature (4).
    А book should Ье marked with the 18+ label if the book contains а natu­
ralistic description of illnesses, disasters, non-condemned drug and alcohol use,
naturalistic scenes of sexual relations, non-traditional relationships, obscene lan­
guage, scenes that encourage suicide. In our topic model, love stories (topic 1),
horoscopes (2), human condition fiction (3-4) and possiЬly relationship psychol­
ogy books (5) were detected in this category.

3.2    Most Common Topics for Categories
ТаЫе 3 shows most of the common topics discovered using the three sigma rule.
These topics are most widely represented in their category. The column <<Main
topic>> shows the percentage of texts for which this topic is the main, i.e. it has
the largest proportion in the topic distribution.

                 ТаЫе 3: The most typical topics for categories.

 Category                                  Keywords                                    Main
                                                                                       topic

      о+     «tale», «magic», «fairy tale», «series», «Japanese», 7,553/с
             <<Ukrainian>>, <<lndian>>, <<AraЬic>>
      16+    <<greedy>>, <<epaulet>>, <<to torture>>, <<mercy>>, <<to notice>> <<hap- 1,553/с
             piness», <<wedding>>, <<to swallow»
      16+    <<forefront>>, <<man», «tragedy>>, <<hypothetical>>, <<to catch up>>, 1,533/с
             <<past>>, <<predecessor», <<achievement>>
      18+    <<feeling>>, <<relation>>, <<short story>>, <<to imagine>>, <<reality>>, 1, 783/с
             <<intrigue», <<pleasure», <<fantasy>>
      о+     <<story>>, <<fairy tale>>, <<notebook», <<small>>, <<lesson», <<man- 5,663/с
             ual>>, «intelligence», <<series»
      6+     <<school age>>, «story>>, «younger school student>>,                     2,453/с
      12+    <<children's writer», <<child>>, <<famous>>, <<collection>>, <<story>> 2,483/с




                                          311
3.3    Age-specific Topics
In this subsection, we provide the topics that are typical mainly for one age
rating category using the Dixon's Q-test. The tabulated Q cr it value is equal to
0.642 for confidence level 90% and m = 5.
     We noticed that documents in category 0+ are largely mono-thematic. At the
same time documents of other categories are usually mixtures of topics. There­
fore, our topic model has many specific topics for texts from the 0+ category. In
ТаЫе 4, we present the list of the most common age-specific topics in our data
set.
     As it would Ье logical to assume, age-specific topics generally relate to chil­
dren's books, as well as to specific literature from the 18+ category (in our case,
literature on business and success).

                             ТаЫе 4. Age-specific categories.

Categ >ГУ                                  Keywoгds                                         Qcrit
18+ «success», «activity», «man», «business», «city», «collection», «еuго- 0,94
       pean>>, <<necessary»
о+ <<story>>, <<fairy tale>>, <<notebook>>, <<small», <<lesson>>, «manual>>, <<intelli- 0,93
       gence>>, <<series»
о+ «fairy tale>>, «plunge», «fairytale atmosphere», «emotion», <<character», 0,85
       <<interesting>>, <<exciting», <<impression>>
о+ <<child>>, <<planet», <<to write>>, <<poetic form>>, <<to tell>>, <<fairy tale>>, <<Rus- 0,78
       sian>>, <<adventure>>
о+ <<tale», <<magic>>, <<fairy tale>>, <<series», <<Japanese>>, «Ukrainian>>, <<lndian>>, 0,76
       <<Arablc>>
о+ <<malachite Ьох>>, 1 <<tale», <<storyteller>>, «humor>>, <<funny>>, <<forest>>, 0,75
       <<hero>>, <<сараЫе»



4     Conclusion
In this paper, we empirically analyzed the topics of texts assigned to different age
rating categories. We introduced the distribution of topics for age categories and
the list of the most common topics for categories and age-specific topics. These
list of topics were oЬtained using statistical methods. Our analysis confirmed the
existing differences between the categories and demonstrated that topic models
can Ье а good source of features for age rating identification. ln our future work,
we will try to develop а machine learning classifier for automatically determining
the text age rating.
1
    This topic is рrоЬаЫу related to <<The Malachite Вох» ( <<Малахитовая шкатулка>>).
    This is а book of fairy tales and folk tales of the Ural region of Russia compiled
    Ьу Pavel Bazhov and puЬlished from 1936 to 1945. It is wгitten in contempoгary
    language and Ыends elements of everyday life with fantastic creatures of mountains
    and forests. This book significantly popularized the folklore of the Urals [8].




                                             312
References
1. Вlei, D. М., Ng, А. У., Jordan, М. 1. Latent dirichlet allocation. ln: Journal of
   machine Learning research. Vol. 3(Jan). Рр. 993-1022 (2003).
2. Dixon, W. J.: Processing data for outliers. Biometrics 1(9), 74-89 (1953).
   https://doi.org/10.2307/3001634
3. Federal Law of December 29, 2010 N 436-FZ (as amended on Мау 1, 2019) <<On
   the Protection of Children from Information Harmful to Their Health and Devel­
   opment>> (as amended and additional, entered into force on October 29, 2019) [Fed­
   eral'nyj zakon ot 29.12.2010 N 436-FZ (red. ot 01.05.2019) <<0 zashchite detej ot in­
   formacii, prichinyayushchej vred ih zdorov'yu i razvitiyu>> (s izm. i dop., vstup. v silu
   s 29.10.2019).], http://www.consultant.ru/document/cons_doc_LAW _108808/.
   Last accessed 7 Apr 2020.
4. Glazkova, А., Kruzhinov, V., Sokova, Z.: Dynamic Topic Models for Retrospective
   Event Detection: А Study on Soviet Opposition-Leaning Media. ln: lnternational
   Conference on Analysis of Images, Social Networks and Texts, рр. 145-154, Springer,
   Cham (2019). https://doi.org/10.1007/978-3-030-37334-4_13
5. Gong, Н., You, F., Guan, Х., Сао, У., Lai, S.: Application of LDA Topic Model
   in E-Mail Subject Classification. ln: 2018 International Conference on Тransporta­
   tion & Logistics, Information & Communication, Smart City. Atlantis Press (2018).
   https://doi.org/10.2991/tlicsc-18.2018.24
6. How are age-based gaming ratings set?, https://www.kaspersky.com/Ыog/gaming­
   age-ratings/11647/. Last accessed 7 Apr 2020.
7. Hu Х.: News hotspots detection and tracking based on LDA topic model. In: 2016
   lnternational Conference on Progress in lnformatics and Computing (PIC). IEEE,
   рр. 248-252 (2016). https://doi.org/10.1109/pic.2016.7949504
8. Ilyasova, R. S.: Dialectal lexis of Р. Р. Bazov's narrations <<Malachite casket». Letters
   of the Chechen State University 3(11), 103-107 (2018).
9. Manning, C.D.; Raghavan, Р.; Schutze, Н.: Scoring, term weighting, and the
   vector space model. Introduction to lnformation Retrieval. р. 100 (2008).
   https://doi.org/10.1017/СВО9780511809071.007.
10. Mitrofanova, О., Sedova, А.: Topic Modelling in Parallel and Compara­
   Ыe Fiction Texts (the case study of English and Russian prose). ln:
   Proceedings of the International Conference IMS-2017, рр. 175-180 (2017).
   https://doi.org/10.1145/3143699.3143734
11. Raschka, S.: А questionaЫe practice: Dixon's Q test for outlier identification,
   https://sebastianraschka.com/Articles/2014_dixon_test.html. Last accessed 13 Apr
   2020. https://doi.org/10.13140/2.1.3000.0004
12. Rehurek, R., Sojka, Р.: Gensim-statistical semantics in python, Retrieved from
   genism. org. (2011).
13. Natasha - high quality compact solution for extracting named entities from news
   articles in Russian, https://natasha.github.io/ner/. Last accessed 26 Jul 2020.
14. Loper, Е., Bird, S.: NLTK: the natural language toolkit, arXiv preprint cs/0205028
   (2002).
15. Vorontsov, К., Potapenko, А.: Thtorial on probabllistic topic modeling: Additive
   regularization for stochastic matrix factorization. ln: lnternational Conference on
   Analysis of Images, Social Networks and Texts, рр. 29-46, Springer, Cham (2014).
   https://doi.org/10.1007/978-3-319-12580-03.
16. Wang, Н., Wang, J., Zhang, У., Wang, М., Мао, С.: Optimization of Topic Recog­
   nition Model for News Texts Based on LDA. Journal of Digital Information Man­
   agement 5(17), 257 (2019). https://doi.org/10.6025/jdim/2019/17/5/257-269




                                           313
17. Yang, S. and Zhang, Н.: Text mining of Тwitter data using а latent Dirichlet
   allocation topic model and sentiment analysis. In: Int. J. Comput. Inf. Eng, 12,
   рр.525-529 (2018).
18. Zhao, F., Zhu, У., Jin, Н., Yang, L. Т.: А personalized hashtag rec­
   ommendation approach using LDA-based topic model in microЬlog en­
   vironment //Future Generation Computer Systems 65, 196-206 (2016).
   https://doi.org/10.1016/j.future.2015.10.012
19. Zhao, W. Х., Jiang, J., Weng, J., Не, J., Lim, Е. Р., Yan, Н., Li, Х.: Com­
   paring twitter and traditional media using topic models. In: European confer­
   ence on information retrieval, рр. 338-349. Springer, Berlin, Heidelberg (2011).
   https://doi.org/10.1007/978-3-642-20161-53_4




                                       314

</pre>