Age rating systems аге created to indicate target ages of potential content users based опinformation security and text semantics. Age ratings are usually given as numbers, which tell us the youngest age the content is suitaЫe for. А book or film with а 12+ rating has content which is suitaЫe only гfо people aged 12 уеагs and over, and а book or film with an 18+ rating is suitaЫe fог adults only. Cuпently, content assessment in terms of inrfomation security is caпied out Ьу experts. ln this paper, we empirically compare book abstracts assigned to diferent age ratings using unsupervised topic modeling. We use an LDA model to discover topics from а collection of book abstracts. We then use statistical methods to study relations between the age rating categories assigned to books Ьу experts and the topics oЬtained. We believe that our comparisons show interesting and useful ifndings for age rating automation.
lntroduction Age- based ratings serve as а warning that the content may Ье unsuitaЫe to children. Moreover, age ratings are used to ensure that entertainment content, such as books, but also lfims, games or moЬile apps, is clearly labelled with а minimum age recommendation.
While some books are suitaЫe for readers of all ages, others are only suit aЫe rfo older children and young teenagers. А specific portion of books contain information that is only appropriate for an adult audience.
Age-based rating systems in diferent countries difer. Whereas the classifica tion systems in Russia, Europe and Germany are based purely on age, the rating systems in the USA and Australia might Ье interpreted with consideration of factors other than age. For example, in Australia there are two diferent 18+ ratings applied to either adult content or pornographic materials [6].
The Russian Age Rating System (RARS) includes 5 categories of content: по. МК
Ьу its hautors. Use permited 4.0 lntemational (СС ВУ 4.0).
under Creative ofr ofr ofr ofr children under the age of six (0+); children over the age of six (6+); children over the age of twelve (12+); children over the age of sixteen (16+); prohiЬited rfo
children (18+).
The RARS was introduced in 2012 when the Federal law of Russian Fed eration no. 436-FZ of 2010-12-23 «On Protection of Children from
to Their Health and Development>> was passed [3]. The law prohiЬits the distribution of <<harmful> material that depicts violence, unllfuaw activities, substance abuse, or self-harm.
The aim of this article is to compare the topics of the texts assigned to dif refent benefit age rating categories (according to the RARS). Our nfidings can potential many text classification
applications, such as recommender systems and text filtering
systems. First, through topic analysis, we can gain а deeper under standing of the structure of age rating systems. Since the age rating of а book is currently being assessed empirically, the topic analysis will Ье an important step towards rfomalizing
this task. Based on the results of the analysis, it will become more clear which books on which topics most often contain information that is unsuitaЫe for children or addressed to а particular age audience. In addition, the results of topic modeling will help to highlight specific topics rfo diferent age groups. This is the reason why topic distributions can Ье used as additional efatures for
automatic age rating classifiers in our further research.
The paper is divided into six sections. The rfist section is introduction. The second section is concerned with the data preprocessing used for this study and the description of our topic model. The third part is related to empirical analysis of topics. The last section is conclusion. 2 eW 2.1
Methodology
Data Pгepaгation use а collection of abstracts for books in Russian. These abstracts was col lected on the basis of puЬlic online libraries.
xeTt
preprocessing included the following actions: standard steps, such as conversion into lower case letters, removing punctu ations and digits, lemmatization, removing extra white spaces; excluding all the stop words using Natural Language oTolkit (NLTK) Python library [14] and words with fewer than 3 symbols; removing words with TF-IDF weights less than 0.15. TF-IDF (term fre quency-inverse document frequency)
is а statistical measure that shows how important а word is to а document in а text collection. The TF-IDF value increases proportionally to the number of times а word appears in the doc ument and is ofset
Ьу the number of documents in the text collection that contain the word [9]. So, this action allowed us to exclude words typical of book abstracts (these are usually the words <book>>, <<author», «reader», etc.). The threshold 0.15 was chosen empirically. During the study, we tested values in the range [0.1, 0.3] with an increment of 0.05 and compared the co herence of the models; - excluding personal names to delete the mentions of authors and characters.
This allowed our topic model to rfom
themes according to the semantic prox imity of abstracts, and not according to the belonging of books to one author or the coincidence of the characters names. То recognize named entities, we used the Natasha Python library [13]; - we have comblned common phrases (with а frequency of mutual occurrence of more than 5) into Ьigrams using the Gensim library [12].
Some statistics of the data are summarized in ТаЫе 1.
аТЫе 1.
Some characteristics of the data.
Category Number of texts gAv
number of words per text о+ 6+ 12+ 16+ 18+ 53 3107 3110 3989 3986 45.33 72.03 72.03 91.76 76.36 2.2
LDA То discover topics from the collection of abstracts, we choose to apply standard Latent Dirichlet Allocation (LDA) [15]. pToic modeling is а type of statistical modeling for
recognizing main topics in а collection of documents. As а rule, topic modeling is based on LDA, а hierarchical network that relates words and documents through latent topics [1]. Topics are characterized Ьу diverse fre quency of words. The document is presented as а bag-of-words approach, and the topic looks like а set of words ranked in decreasing order of their probabll ities. LDA topic model were applied to analyze various subject areas, such as social media analysis [17,19,18], analysis of emails [5], news [7,16], cfition texts [10], and others.
We designed а topic model for 100 topics, which reflect the main content of the collection of abstracts. Our approach to choosing the optimal number of topics was to build topic models with diefrent values of number of topics and pick the number that gives the highest coherence value (Fig. 1). 2.3 Topic
Distribution Estimation eW calculated the topic distribution rfo each document in the collection. It is а vector of length equal to the number of topics in the LDA model. The topic 0,4 1j! 0,3
------ -<1 ,-51 -------О,1 _,_ __,__,,_ •--■.-■
-.Торiс(Щ
distribution vector shows how а document extracted from the collection of ab
stracts corresponds to each topic. Each element of the topic distribution vector
is а number from О to 1, where О is а complete non-шatch and 1 is an abso
lute match. Then, we got the averaged topic distribution vector for each age
rating category to oЬtain the average values of topic distribution for а group of
documents:
(
For the oЬtained averaged topic distribution vectors а1, а2, ... , an, we determined: the most typical topics for each age category. We calculated the standard deviations for each vector а1, а2, ... , an . Next, we marked values that ex ceeded the three standard deviations as corresponding to the most typical topics for the category (the three sigma rule). age-specific topics that are typical mainly for one age rating category. For each topic, we have а vector where Vz (z Е [1, m]) from the vector ti corresponds to the value щ from the vector az.
In the case when the number of observations (in our case, the number of categories) is rather small, we cannot use the three-sigma rule to search for outliers. In this case, it is necessary to use other statistical techniques for small-sized samples [4]. We then applied the Dixon's Q-test [2] to determine age-specicfi topics. The Dixon's Q-test is the simpler test that allows us to examine if one observation from а small set of replicate observations (typi cally the number of observations is шоrе than 3 and no шоrе than 10) is an outlier or not [11].
First, we arrange the values v1,v2,---,V m for each vector ti in ascending
order (from the lowest to the highest value):
(
Empirical Analysis of Topics As mentioned earlier, the purpose of this study is to compare the topics of texts assigned to diefrent age rating categories (according to the RARS). Understand ing the diferences between categories will help us highlight topics specific to the categories. Thus, in the future, we will Ье аЫе to use the empirical results of topic modeling as а set of additional features for automatic text classification based on the age of the addressee. 3.1
Distribution of Topics According to the Russian law, books of any genre in printed and electronic versions are subject to labeling in accordance with age restrictions. The law distinguishes vfie age categories: 0+, 6+, 12+, 16+ and 18+ (prohiЬited for children). The diefrence between the books of these age categories is determined Ьу the presence of scenes of violence, cruelty, descriptions of antisocial actions, diseases, the mention of narcotic substances, alcoholic beverages, tobacco and swear words. ТаЫе 2 shows the most common topics for age categories based on our topic model.
ТаЫе 2: Тор-5 topics per category.
о+ 1 <Сказка>>, <волшебный>>, <<сказочный>>, <<серия>>, <<ЯПОНСКИЙ>>, «украинский», «индийский», «арабский» («tale», «magic>>, «fairy tale,> <<series>, <<J apanese», << Ukrainian>, <<lndian>>, <AraЬic>>) 2 <Рассказ>>, <<сказка>>, <тетрадь>>, <<маленькие>>, <<занятие>>, <пособие>>, «интеллект», «серия» ( <<story», «fairy tale», «notebook», «small», «lesson>>, m<anual>>, <intelligence>, <<series>) 3 <Любовь», <рассказ>>, С<ТИХ>>, «город>>, «сборнию>, <<сказка>>, <<Пересказ>>, <<перевод>> (<<love>>, <<story>>, <<verse>>, <City>>, <<collection>>, <fairy tale>>, <retelling>>, <translation>>) 4 «Ребенок», «планета», «написать», «стихотворная- ф(<<cоhрiмldа,>>>,, <рассказывать>>, <<сказка>, <<русский>>, <приключение>> <planet>, <to write>, <poetic form>, <<to tell>>, <<fairy tale>, <<Russia n>>, a<dventure>>) 5 <Рассказ», <<ребенок», «поучительный», <<чтение», «церковный>>, л<юбить>>, Б<огородица>>, <<учебнию> ( s<tory>>, <<child>>, <instructive>>, <reading>>, <church>>, t<o love>, «Mother of God>, <textbook>>)
6+ 1 <Школьный _возраст>>, <рассказ>>, <младший _ШКОЛЬНИК>>, д<етский _писатель,> <ребенок>>, <<известный>>, <сборник», <<ПОВеСТЬ>> ( <<school age>>, s<tory>>, y<ounger school student>>, c<hildren's writer>>, <child>>, <famous>, c<ollection>>, s<tory>>) 2 <Сказка>>, <волшебный>>, <<сказочный>>, <<серия>>, <<ЯПОНСКИЙ>>, <украинский», <<ИНДИЙСКИЙ>>, << арабский>> ( <tale>>, <<magic>>, <fairy tale>>, <<series>, <<J apanese», << Ukrainian>>, <<lndian>>, <AraЬic>>) 3 У<пражнение>>, Я<ЗЫК>>, <<закрепление>>, <брошюра>>, <<английский>>, с<амоучитель>>, л<ингвистический>>, <<испанский>> ( e<xercise», <<language>, <fastening», < brochure,> <<English», <checkbook>>, <linguistic>>, <Spanish> >) 4 <Ребенок», п<ланета>>, н<аписать>>, «стихотворная_форма>>, <рассказывать>, <<сказка>>, <<русский>>, п<риключение>> ( <<child>>, <planet>> , <to write>>, «poetic rfom», «to tell», <<fairy tale», «Russian>>, <adventure>>) 5 < Стихотворная_форма>>, «планета>>, <<рассказывать>>, р<астение>>, а<лфавит>>, <иллюстрация», <<стихотворение» ( <Poetic form», «planet>>, t<o tell», <<plant», <<alphabet>>, <illustration>>, <<poem>>)
12+ 1 <Школьный _возраст>>, <рассказ>>, «младший-ШКОЛЬНИК>>, <детский _писателЬ>>, <ребенок>>, <<известный>>, <сборник», <<ПОВеСТЬ>> ( «school age», «story», «younger school student», «children's writer», <child>>, <famous>>, c<ollection>>, s<tory>>) 2 <Сказка>>, <волшебный>>, <<сказочный>>, <<серия>>, <<ЯПОНСКИЙ>>, «украинский», «индийский», «арабский» ( «tale», «magic», «fairy tale>, <<series>, <<J apanese», << Ukrainian,> <<lndian>>, <AraЬic>>) 3 У<пражнение>>, Я<ЗЫК,> <<закрепление>>, <брошюра>>, <<английский>>, «самоучитель», «лингвистический», «испанский>> ( «exercise», «language>, <<consolidation>>, <<brochure>, <<English>, <<checkbook>>, <linguistic>>, < Spanish>>) 4 <Ребенок», п<ланета>>, н<аписаты>, <<стихотворная -форма>, <рассказывать>>, <<сказка>, <<русский>>, п<риключение>> ( <<child>>, <planet>, <to write>>, <poetic form>, <<to tell>>, <<fairy tale>, << Russian>>, <adventure>>) 5 < Стихотворная_форма>>, «планета>, <<рассказывать>>, р<астение>, а<лфавит>>, и<ллюстрациЯ>>, <<стихотворение>> ( <Poetic form>, <<planet>>, t<o tell», <<plant», <<alphabet>>, <illustration>>, <<poem>>) 16+ 1 <Жадный>>, <<погон>>,<<замучить», «милость», «заметить>>» счастье>>, <свадебный>>, «глотать>>( <<greedy>>,<<epaulet>>,<<to torture>>,<mercy>>, <to notice» «happiness», «wedding», «to swallow») 2 Фронт», «мужчина», «трагедия>>, «гипотетический>>, «догнать>>, <былой>>, <предшественник>>, <достижение>> ( <forefront >>, <<man>>, <tragedy>>, «hypothetical», «to catch up», <<past», «predecessor>>, <achievement>>) 3 <Метод>, <французский>, «упрощение,> п<овторяемость>>, з<чауивание», «уникальность», «текст>>, «лексический>> ( «method>>, erF<nch», <<simplification>>, <repeataЬility>, <<memorization>>, <<uniqueness>, <<text>>,<<lexical>>) 4 <Детский>>, <ребенок>>, о<тношение», <<педагог>, <искусство>>, <воспитание>>, <<зависимость>>, <женщина>> ( <<children's>>, <<child>>, <relation>>, <<teacher>>,<<art>, <<upbringing>>,<<addiction>>,w<oman>>) 5 К<нига>>, <<прошлое>, <человек>>, <<любовь>>, <<ГОрЬКИЙ>>,т<ерять>>, <ПРОЙТИ>>, <<МИР> ( b<ook,> <past>, <man>, <love>>, <<Ьitter>>,<to lose>>,<to pass>>,<реасе>>)
18+ 1 < Чувство>>, <<отношение>>, <<новелла», <<представлять>>, <<реальность>>, и<нтрига>>, <<оудвольствие>>, ф<антазиЯ>> ( f<eeling>>, r<elation>>, s<hort story», <<to imagine>>,<<reality», i<ntrigue», «pleasure», <fantasy») 2 З<одиак>>, г<ороскоп>>, с<ветить>>, <знак», <<поведение>, <<ТИП>>, <предопределять>>, <<сексуальный> ( Z<odiac>>, <horoscope>>, <to shine>>, <sign>>, <<behiavor», <<type>>,«to determine», <<sexual») 3 К<нига>>, <<прошлое>>, ч<еловек>>, <<любовь>>, <<горький>, т<еряты>, п<ройти>>, <<мир> ( < book>>,<past>>, <man>>, <love>, <<Ьitter>>,<to lose>>,<to pass>>,«реасе>>) 4 <История», «друг>>, «жанр», «смешной», «личность», «диалог>>, г<ерой>>, <весёлый> ( <<history>>,<<friend>>, <genre>>, <funny>>, <personality>>, <dialogue>>, <<hero», <funny») 5 <Знание>>, «отличие>>, «мужчина>, <<женщина», <мир», «здоровье>>, <действие>>, <<тело» ( <<knowledge>>, <diference>>, <<man>>, <woman>>, <world>>, <<health>>,<<action>>,<<body»)
The books for children under 6 years old (О+) may contain episodic unnatu
ralistic images justified Ьу the genre or descriptions of physical or psychological
violence, provided that the victim is compassionate and happy ending. The pre
vailing topics in the О+ category are fai ry tales of the world (topic 1), developing
children's benefits (
ln the books for children over the age of 6 (6+), non-naturalistic images or
descriptions of human diseases, accidents, catastrophes or violent death without
demonstrating their consequences are permissiЫe. The books for children over
12 (12+) may contain scenes of violence or murder, descriptions of illnesses,
disasters, but without details. Alcohol, tobaccaond drug use may Ье present,
but should Ье condemned. А schematic description of the hugs and kisses of men
and women may Ье present. These two categories are described Ьу similar topics
in our topic model. These are short stories and tales for primary and secondary
school age (topic 1), fairy tales of the world (
The books rfo children over 16 (16+) may contain scenes of illnesses,disasters
without detailed descriptions. Violence, alcohol and drug use can Ье described,
but should Ье condemned. Rough words may Ье present, with the exception
of swear words. Scenes of sexual relations cannot Ье described with anatomical
details. In our example, this category is represented Ьу military (topics 1-2) and
human condition (
А book should Ье marked with the 18+ label if the book contains а natu
ralistic description of illnesses, disasters, non-condemned drug and alcohol use,
naturalistic scenes of sexual relations, non-traditional relationships, obscene lan
guage, scenes that encourage suicide. In our topic model, love stories (topic 1),
horoscopes (
Most Common Topics
for Categories аТЫе 3 shows most of the common topics discovered using the three sigma rule. These topics are most widely represented in their category. The column <<Main topic>> shows the percentage of texts for which this topic is the main, i.e. it has the largest proportion in the topic distribution.
ТаЫе 3: The most typical topics for categories.
о+ 16+ 16+ 18+ о+ 6+ 12+ «tale», «magic», «fairy tale», «series», «Japanese», 7,553/с < Ukrainian>, <<lndian>>, <AraЬic>> <greedy>>, <epaulet>>, <<to torture>>, <mercy>>, t<o notice>> <<hap- 1,553/с piness», <wedding>>, <<to swallow» f<oreofrnt>>, <man», «tragedy>>, <<hypothetical>, <<to catch up>>, 1,533/с p<ast>>, <<predecessor», a<chievement>> <feeling>>, <<relation>>, <short story>>, <<to imagine>>, <<reality>>,1,783/с i<ntrigue», <pleasure», <fantasy>> <story>>, <fairy tale>>, <<notebook», <<small>>, <<lesson», <<man- 5,663/с ual>>, «intelligence», <series» <school age>>, «story>>, «younger school student,> 2,453/с c<hildren's writer», c<hild>>, <famous>>, <collection>>, <story>> 2,483/с 3.3
rating category using the Dixon's Q-test. The tabulated Qcrit value is equal to 0.642 for confidence level 90% and m = 5.
same time documents of other categories are usually mixtures of topics. There rofe, our topic model has many specific topics for texts ofrm the 0+ category. In аТЫе 4, we present the list of the most common age-specific topics in our data set.
dren's books, as well as to specicfi literature orfm literature on business and success). topics generally relate to chil the 18+ category (in our case, аТЫе 4.
Categ >ГУ Keywoгds Qcrit 18+ «success», «activity», «man», «business», «city», «collection», «еuго- 0,94 pean>>, <<necessary» о+ <story>>, <<fairy tale>>, <notebook>>, <<small», <<lesson>>, «manual>>, <intelli- 0,93 gence>>, <series» о+ «fairy tale>>, «plunge», «fairytale atmosphere», «emotion», <character», 0,85 <interesting>, <exciting», <<impression>> о+ <child>, <planet», <<to write>, <<poetic form>>, <<to tell>>, <<fairy tale>, <<Rus- 0,78 sian>>, a<dventure>> о+ t<ale», <<magic>>, <<fairy tale>>, <series», <Japanese>>, «Ukrainian>>, <lndian>>, 0,76 <Arablc>> о+ <malachite Ьох>>, <<tale», <<storyteller>>, «humor>>, nfu<ny>>, <<forest>, 0,75 h<ero>>, с<араЫе» 1 4
Conclusion In this paper, we empirically analyzed the topics of texts assigned to diefrent age rating categories. We introduced the distribution of topics for age categories and the list of the most common topics for categories and age-specicfi topics. These list of topics were oЬtained using statistical methods. Our analysis confirmed the existing diferences between the categories and demonstrated that topic models can Ье а good source of features for age rating identicfiation. ln our future work, we will try to develop а machine learning classifier for automatically determining the text age rating.
17. Yang, S. and Zhang, Н.: Text mining of Тwitter data using а latent Dirichlet allocation topic model and sentiment analysis. In: Int. J. Comput. Inf. Eng, 12, рр.525-529 (2018). 18. Zhao, F., Zhu, У., Jin, Н., Yang, L. Т.: А personalized hashtag rec ommendation approach using LDA-based topic model in microЬlog en vironment //Future Generation Computer Systems 65, 196-206 (2016). https://doi.org/10.1016/j.future.2015.10.012 19. Zhao, W. Х., Jiang, J., Weng, J., Не, J., Lim, Е. Р., Yan, Н., Li, Х.: Com paring twitter and traditional media using topic models. In: European confer ence on information retrieval, рр. 338-349. Springer, Berlin, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20161-53_4