The Problem of Selecting a Reference Group for the Thematic Rating of Scientific Journals Tatiana A. Polilova, Mikhail M. Gorbunov-Posadov Keldysh Institute of Applied Mathematics, Miusskaya sq., 4, Moscow, 125047, Russia Abstract The Russian Academy of Sciences (RAS) together with the Electronic Library eLibrary.ru has proposed an Academic rating of scientific journals from a limited collection, which includes about 700 journals. Using the example of a rating of journals compiled by the impact factor in the thematic section “Mathematics”, the methodological errors made in the selection of the reference group of journals are shown. It is concluded that the Academic rating has significant distortions associated with the thematic heterogeneity (multidisciplinary) of journals in refer- ence groups formed on the basis of the existing classifier. The rating indicators are influenced by articles that are not related to the topic of the rating. We believe that an alternative technol- ogy for forming reference groups “from the bottom up” is a more reliable basis for thematic ratings. The bottom-up technology reveals the thematic proximity of journal articles based on the analysis of their bibliographic references, launching the clustering process. The thematic proximity of journals can be expressed in numerical form and further serve as the basis for the formation of reference groups for the thematic rating. A carefully constructed reference group is the key to the adequacy of the thematic rating of journals. Keywords1 scientific publication, rating of scientific journals, reference group, Academic rating, citation graph, multidisciplinarity 1. Introduction Modern bibliographic databases have accumulated a huge amount of information about scientific publications. The data sources of bibliographic databases are scientific journals, collections of articles, conference proceedings and other types of publications. Publishers regularly supply bibliographic da- tabases with a wide set of metadata of published scientific products: titles and annotations of articles, keywords, codes of thematic areas, information about the authors of articles and organizations affiliated with the authors. Bibliographic lists of used literature, which are converted into formal structures, also are loaded into the databases. These structures allow for machine processing and ensure identification of the bibliographic list item with the article. In recent years, thanks to the international codes ISSN, DOI, ORCID, RAR, the quality of processing bibliographic lists and procedures for identifying biblio- graphic references with articles has improved markedly. The data on the citation of scientific publica- tions became more accurate. One of the ways to confirm the authority of a scientific journal is the participation of the journal in the ranking based on the citation indicators of articles published in the journal. To assess the authority of journals, such a bibliometric indicator as the impact factor is traditionally used [1]. The impact factor of the journal is calculated as the ratio of the sum of citations of journal articles to the number of articles published in a certain period of time. Such a period is often a two-year or five-year period. The more articles are cited, the higher the position of the journal in the rating. Building ratings of the journals is one of the popular services of bibliographic databases. For example, Journal Citation Reports (JCR) SSI-2021: Scientific Services & Internet, September 20–23, 2021, Moscow (online) EMAIL: polilova@keldysh.ru (Tatiana A. Polilova); gorbunov@keldysh.ru (Mikhail M. Gorbunov-Posadov) ORCID: 0000-0003-4628-3205 (Tatiana A. Polilova); 0000-0002-7044-8287 (Mikhail M. Gorbunov-Posadov) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) regularly publishes impact factor based ratings of journals included in Web of Science Core Collections. JCR is an annual publication by Clarivate Analytics. The importance of the journal's impact factor should not be absolutized. Not always high positions in the ratings of journals in terms of the impact factor indicate the authority and influence of journals. The paper [2] lists serious arguments supported by examples proving the fallacy of a straightforward relationship between the impact factor indicator of the journal and the quality of published articles. The validity of the averaged formula for calculating the impact factor is questionable: several highly cited articles can significantly increase the impact factor of the journal against the background of most articles with low citation. The proposed indicators are easy to manipulate, which is also confirmed by numerous examples. The competition of journals by the indicator of the impact factor provokes unethical behavior of authors and negatively affects the quality of research. Various methodological errors in the calcula- tion of journal indicators for the selected reference group lead to distortion of ratings by the impact factor. Some of these errors are discussed in this article. The bibliographic database provides its users with analytical tools for processing bibliometric infor- mation. These tools can be used by users who do not have special training in the field of bibliometrics, when conducting simple analytical studies. As noted in [3], a new target audience has appeared which is not engaged in bibliometrics research at a professional level, but shows great interest in bibliometrics indicators and ratings of scientific journals. This audience includes employees of organizations and foundations that finance scientific research, as well as researchers. Specialists from different scientific fields actively interpret the results of bibliometrics research obtained by professionals. Scientists, being interested consumers of the results of ranking journals by bibliometric indicators, are able to assess the adequacy of ratings in their scientific field. Scientists have personal experience in preparing and publishing scientific articles in journals on their subject and, of course, know firsthand the authoritative journals of their scientific field. Bibliometrics does not stand still. Among professional specialists in the field of bibliometrics, dis- putes still continue on fundamental methodological issues: there is a search for new indicators and new approaches that contribute to a more adequate comparison of journal indicators [4, 5]. In this paper, the task of studying the influence of various indicators in a complex rating formula is not set. The purpose of our work is to identify the problems of taking into account multidisciplinary at the level of single journal articles. The task of constructing a thematic rating is usually divided into two stages:  the selection of a reference group of journals in the thematic direction under study,  the calculation of rating indicators for journals of the reference group on the selected field of citing journals in a certain time interval. The results of the rating depend on the methodological and algorithmic decisions that are made at each of these stages. The analysis of the methods of rating scientific journals in some cases allows you to identify the sources of distortions, errors or inaccuracies, to detect group interests or biases of the rating compilers. Therefore, the openness of data collection and analysis procedures for conducting bibliometrics assessments, as well as the ability of research scientists themselves to verify the correct- ness of determining the results of bibliometrics analysis, are declared important fundamental principles formulated in the Leiden Manifesto of Bibliometrics [6]. It is appropriate to draw a parallel with the methods of mathematical modeling. If the behavior of the constructed mathematical model does not correspond to physical realities, than theoretical provi- sions, limitations or assumptions in the mathematical model require additional careful analysis. Simi- larly, if the results of the thematic rating of scientific journals do not meet the expectations of specialists in this scientific field, it is necessary to analyze not only the formula for calculating the rating (it can be very simple), but also the methodology for selecting the reference group, the accepted limitations (extensions) of the set of citing journals, ways of taking into account the features of citation in various scientific fields. We will consider the problem of building thematic ratings on the example of the Russian electronic library eLibrary.ru [7]. The Russian Index of Science Citation (RISC) information and analytical system functions on the basis of this library [8]. Currently, RISC indexes more than 5,700 scientific journals and generates journal ratings using several bibliometric indicators. The methods of constructing the- matic RISC ratings formed the basis for the construction of the Academic rating of journals. 91 2. RISC rating in the thematic area “Mathematics” The RISC system is quite open to users. Moreover, RISC provides users of the system with access to a large array of statistical information about Russian scientific journals. RISC allows you to build and examine thematic ratings of journals in a wide range of scientific fields. RISC provides an on- screen form for setting the parameters of the rating the user is interested in. In the form, you can select the scientific area of the journals, the year, and also select a specific type of rating, for example:  two-year impact factor RISC,  two-year impact factor RISC, taking into account citations from all sources,  five-year impact factor on the RISC core,  the Herfindahl index by author organizations,  the Hirsch index of the journal for 10 years etc. The analytical system makes it possible to find out more detailed statistical reports about each journal, such as:  distribution of publications by subject,  distribution of publications by the number of citations,  distribution of citing publications by subject,  distribution of citations by the subject of citing publications and others. Consider the RISC rating “Two-year impact factor RISC” in the thematic area “Mathematics” for 2019. In this RISC rating, citations from all journals included in the RISC are taken into account when calculating the impact factor indicators. Citations of the translated version are not taken into account. The most authoritative mathematical journals in Russia are academic journals, the founders and publishers of which are the Russian Academy of Sciences, Moscow State University and academic institutes conducting research in the mathematical field. It would be natural to expect that these aca- demic journals will occupy the first positions in the ranking, taking into account the citations of journal articles. However, these expectations were only partially fulfilled. The following journals were placed in the first three positions of the rating (in descending order of the impact factor indicator): 1. Geometry & Graphics 2. Informatics and Automation 3. Journal of Samara State Technical University, Ser. Physical and Mathematical Sciences 4. Izvestiya: Mathematics The mathematical journal “Izvestiya: Mathematics”, widely known and respected in the mathemat- ical community, took only the 4th place in the ranking. In the given RISC rating, the leading position of the “Geometry & Graphics” journal attracts atten- tion. The journal is not a classical mathematical journal. What is the reason for such high indicators of “Geometry & Graphics” journal in the RISC rating on the two-year impact factor in the area of “Math- ematics”? The journal in its profile indicated two thematic headings of the OECD: 101. Mathematics 503. Educational sciences In the statistical report “Distribution of publications on thematic classification” of this journal, 7 topics of articles are recorded (the number of articles is indicated in parentheses): 1. Mathematics (178) 2. Public education. Pedagogy (102) 3. Mechanical engineering (5) 4. Automation. Computer engineering (3) 5. Cybernetics (3) 6. Literature. Literary studies. Oral folk art (1) 7. Power engineering (1) Most of the articles published on the topic “Mathematics” – 178 articles (61% of all articles). On the topic “Public education. Pedagogy” was published 102 articles. As statistics show, in the journal “Ge- ometry & Graphics” all the articles registered in the bibliographic database have one thematic heading. 92 Consider the statistical report “Distribution of citing publications on thematic classification” of the journal “Geometry & Graphics”. The following topics are located in the first 4 positions (the number of articles is indicated in parentheses): 1. Public education. Pedagogy (410) 2. Mathematics (290) 3. Mechanical engineering (31) 4. Power engineering (28) We will assume that the thematic of the article in this report and the thematic of cited articles are the same. The database does not provide more accurate statistics. However, this uncertainty can be ignored, since we will be primarily interested not so much accurate calculations as in qualitative estimates. In total, the report recorded 832 citing articles. It was 290 (about 35%) citing articles on the topic “Mathematics”. High citation rates were provided to the journal by articles related to the topic “Public Education. Pedagogy” (49% of all citing articles). The average citation of articles on this topic is 4.12, while the average citation of mathematical articles is only 1,63. For this reason, the journal “Geometry & Graphics” unreasonably took the leading position in the ranking in the “Mathematics” topic. If articles and citing articles related only to the mathematical area were taken into account when calculating the RISC impact factor indicators, then “Geometry and Graphics” journal would not have reached the first positions of the RISC rating in the Mathematics section. The considered case of getting the journal, which is not the most famous among mathematicians, to high positions in the RISC mathematical rating demonstrates the imperfection of the methodology for constructing thematic RISC ratings based on the impact factor, which has the simplest formula for calculating the value of the rating indicator. The situation is similar with the following journal – “Informatics and Automation”, which took 2nd place in the RISC rating in the thematic area “Mathematics”. The journal is classified under the follow- ing OECD headings: 01.01. Mathematics 01.02. Computer and information sciences 02.02. Electrical engineering, electronic engineering The topics of the articles published by the journal are presented in the RISC statistical report “Dis- tribution of publications on thematic classification” for the journal “Informatics and Automation”. In total, 29 topics are registered in the statistical report. The first 4 positions in the list of topics of articles are given below, indicating the number of articles published in the journal: 1. Automation. Computer technology (426) 2. Mathematics (270) 3. Cybernetics (226) 4. Computer Science (42) These data show that mathematics is not the dominant thematic areas of the journal. Consider the statistical report RISC of the journal “Informatics and Automation” “Distribution of citing publications on thematic classification”. Below are the first 4 positions in the list of topics for which the number of citations is more than 100: 1. Cybernetics (1281) 2. Automation. Computer technology (1246) 3. Mathematics (632) 4. Computer Science (394) Percentage of articles on the topic “Mathematics” from the number of articles on Computer Science topics (three headings “Automation. Computer Engineering”, “Cybernetics” and “Computer Science”) is 39%. The percentage of citing articles on the topic “Mathematics” in relation to the number of citing articles on Computer Science topics is only 22%. Thus, in the journal “Informatics and Automation” mathematical articles are cited worse than articles on Computer Science topics. This conclusion coin- cides with the generally accepted ideas about a more modest citation of articles in mathematics. There- fore, it can be argued that taking into account articles on Computer Science topics and their citations significantly increases the indicator of the journal “Informatics and Automation” in the rating by impact factor in the area of “Mathematics”. 93 Let's consider the thematic areas of articles in the journal “Journal of Samara State Technical Uni- versity, Ser. Physical and Mathematical Sciences”. The first 4 positions of the list of topics from report “Distribution of publications on thematic classification” are presented below: 1. Mathematics (639) 2. Mechanics (375) 3. Physics (281) As follows from the data of the statistical report, mathematics is the dominant discipline in this journal. The number of articles on the topics “Mechanics” and “Physics” in total is slightly more than the number of articles on mathematics – 656. Let's see how the journal articles are cited. Below are the first 3 positions of the list from the statis- tical report “Distribution of citing publications on thematic classification” (for each of the topics pre- sented in the list, the number of citing articles is more than 100): 1. Mathematics (1454) 2. Mechanics (876) 3. Physics (328) The following three thematic areas were expected to dominate: “Mathematics”, “Mechanics” and “Physics”. The appearance of this journal in the third position of the thematic rating in the area of “Mathemat- ics” does not cause surprise or sharp rejection. However, even here, more adequate rating indicators in the area of “Mathematics” can be obtained if, when calculating the impact factor indicator, only articles of the area of “Mathematics” and citations of these articles are taken into account. Finally, let's consider the statistical reports of the journal “Izvestiya: Mathematics”. There are 3 positions in the statistical report “Distribution of publications on thematic classification”: 1. Mathematics (811) 2. Physics (10) 3. Mechanics (1) As this report shows, almost all the articles of the journal belong to the field of “Mathematics”. If you look at the statistical report on the citing articles, you can again make sure that the vast ma- jority of articles are recorded on the topic of “Mathematics”: 1. Mathematics (18734) 2. Physics (3483) 3. Automation. Computer technology (241) 4. Chemistry (210) 5. Mechanics (182) For the rest of the topics present in the report, less than a hundred citing articles were recorded. Thus, the overwhelming number of citations was made from mathematical articles. “Izvestiya: Mathematics” journal is a classical mathematical journal that should have deservedly taken a higher position in the ranking in the “Mathematics” section. The above example again shows that the methodology for the formation of thematic RISC ratings has significant flaws. The entry of non-core journals into high positions in the impact factor rating in the direction of “Mathematics” is explained by the following circumstances:  the multidisciplinary nature of journals at the article level is not taken into account,  the rating calculation involves indicators of articles related to other, non-mathematical, topics in which the average citation rates are higher than in the direction of “Mathematics”; Normalization procedures are often used when calculating rating indicators in bibliographic data- bases on a variety of journals related to various thematic areas. The normalization procedure allows you to take into account the features of citation in different scientific fields. If normalization is applied to the journal as a whole, then it is assumed that all the articles of the journal belong to the same theme – the theme of the journal. But there is also a problem of adequate consideration of the citation of the- matically diverse journal articles. And taking into account such multidisciplinarity at the level of articles often remains beyond the interests of rating compilers. The citation of scientific publications is considered as one of the generally acknowledged assess- ments of the authority of a scientist in the scientific world. In [9], the thesis is put forward that the assessment of the influence of the author of publications in a certain scientific field should be carried out on the basis of citation indicators of publications related to this scientific field. For example, the 94 author's authority in the scientific field of “Physics” is determined based on the calculation of citations of his publications in physics, and at the same time author's publications in other scientific fields and their citing should not be included in the calculation of ranking formula. This idea can be extended to the evaluation of the impact of the journal in different thematic areas. In [10], it was suggested that when calculating RISC ratings in the Mathematics section, articles and their citations related to the Mathematics area should be taken into account, and articles and citations related to other scientific areas should not be taken into account. This approach will increase the ade- quacy of thematic ratings and give a more accurate assessment of the impact of journals in the scientific field within the framework of the used mechanism for accounting bibliographic references. 3. Academic ranking The Russian Academy of Sciences has attempted to create another rating – Academic rating. A meeting of the RAS Presidium was devoted to this subject on March 23, 2021 [11]. The Academic rating is based on the methods of building RISC ratings, but differs in some essential details. Has the Academic rating get rid of the methodological errors of the RISC rating? The Academic ranking is built for journals included in the Special Collection (SC), which is a subset of journals included in RISC. This collection was named RSCI (Russian Science Citation Index). The project to create RSCI [12] was launched in 2014 on the initiative of the Russian Academy of Sciences and the Higher School of Economics. The project assumed the identification of the most popular Rus- sian scientific journals both in Russia and abroad and their placement on the Web of Science platform in the form of a separate database, fully integrated with the Web of Science, by analogy with the Chinese (Chinese Science Citation Database) and Latin American (SciELO Citation Index) scientific citation indexes [6]. Journals with high citation rates were selected for the SC collection. In addition, formal assessments were supplemented with expert assessments. Well-known scientists of the Academy of Sciences and leading Russian universities were involved in the expertise. The Academic rating offered by the Russian Academy of Sciences focuses exclusively on journals from the SC collection. The RISC core includes journals from the SC collection (about 700 journals), as well as journals indexed in the international bibliographic databases Web of Science and Scopus (we are talking about several dozen Russian jour- nals published in English). The RISC core contains about 15% of all journals included in RISC. The Academic rating formula is based on a two-year (IF-2) or five-year (IF-5) impact factor. For journals in social and humanitarian fields, the five-year indicator IF-5 is used, for other areas – the indicator IF-2. The rating formula also includes several other bibliometrics indicators, but they do not significantly affect the position of journals in the rating. An important circumstance is the restriction on the field of citing journals. When calculating the Academic ranking of journals, citations are taken into account only from journals included in the RISC core. Thus, the field of citing journals is limited to 15% of all journals included in RISC. We will demonstrate the impact of the accepted restrictions on the results of several journals leading in the RISC rating according to the 2-year impact factor, which was discussed above. Figure 1 (a, b) shows the graphs of results of the ratings of the two-year impact factor in 2011-2019 for four journals that occupy leading positions in 2019:  blue – the journal “Geometry & Graphics” (leader-1),  red – the journal “Informatics and Automation” (leader-2),  green – the journal “Journal of Samara State Technical University, Ser. Physical and Mathemat- ical Sciences” (leader-3),  purple – “Izvestiya: Mathematics” journal (leader-4). 95 Figure 1: (a) – impact factor of journals when citations from all journals included in RISC were taken into account; (b) – impact factor of journals when only citations from the RISC core were taken into account. Graphs (a) show the values of the impact factor of journals in the case when citations from all jour- nals included in RISC were taken into account. Graphs (b) show the values of the impact factor of journals in the case when citations from the RISC core alone were taken into account (15% of all RISC journals). 96 Note that in 2019, in the two ratings (a) and (b), the leaders were arranged in reverse order. When calculating the impact factor in the RISC core Academic ranking, citations of the translated version are taken into account. This approach immediately provides advantages to academic journals that have a translated version. Graphs (b) shows that the impact factor graph of the academic journal “Izvestiya: Mathematics” (purple graph) is located above all three other graphs. The second place in 2019 is taken by the “Journal of Samara State Technical University, Ser. Physical and Mathematical Sciences” (green graph). The third place in 2019 is taken by the journal "Informatics and Automation" (red graph). The last place is taken by the journal “Geometry & Graphics” (blue graph), which has a weak citation on the RISC core. The Academic rating for mathematical journals fulfilled the main function – it brought academic mathematical journals to the leading positions. Is this methodological approach associated with limiting the field of citing journals manipulation? Or are there reasonable arguments in favor of using this tech- nique? Oddly enough, the answer to this question can be found by analyzing the citations of mathemat- ical journals in the bibliographic database MathNet.ru [13]. This bibliographic database contains metadata of the leading journals of physics and mathematics published by the Russian Academy of Sciences and leading Russian universities, including the journals “Izvestia: Mathematics”, “Informatics and Automation”, “Journal of Samara State Technical University, Ser. Physical and Mathematical Sci- ences”. The paper [14] describes the results of constructing a graph of journal citations (citation network) based on the analysis of bibliographic lists of articles from journals MathNet.ru. In essence, a graph of journal citations was formed in the “from the bottom up” technology, which is also called “Citation Topics” [15]. The bottom-up approach suggests using mutual bibliographic links as an indicator of the thematic proximity of articles. In this case, direct citation and the intersection of bibliographic lists (co-citation) are taken into account. Thematic proximity can be expressed in numerical form and further serve as the basis for the formation of reference groups for thematic rating. Identifying links of articles by biblio- graphic references starts the clustering process. Articles that are most strongly associated with citations are combined into a single thematic cluster. The thematic cluster is constantly increasing as new citing articles appear. There is another effect: with the appearance of new citations, the article may be included additionally in another thematic cluster. The thematic clusters of the lower level are combined into larger clusters by relaxing the requirements for the proximity of publications. Thus, higher-level clus- ters are formed, reflecting more general thematic categories. Clusters of different levels get names (labels). For lower-level clusters, the label can be generated automatically, for example, based on the most significant keywords from the cluster articles. For clus- ters of intermediate and higher levels, labels (categories) can be assigned by experts in the thematic area, being tied to familiar or convenient names reflecting the thematic of articles grouped in the cluster. At the same time, experts can rely on characteristic keywords and categories familiar to scientists, in particular, the categories of Web of Science. Clusters of articles naturally generate clusters of journals. In MathNet.ru the journal citation graph has split into five modules (clusters), which can be inter- preted according to research areas: fundamental mathematics, mathematical modeling, experimental and theoretical physics, discrete mathematics and applied mathematics, computer science. Almost all journals MathNet.ru they are part of the RISC core. At the same time, there is a high density of the citation graph for journals MathNet.ru, which explains the high citation rates and high impact factor indicators of these journals in the Academic ranking. The Academic ranking did not use thematic catalogs based on cluster analysis. Reference groups of journals in thematic areas were formed on the basis of the traditional OECD catalog, organized in the top-down technology. The problems of accounting for multidisciplinarity at the level of journals and at the level of articles were not solved. The rating indicators were calculated based on the citation indica- tors for the RISC core. This limitation of the field of quoting journals gives an advantage to academic journals. In some areas, for example, in the field of “Mathematics”, well-known and reputable journals have taken leading positions in the ranking. However, limiting the field of quoting journals, in our opinion, is not an impeccable methodological technique. As a result, the academic rating in many posi- tions gives inadequate estimates of the impact of journals. 97 Here is an illustrative example. The previously reviewed journal “Informatics and Automation” is obviously closer to the topic of Computer sciences than to the topic of Mathematics. Note that this journal participates in RISC ratings on three topics: “Mathematics”, “Automation. Computer engineer- ing”, “Cybernetics”. In each of these three ratings, the journal takes the 1st position with the integral value of the rating indicator 7,135. This means that the same array of articles and citations is taken into account in each of the listed areas. This fact confirms the conclusion made earlier that in the direction of “Mathematics” the high index of the journal in the rating of the RSCI is achieved due to citations from articles not related to mathematics. Accordingly, in the other two directions on the topic of Com- puter sciences, not all of the citations taken into account relate to these topics. In the Academic ranking, when calculating indicators, the total array of articles and citations is also taken into account, without taking into account their subject matter. For unknown reasons, in the Academic ranking, the journal “Informatics and Automation” participates in the less profitable thematic direction “Mathematics” and occupies only the 19th place in it with an indicator value 1,226. If the journal participated in the thematic rating on the topic “01.02.00 – Computer and information sciences”, then with such an indicator it would take 2nd place and would significantly increase its status. Here is another example. The journal “Doctor and Information Technologies” (published by the National Medical and Surgical Center named after N.I. Pirogov) in the Academic ranking is included in the group “01.02.00 – Computer and information sciences”. This journal ranks 5th position in the Academic ranking in this group. Does the journal correspond to such a high rating? The following OECD headings are stated in the RISC journal profile: 102. Computer and information sciences 302. Clinical medicine However, if we look at the distribution of the journal's articles by subject, then without any doubt the journal should be attributed to the medical field: the number of articles on the topic "Medicine and healthcare" in the journal is 93%. The number of articles on two topics “Cybernetics” and “Computer Science” is 2.5%. The journal does not participate in the RISC thematic ratings on information technol- ogies. In the direction of “Medicine and healthcare” in the ranking of the two-year impact factor on the RISC core, the journal “Doctor and Information Technology” ranks 65th. For what reasons is the journal included in the Academic ranking in the direction of “Computer and information sciences”? It is possible that the journal was assigned to this category according to a formal algorithm based on the subject of the journal stated in the questionnaire. However, such a decision can be confidently called erroneous. The argument about the high citation of the journal's articles on the topic of “Cybernetics” and “Computer Science” does not stand up to criticism either. Indeed, the average citation of articles on the topic of “Computer and information sciences” (under two topics “Cybernetics” and “Computer Sci- ence”) is 4.98. The average citation of articles on medicine in this journal is only 2.1. However, it is likely that the given values of the average citation indicators are due to an unrepresentative set of sta- tistical data. Since 2011, the journal has published a total of 1029 articles on the following topics (the number of articles and citations is indicated in parentheses):  Medicine and Healthcare (956, citations – 2004)  Economics. Economic Sciences (23, citations – 211)  Cybernetics (15, citations – 71)  Informatics (11, citations – 65)  Sociology of science (7, citations – 2)  Public education. Pedagogy (6, citations – 77)  Automation. Computer engineering (3, citations – 95)  State and Law. Juridical sciences (2, citations –43)  Physics (2, citations –20)  Standardization (1, citations – 10)  Electronics. Radio Engineering (1, citations –17)  Demography (1, citations – 1)  Biology (1, citations – 15) These data show that high rates of average citation may have thematic areas in which single articles that have received significant citation have been published. For example, high average citation rates are 98 noted in economics, computer engineering, pedagogy, law, physics, electronics and radio engineering, biology. If we focus on the average citation, then the journal could be attributed, for example, to eco- nomic sciences or pedagogy. However, it is obvious that it is the medical area that has statistically meaningful indicators. Here we are again faced with a situation where a non-profile journal occupies high positions in the thematic rating due to non-profile articles and the high citation rate of such non-profile articles. In the considered case, high rates of average citation were demonstrated by articles with unrepresentative sta- tistical indicators. Note that in the bibliographic database, the technology of building a thematic catalog bottom-up creates a more objective basis for assigning the journal to thematic category. Thematic clustering based on the analysis of bibliographic links sets the parameters of the journal's proximity to a particular topic. These parameters should influence the formation of a reference group for thematic rating. For the jour- nal “Doctor and Information Technologies”, thematic clustering bottom-up would definitely show the journal's proximity to the medical field. 4. Conclusion The rating of journals in any thematic area should highlight the most authoritative and popular jour- nals in the relevant scientific field. There are a lot of comments and complaints about RISC ratings. From the Academic rating the scientific community expected a more objective picture to appear in the field of Russian journals. However, the resulting Academic rating is just a variation of the RISC rating calculated according to the indicator “Two-year impact factor on the RISC core”. The Academic rank- ing inevitably inherits all the problems and shortcomings that RISC suffers from. Unfortunately, a com- plete and objective picture defining the most influential Russian journals in scientific fields has not developed in the Academic ranking. The problems considered in the article of accounting in the ratings of multidisciplinarity at the level of articles are characteristic of many bibliographic databases. A more reliable basis for the formation of reference groups in thematic ratings is the technology of building a thematic catalog “from the bottom up”. The clustering of journals used in this technology, based on the analysis of bibliographic links of published articles, reveals the dominant topic of the journal and determines the parameters of proximity to other thematic areas. Thus, the bottom-up technology contributes to the development of methods for taking into account multidisciplinarity at the article level to create more adequate thematic ratings of journals. 5. References [1] E. Garfield, The History and Meaning of the Journal Impact Factor. JAMA 295 (1) (2006) 90–93. https://doi.org/doi:10.1001/jama.295.1.90. [2] J. P. Tennant, H. Crane, T. Crick, J. Davila, A. Enkhbayar, J. Havemann, B. Kramer, R. Martin, P. Masuzzo, A. Nobes, C. Rice, B. Rivera-Lopez, T. Ross-Hellauer, S. Sattler, P. D. Thacker, M. Vanholsbeeck, Ten Hot Topics around Scholarly Publishing, Publications 7(2) (2019) 34. https://doi.org/10.3390/publications7020034. [3] L. Leydesdorff, P. Wouters, L. Bornmann, Professional and Citizen Bibliometrics: Complementa- rities and Ambivalences in the Development and Use of Indicators – a State-of-the-Art Report, Scientometrics 109 (2016) 2129–2150. https://doi.org/10.1007/s11192-016-2150-8. [4] O. V. Moskaleva, Nauchnye publikatsii kak sredstvo kommunikatsii. V knige: Rukovodstvo po naukometrii: indikatory razvitiia nauki i tekhnologii. Ekaterinburg, 2021. https://doi.org/10.15826/B978-5-7996-3154-3. [5] M. Akoev, V. Markusova, O. Moskaleva, V. Pislyakov, Handbook on Scientometrics: Science and Technology Development Indicators, Second edition, 2021. https://doi.org/10.15826/B978-5-7996-3154-3. [6] D. Hicks, P. Wouters, L. Waltman et al., Bibliometrics: The Leiden Manifesto for research metrics, Nature 520 (2015). https://doi.org/10.1038/520429a. [7] Russian Scientific Electronic Library. URL: https://www.elibrary.ru. 99 [8] Russian Index of Science Citation. URL: https://www.elibrary.ru/project_risc.asp. [9] P. Vinkler, Evaluation of publications by the part-set method, Scientometrics 126 (2021) 2737– 2757. https://doi.org/10.1007/s11192-020-03841-7. [10] T. A. Polilova, Journal ratings in the RSCI as tools for analysis and influence. KIAM Preprint (2021), 40. https://doi.org/10.20948/prepr-2021-40. [11] Meeting of the Presidium of the Russian Academy of Sciences on March 23, 2021. URL: https://www.youtube.com/watch?v=LtFijzBOO0E. [12] Project of Russian Science Citation Index. URL: https://www.elibrary.ru/project_rsci.asp. [13] All-Russian Portal Math-Net.Ru. URL: http://www.mathnet.ru/index.phtml?&option_lang=eng. [14] A. A. Pechnikov, D. E. Chebukov, Structure of the journal citation network Math-Net.Ru, in: Pro- ceedings of 23th Scientific Conference “Scientific Services & Internet – 2020”, Moscow, 2021, pp. 265–278. https://doi.org/10.20948/abrau-2021-2. [15] I. Potter, Introducing Citation Topics in InCites – Clarivate. Dec 3, 2020. URL: https://clari- vate.com/blog/introducing-citation-topics/. 100