Maintaining the scientific research quality and publications is a prerequisite for ensuring the science development. The solution of this problem is provided by work in various directions, such as popularization of ethics and research integrity; ensuring transparency of the peerreview process, use of intellectual text processing methods. The reviewer plays an important role in the paper evaluation during the process of expert scientific evaluation. This paper is presented the use of topic modeling and natural language processing methods to find reviewers who are experts in this paper's field. The proposed information-topic model not only helps to find potential reviewers for articles, but also helps to resolve conflicts of interest, based on the use of metadata. The proposed method can be used for any industry, for any language, with little adaptation of pre-processing methods of textual information. The presented method works well, quickly and really people who have been found by our software, fit the paper subject.
A significant number of issues related to the activities of a scientist and an academic institution
depend on the quality of their evaluation. For example, in Ukraine, the serial number of the university
in one of the international rankings is taken into account in the distribution of funding (state budget
expenditures between higher education institutions) [
The practice shows rapid growth in the number of academic products, they are academic articles, conference proceedings, books, reports, presentations. Evaluating such a large number of scientific documents is a complex multifaceted task.
Also, the quality of scientific publications is a crucial issue not only for researchers and academic institutions but also for organizations that fund research, shape science policy, engage in employment, organize scientific awards based on the evaluation of academic research.
The definition and implementation of scientific article quality indicators are described in many
publications [
Studies [
But, even after the publication of a scientific paper, as world practice shows, it can be retracted. According to the recommendations of the Committee on Publication Ethics, editors should consider retracting the publication in the following cases [9]:
• there is evidence that the conclusions are unreliable (errors in calculations, falsifications, manipulation); • plagiarism was exposed; • the document contains material or data without permission for use; • copyright has been infringed; • unethical research is described; the document was published based on a compromised peer review process.
That is why the authors of this paper pay attention to determining the article quality at the stage of scientific peer review and ensuring the objectivity of the peer-review process. To do this, measures should be taken in the following areas: 1. ensuring the transparency of the scientific review process by creating instructions and tools; 2. selectivity in choosing the reviewer.
To ensure the first direction we can identify the main factors that potentially indicate the quality of
the article [
To guarantee the appropriate reviewer authors of the article propose an approach to the selection of a reviewer. A reviewer is an expert in the scientific area to which the peer-reviewed document belongs and has not biased against the authors.
The urgency of the task of choosing an unbiased professional expert to conduct a scientific examination of the article is confirmed by the difficulties of choosing a reviewer, which takes place in a rapidly growing volume of information and development of interdisciplinary science [14].
A study conducted by the Swiss National Science Foundation found that when grant applicants are selected, grant authors are four times more likely to receive a positive response than a negative one [15].
Each editorial board decides on this issue in its own way. To facilitate this task, the web service Publons (Clarivate) [16] can be used, which contains information on more than 3 million researchers from around the world. Publons was launched in 2012 as a platform to recognize reviewers for their important work. The selection of potential reviewers is done by selecting applicants for a particular topic from the list of standard topics of Web of Science (Clarivate) and further analysis of their scientometric indicators and the list of publications [17]. In total, the Web of Science includes approximately 250 subject areas in the natural, social and human sciences, making it difficult to find reviewers for publications on narrow or interdisciplinary topics.
An alternative approach to finding a reviewer may be to create and use your own database of scholars. However, self-formation and maintenance of the relevance of the reviewer database take a long time and cannot be effective.
The search for a reviewer should be carried out very carefully and responsibly so as not to fall for unscrupulous scientists. There are known cases where reviewers use Publons to gain recognition for superficial or poor expert judgment [18].
Reviewing remains an important element in scientific publishing. Therefore, innovative
technologies are being introduced to improve it. Artificial intelligence and specialized programs, which
create auxiliary tools for the reviewer [
Springer has implemented a tool for finding reviewers "Springer's reviewer finder", which offers to select an expert based on the metadata of publications [19]. The correspondence algorithm returns a list of researchers who have a publishing profile similar to the profile of the manuscript author(s) [20].
Frontiers (Switzerland) has developed and presented to external editors the Artificial Intelligence Review Assistant (AIRA) software. AIRA helps editors, reviewers and authors assess the quality of manuscripts: language quality assessment, number integrity, plagiarism detection, and potential conflicts of interest. The software finds whether authors, editors, or reviewers of articles have coauthored in the past. It also notes articles on controversial topics that require increased attention from the editor [21].
The authors of this article aim to explore the use of modern means of choosing a reviewer for the scientific examination of manuscripts. The authors propose a method of optimizing and simplifying the process of finding potential reviewers by the editorial boards of scientific publishers.
The ways to optimize/simplify the process of finding a reviewer are offered. The reviewer is an expert in the qualitative evaluation of scientific information. He is a specialist who has high educational, scientific and professional qualifications, has the experience, has scientific publications in authoritative scientific journals around the world.
That is why the authors consider global multidisciplinary academic resources as a source of data for the reviewer, which contains information about the authors and their publications, as well as the ability to export data for automated processing. Such resources include publications from various scientific journals and publishers, not limited to specific geography. In this study, the authors analyzed the following resources: Scopus (http://scopus.com), Web of Science Core Collection (http://webofscience.com), Dimensions (https://app.dimensions.ai/discover/publication).
Data from the Scopus bibliographic and abstract database were used to form the information-topic model (ITM). Scientific publications included in this database meet strict selection requirements and are considered authoritative scientific publications. Thus, authors published in these publications can be considered potential reviewers and involved in scientific reviews.
This paper uses data from the Computing science area (Scopus). In total, Scopus contains 7.5 million documents in the subject area of Computing Science. To ensure the use of exclusively relevant published data in experimental research, only articles from journals published in 2020-2021 were selected. Publication data is also limited to articles authored (one of the authors) by scientists from Ukraine. The proposed method can be easily adapted to other data (topic areas), regardless of the language of the data. 3.1.
For automated processing and formation of information-topic model from Scopus exported information about 2550 publications (Computing science, journal articles, 2020-2021 publications, authors from Ukraine). Statistics of publications are presented in Table 1.
Total articles
Year of publication 2020/2021
Authors with 4 or more publications Maximum number of articles per author
Geography of the authors
Number of original sources (journals) Subject Area to which the articles belong It should be noted that articles may be interdisciplinary. It means that the same paper simultaneously relates to two areas. One field is Computing science, and the other field is determined by the scope. This is evidenced by 24 Subject Area, which includes 2550 publications in Computing science. The TOP-10 Subject Area is presented in Table 2.
The interdisciplinary nature of articles submitted to the publication makes it difficult to solve the problem of finding a reviewer because he must be an expert in two disciplines (areas).
The method described in this publication takes into account these features and shows good results for interdisciplinary articles. 3.2.
Publishers' websites present a significant amount of material that describes the algorithm of actions and the review process, but not the search for a potential reviewer. In our opinion, in general, the process of finding a reviewer can be described by such an algorithm (Figure 1).
The method described in this article optimizes the most resource-intensive stage "Working with databases of publications and databases of authors (reviewers)", the result of which is a list of authors who could be involved in the expert analysis of the article. Also, the presented method partially simplifies the stage "Analysis of the list of authors for compliance with the requirements for the reviewer" in terms of monitoring the publication of applicants. We believe that this monitoring is included in the method presented in the article. The general algorithm of the method is presented on Figure 2.
Due to the use of methods of natural language processing and topic modeling, the presented method differs significantly from the general algorithm for finding a reviewer. The new method does not require additional subject area analysis of the article and carefully analyzes the publishing activities of applicants. The topic branch is chosen based on the received information-topic model. A list of reviewers is offered on the basis of BMI publication information.
Data from the Scopus database, namely metadata from Compute Science articles, were used as experimental data, 2,550 articles were taken in 2020-2021. Fields such as "Title", "Abstract", "Author Keywords", "Index Keywords" were used as basic data. There is enough data to use the Latent Dirichlet Allocation (LDA) method as a basic topic modeling algorithm.
Latent Dirichlet Allocation is a generative probabilistic model of a corpus. The basic idea is that the documents are represented as random mixtures over latent topics, where a topic is characterized by a distribution over words [22, 23]. The LDA provides information on topic collections of any text in general, individual documents, and the relationships between them. Article 1 belongs to class 1 with a probability of 0.8500712. The probability for 11 topic is very low. [(1, 0.8500712), (11, 0.14165811)]
Article 2 belongs to class 29 with a probability of 0.8471544. Other classes have a low probability value.
[(29, 0.8471544), (10, 0.066285245), (34, 0.050520007), (1, 0.024828823)]
Article 3 belongs to class 18 with a probability of 0.8551518. Other classes have a low probability value.
[(18, 0.8551518), (14, 0.03998026), (4, 0.039681997), (7, 0.033085056), (26, 0.021731576)] Article 4 belongs to class 22 with a probability of 0.9812607. Other classes have a low probability value.
[(22, 0.9812607), (28, 0.010214581)]
Article 5 belongs to class 30 with a probability of 0.9861704. Other classes have a low probability value.
[(30, 0.9861704)]
Class 30 is described by the following words "network", "artificial_intelligence", "mobile_robot", "mobile_robot", "measure", "code", "information", "service", "program", "geometric" and others. Thus, we can say that the model correctly classified Article 5 to 30 class.
Thus, we can conclude that the model works correctly and is suitable for optimizing/simplifying the process of finding a reviewer.
The next step was to find the names of potential reviewers. And for our data, the method found the following names of potential reviewers:
Article 1 Shevchenko I., Shastalo V., Kozak Y., Abramov Y., Basmanov O., Teslia I., Khlevna I., Yehorchenkov O., Latysheva T., Grigor O., Tryus Y., Prokopenko T., Polishchuk O.
Article 2 Kalyna V., Koshulko V., Ilinska O., Tverdokhliebova N., Tolstousova O., Bliznjuk O., Gavrish T., Stankevych S., Zabrodina I., Zhulinska O., Vedel Y.I., Denisov S.V., Semenov V.V.
Article 4
Hnativ L.O., Luts V.K., Zhdaniuk V., Volovyk O., Kostin D., Lisovin S.
There are some examples when the model with a small probability determines which topic the article belongs to. Examples of such cases are given below:
N.Y. Kuznetsov, I.N. Kuznetsov, Fast Simulation of the Customer Blocking Probability in Queueing Networks with Multicast Access, Cybern Syst Anal 57(4) (2021) 530-541.
[(27, 0.33189335), (30, 0.2498508), (2, 0.12633349), (22, 0.12138765), (15, 0.09545717), (11, 0.023521349), (29, 0.022272551), (25, 0.01759534)]
V.M. Bulavatsky, Mathematical Models with Local M-Derivative and Boundary-Value Problems of Geomigration Dynamics, Cybern Syst Anal 57(4) (2021) 563-577.
[(22, 0.64569837), (9, 0.18071947), (11, 0.16654922)]
V.M. Bulavatsky, Closed Solutions of Some Boundary-Value Problems of Filtration-Consolidation Dynamics within the Fractured-Fractal Approach, Cybern Syst Anal 57(3) (2021) 383-395.
[(11, 0.2851506), (12, 0.17952716), (1, 0.17065176), (19, 0.16245417), (22, 0.11737535), (25, 0.058207653), (32, 0.020618271)]
L.S. Stoikova, Exact Estimates of the Probability of a Non-Negative Unimodal Random Value Hitting Special Intervals under Incomplete Information, Cybern Syst Anal 57(2) (2021) 264-267.
[(12, 0.25641593), (25, 0.2526573), (15, 0.19586651), (8, 0.089407556), (11, 0.0873035), (0, 0.06457444), (10, 0.04478603)]
A.I. Ivaneshkin, A New Approach to Operating with Undirected Trees, Cybern Syst Anal 57(1) (2021) 124-132.
[(31, 0.2491094), (2, 0.22874023), (25, 0.21450539), (34, 0.09106176), (10, 0.060724407), (17, 0.05571771), (23, 0.051634632), (33, 0.024108535), (4, 0.01810252)]
For such publications, we can use the proposed reviewers, but with a more detailed analysis of the proposed candidates.
The process of finding reviewers for a journal article or conference is always a rather complicated process. It is not always possible to understand by the scientific interests of the reviewer (by keywords), whether he will be able to review the paper qualitatively, whether it is in the area of his scientific interests.
The approach we proposed using the methods of topic modeling and natural language processing allowed us to obtain not only research topics, but also take into account possible conflicts of interest, thanks to information from metadata. The topic modeling model was built on the basis of Scopus data and tested on journal articles. The results showed that our approach works quickly and efficiently. The authors proposed by our method are relevant to the research topic.
In addition, reference metadata that reflects the content of the article includes "References". However, references can contain both positive and negative links. These metadata also contain many words (tokens) that cause white noise. The use of "References" metadata needs further analysis and is planned for future research. In the future, we plan to test and evaluate the performance of our method for a real journal with existing data about reviewers.