The paper proposes the formation problem of the Common Digital Space of Scientific Knowledge (CDSSK) content. The CDSSK is a computer environment, where the user should get exhaustive answers to questions concerning the achievements in various fields of science. The CDSSK should contain both basic and popular-scientific information and consist of a subspaces related to the different science fields. Its content should include factual data, source link where they are from, and these sources full texts. One of the major challenges in the formation of CDSSK is the selection of materials to be included in its content. One of the solutions to this problem is expert assessment, which should be carried out by a fairly wide range of highly qualified specialists in this field of science. This paper describes the expert WEB-based selection material system for inclusion in the CDSSK. This system was carried out by the Joint Supercomputer Centre of the Russian Academy of Sciences in 2019. The structure of the system, its functions, metadata profiles included in it, and the results of practical implementation are given in the paper. The system was tested at two all-Russian competitions - for the best scientific monograph and for the best student qualification work. Several hundred works were submitted to the competitions, and several dozen experts from various regions of the country participated in their evaluation.
The problem of selecting materials for inclusion in databases, electronic libraries and other information systems is one of the most important in the formation of such resources. It is extremely important when forming the content of of a common digital space of scientific knowledge (CDSSK). CDSSK is a computer environment, where 1 The research is carried out by JSCC RAS — branch of SRISA within the framework of the State task 0580–2021–0016. the user should get exhaustive answers to questions concerning the achievements in various fields of science. This environment should contain the true basic and popularscientific information represented as different object types such as encyclopedic articles, factographic databases, some phenomena digital models, archival , museum and library storage materials, presented in the form of digitized texts, images, and multimedia resources.
The CDSSK consists of a number of subspaces complex related to separate scientific
disciplines, each of which reflects its own specifics and has its own subject ontology,
built on the same principles for all subspaces. The join ontology provides the
connection between subspaces [
Despite each subspace specifics, all of them must have a documentary component, including full document texts that record scientific results. It is based on monographs, reference books and textbooks that reflect generally recognized, time-tested scientific knowledge, and are "classic" for each science field. It is clear that in any scientific field there is a significant number of classical works, many of which duplicate each other. In this connection there is the selection problem of the most important classical publications for inclusion of their full texts in CDSSK.
In addition to the time-tested publications, the CDSSK should contain new knowledge, which is also in scientific publications. The annual number of scientific publications appearing in the world (both in print and electronic forms) is the millions. As far as, by definition, the CDSSK should contain only reliable scientific information that is relating to knowledge, there is a selection problem of the most significant scientific publications which are the knowledge community, "worthy" for presenting to the CDSSK.
To solve these problems, the authors suggest to use an iterative process, which provides the first material choice with using certain objective indicators (the publishing prestige, citation, author's qualifications, etc.), followed by expert evaluation of preselected materials. The fairly wide range of highly qualified specialists must implement the expert assessment in this field of science.
As already pointed out, there are not only published materials but also various
problem-oriented information systems in the CDSSK. The appreciable quantity of such
systems is in many areas of both the natural and humanitarian sciences.
As [
The organization of an expert community in various areas within a particular science field is a separate task that should be solved by the relevant structures that organize the work on the formation of the CDSSK subspaces. In particular, experts from the Russian Academy of Sciences, which is responsible for examining all scientific developments in the country, can act as the UDSSK experts.
It is obvious that the examination of materials, which are for showing in the CDSSK,
is necessary to be in networking with the experts from different regions. It is also
necessary to develop a typical automated system that can be configured for different
types of objects and for different assessment systems should be developed for
conducting the examination. Such a system should contain a database of objects subject
to examination and a database of experts. The result of its work is a rating list of
objects, ordered in accordance with the received estimates, taking into account the
possible "weights" of experts. The prototype of such a system can serve as the expert
system of acquisition [
«Expertise» system structure
The storage of digital objects to be evaluated can be centralized (in this case, objects are uploaded to it from the outside), or it can be distributed (for example, publications stored in various external electronic libraries are evaluated). It is important that each object has its own unique identifier, which can be used to call it to the expert's computer screen. Objects can be text files, image files, or multimedia files. When an external information system is evaluated, the identifier included in its metadata profile should allow for versatile work with the system to form its multidimensional assessment.
On par with metadata describing experts and assessment objects, in the «Expert» system generates a number of overhead generic tables: - a rating table containing verbal and numeric expressions; - Thematic areas table; - Academic degrees table; - Table of organizations; - User rights table.
The system provides for the work of various groups of users, each of which has its own rights. These groups include administrators, editors, experts, and managers. The administrator has the rights to configure the system, enter users and set their rights. The editor has rights to enter, view, and correct data. The expert has the right to view and evaluate objects. The Manager has the right to view the results of experts' work and create rating lists. 3
The Expert system implements the following functions.
The system configuration by the administrator for specific metadata profiles, types of ratings, and user rights.
User authorization in accordance with the settings and providing opportunities to work with the system depending on their status.
The object and expert metadata batch loading from files of the specified structure (performed by the editor).
Manual loading of metadata using a special user interface that provides formal logical information control (performed by the editor).
Editing and deleting metadata (performed by the editor).
Search for objects by various metadata elements and their combinations using Boolean logic operators (performed by any user).
Visualization of found objects (available to any authorized user).
Input of object evaluations by experts who have the same thematic focus in their metadata as the object being evaluated.
Changes in experts ' previous estimates.
Visualization of estimates given to each object by each of the experts (available to the Manager).
Calculation of total and average estimates of individual objects or their groups (available to the Manager).
Creating rating lists of objects within a given group (available to the Manager). Output of calculation results to Excel files.
Automatic e-mailing of information to experts about the receipt of new objects that correspond to their thematic interests.
Data backup and recovery in case of failures.
In the process of operation, the system implements the "publication - expert – assessment" connections. Many experts can evaluate one publication; one expert can evaluate many publications; each expert can assign only one estimate to the publication. The system is based on Microsoft technology ASP.NET 4 on the Microsoft .NET Framework in the Microsoft Visual Studio 2017 development environment. 4
The system feature "Expertise" is the ability of its tenability to solve different
problems. One of the classes of such problems is the tendering netting process. The system
«Expert» modification, called "Competition", was of use in 2019 to conduct
competitions for the best scientific monograph and the best qualifying student work,
organized by the publishing house "Direct-Media "(PH DM) [
As far as a number of experts evaluated papers for both competitions, to avoid duplication, a single database of experts was being created, but in the profile of their metadata, there was an indication which competition (s) this expert evaluates. A screenshot of the page for entering expert metadata is shown in Fig. 1. Since the expert system has a Russian-language interface, the translation of their text elements is given below the screenshots (here and below). The expert's academic degree and the science branches in which they specialize are from the tables of academic degrees and subject areas. • Competition expert (Yes/no).
Search terms are being input in one or more fields. The «academic degree» values and "science sections" fields the users select from drop-down lists. When you enter terms in multiple fields, they will be linked by the logical operator «And». In fig. 2 there is an example of searching for records registered in the experts’ system that have a candidate's degree in Economics and are experts in the «Monographs» competition. In the list of experts obtained, their names and related organizations are active links. When you click on the selected link, the system shows the expert's metadata, which can be edited if necessary.
The rating tables for various contests had configurations by the system administrator to the lists set by the contest organizers (see Fig. 3).
During the examination, the system provides the expert with a list of available thematic areas in accordance with the specified area of thematic interests when registering. After selecting a focused area, the expert will be able to view the publication descriptions in the specified subject area. Under each publication description is a link to its full text, as well as a link to "Evaluate" (if the publication doesn’t have an assessment by this expert). Clicking on the "Value" link allows the expert to select a rating from the list downloaded from the rating table and enter the necessary comment (see Fig. 3). After selecting an estimate, the system returns to publication list in the corresponding thematic area, where the evaluated publication appears with the link "the Publication already has your estimate ". The expert can always change their estimate and comment by clicking on this link.
The system Manager access to the estimates issued by competition experts is given. After selecting a theme, the system will display descriptions of publications sorted in descending order of the total points received. Under the description of publications that have already been evaluated by at least one expert, there will be an average estimate, as well as a list of experts who have evaluated the publication, indicating the surname, first name and patronymic of the expert, his organization, the estimate and comment (see Fig. 4).
Fig. 4. Example of viewing publication ratings The evaluation of student papers submitted to the competition was conducted from mid- July to mid -August 2019. The competition received 1,100 papers from 425 universities. The thirty two experts evaluated them. The competition jury reviewed the final rating lists of works. The winners in the “bachelor’s” and “master’s” categories were selected and awarded for each of the 12 research areas.
There were 315 books from scientific organizations and universities for the monograph competition. There was more competition time, taking into account the estimate complexity, compared to the student papers competition - it lasted from 1.05 to 15.11 2019. The forty specialists took part in the examination, and the winners were for each of the two categories (scientific and popular science publications). 5
Conducted competitions show that the developed system is quite reliable, simple and
easy to use, both from the editor viewpoint and from the expert viewpoint. There were
no questions about working with the system during the contests that lasted for several
months. The Expert system can be configured to hold online competitions of various
topics and scales, ranging from intra-University competitions to national and
international ones. It can be successfully used for the selection of literature to be included in
the National electronic library of Russia [
The examination work of scientific materials takes significant time and intellectual
expenses. In this regard, one of the serious issues that must have a solution when
organizing an expert examination is expert motivation. The LNS RAS expert’s work the
evaluation of the book market was motivated by the fact that (a) the experts received
information on new publications and (b) when LNS RAS receives a book that was
positively evaluated by an expert the LNS automated system sends
him a special notification and he can immediately receive this book. During the
competitions of scientific papers mentioned above, the incentive to participate in the
expert examination was to provide the expert with free access for a year to all
publications available in the «Direct-media» electronic library. Currently, this library
contains «more than 70,000 scientific and educational publications, educational
multimedia, interactive tests, 150,000 painting reproductions» [
Given the importance of forming a Common digital space of scientific knowledge and its reliable and most significant content, leading scientists of the country should be involved in expert selection of materials must be included in this space. They must have the appropriate motivation to do this work. Motivation should be not only and not so much material, but also an understanding of the necessity and social utility of this work. One of the possible solutions to this problem is to include work on the examination of scientific materials in the state task of leading scientific organizations in the country.