The paper considers current research in the field of evaluation of scientific publications. The main methods of expert evaluation of information resources are analyzed. The use of the Delphi method for evaluation with the formation of a weighted assessment of scientific material is proposed. The main advantages of using the method for evaluation in the system of scientific libraries are described. The principles of the formation of an information product for the implementation of a system for evaluating scientific publications in libraries are considered.
2. Modern trends in the evaluation of scientific publications numbers. Indicators are not a substitute for well-founded assessments. Everyone is responsible for their own assessment. 2. Measure performance against the research missions of the institution, group or researcher. Programme goals should be stated at the start, and the indicators used to evaluate performance should relate clearly to those goals. The choice of indicators, and the ways in which they are used, should consider the wider socio-economic and cultural contexts. Scientists have diverse research missions. Research that advances the frontiers of academic knowledge differs from research that is focused on delivering solutions to societal problems. Review may be based on merits relevant to policy, industry or the public rather than on academic ideas of excellence. No single evaluation model applies to all contexts. 3. Protect excellence in locally relevant research.
In many parts of the world, research excellence is equated with English-language publication. Spanish law, for example, states the desirability of Spanish scholars publishing in high-impact journals. The impact factor is calculated for journals indexed in the US-based and still mostly English-language Web of Science. These biases are particularly problematic in the social sciences and humanities, in which research is more regionally and nationally engaged. Many other fields have a national or regional dimension – for instance, HIV epidemiology in sub-Saharan Africa. This pluralism and societal relevance tends to be suppressed to create papers of interest to the gatekeepers of high impact: English-language journals. The Spanish sociologists that are highly cited in the Web of Science have worked on abstract models or study US data. Lost is the specificity of sociologists in high-impact Spanish-language papers: topics such as local labour law, family health care for the elderly or immigrant employment. Metrics built on high-quality non-English literature would serve to identify and reward excellence in locally relevant research. 4. Keep data collection and analytical processes open, transparent and simple.
The construction of the databases required for evaluation should follow clearly stated rules, set before the research has been completed. This was common practice among the academic and commercial groups that built bibliometric evaluation methodology over several decades. Those groups referenced protocols published in the peer-reviewed literature. This transparency enabled scrutiny. For example, in 2010, public debate on the technical properties of an important indicator used by one of our groups (the Centre for Science and Technology Studies at Leiden University in the Netherlands) led to a revision in the calculation of this indicator. Recent commercial entrants should be held to the same standards; no one should accept a black-box evaluation machine. Simplicity is a virtue in an indicator because it enhances transparency. But simplistic metrics can distort the record (principle 7). Evaluators must strive for balance – simple indicators true to the complexity of the research process.
Simplicity is a virtue in an indicator because it enhances transparency. 5. Allow those evaluated to verify data and analysis.
To ensure data quality, all researchers included in bibliometric studies should be able to check that their outputs have been correctly identified. Everyone directing and managing evaluation processes should assure data accuracy, through self-verification or third-party audit. Universities could implement this in their research information systems and it should be a guiding principle in the selection of providers of these systems. Accurate, high-quality data take time and money to collate and process. Budget for it. 6. Account for variation by field in publication and citation practices.
Best practice is to select a suite of possible indicators and allow fields to choose among them. A few years ago, a European group of historians received a relatively low rating in a national peer-review assessment because they wrote books rather than articles in journals indexed by the Web of Science. The historians had the misfortune to be part of a psychology department. Historians and social scientists require books and national-language literature to be included in their publication counts; computer scientists require conference papers be counted.
Citation rates vary by field: top-ranked journals in mathematics have impact factors of around 3; top-ranked journals in cell biology have impact factors of about 30. Normalized indicators are required, and the most robust normalization method is based on percentiles: each paper is weighted on the basis of the percentile to which it belongs in the citation distribution of its field (the top 1%, 10% or 20%, for example). A single highly cited publication slightly improves the position of a university in a ranking that is based on percentile indicators, but may propel the university from the middle to the top of a ranking built on citation averages. 7. Base assessment of individual researchers on a qualitative judgement of their portfolio. The older you are, the higher your h-index, even in the absence of new papers. The h-index varies by field: life scientists top out at 200; physicists at 100 and social scientists at 20-30. It is database dependent: there are researchers in computer science who have an h-index of around 10 in the Web of Science but of 20–30 in Google Scholar. Reading and judging a researcher's work is much more appropriate than relying on one number. Even when comparing large numbers of researchers, an approach that considers more information about an individual's expertise, experience, activities and influence is best. 8. Avoid misplaced concreteness and false precision.
Science and technology indicators are prone to conceptual ambiguity and uncertainty and require strong assumptions that are not universally accepted. The meaning of citation counts, for example, has long been debated. Thus, best practice uses multiple indicators to provide a more robust and pluralistic picture. If uncertainty and error can be quantified, for instance using error bars, this information should accompany published indicator values. If this is not possible, indicator producers should at least avoid false precision. For example, the journal impact factor is published to three decimal places to avoid ties. However, given the conceptual ambiguity and random variability of citation counts, it makes no sense to distinguish between journals on the basis of very small impact factor differences. Avoid false precision: only one decimal is warranted. 9. Recognize the systemic effects of assessment and indicators.
Indicators change the system through the incentives they establish. These effects should be anticipated. This means that a suite of indicators is always preferable – a single one will invite gaming and goal displacement (in which the measurement becomes the goal). For example, in the 1990s, Australia funded university research using a formula based largely on the number of papers published by an institute. Universities could calculate the 'value' of a paper in a refereed journal; in 2000, it was Aus$800 (around US$480 in 2000) in research funding. Predictably, the number of papers published by Australian researchers went up, but they were in less-cited journals, suggesting that article quality fell. 10. Scrutinize indicators regularly and update them.
Research missions and the goals of assessment shift and the research system itself co-evolves. Onceuseful metrics become inadequate; new ones emerge. Indicator systems have to be reviewed and perhaps modified. Realizing the effects of its simplistic formula, Australia in 2010 introduced its more complex Excellence in Research for Australia initiative, which emphasizes quality.
Similar ideas were implemented at the assembly of stakeholders. Starting from January 2022 the
assembly formed the concept an agreement for reforming research assessment. On 8 July 2022, the final
version of the agreement was presented at a Stakeholder Assembly bringing together more than 350
organizations from more than 40 countries having expressed interest in being involved in the process.
On 20 July, the final Agreement is made public [
Consider the main ideas of this agreement.
The main idea is to evaluate scientific research based on qualitative rather than formal indicators. That is, to refuse to use in the evaluation of studies of impact factors of journals, Hirsch indices and other indicators that do not indicate the quality of a particular study. Instead, it is necessary to focus primarily on peer review, which includes the assessment of independent experts, which may be the scientists themselves.
The second, equally important idea, is a negative attitude towards plagiarism. Scientists engaged in plagiarism and academic dishonesty should not only be subjected to public criticism, but also deprived of their titles. Severe intolerance to academic integrity will help improve the quality of scientific publications.
The next important idea is to ensure the freedom of scientific research. Scientists should not conduct research under the pressure of their organizations or to provide certain criteria.
In addition to international discussions, some scientists published their vision of methods and
technologies for evaluating scientific publications [
Since users interested in scientific publications get access to them through scientific libraries or their Internet services, it is worthwhile to introduce an assessment system there. After reviewing the relevant scientific work, the user will immediately be able to evaluate it in the appropriate library system. 3. Model of the functioning of the scientific publication evaluation system
Consider the model of the functioning of the system for evaluating scientific publications (Figure 1). It functions at the expense of incoming information – scientific publication and information about the appraiser, and at the exit of the system information about the assessment is formed. The system is influenced by the requirements for scientific publications and legislation. The legislation refers to the requirement for scientific publications at the state level. Separate requirements for scientific publications are the requirements of a specific journal or resource where they are published.
To establish the functioning of the system, two methods will be used: work with the database (all information about the assessment of scientific publication will be stored in the database) and the method of expert assessments (it will determine the assessment).
It is advisable to break the process of evaluating a scientific publication into subprocesses in order to better understand the functioning of the system (Figure 2). It all starts with the process of establishing the compliance of a scientific publication, at the entrance of which there is a scientific publication, and at the exit – data on compliance, guided by the process of requirements for scientific publications and legislation. There is also a process of determining the weight of the assessment. The process takes place based on information about the appraiser. The result of the process is the evaluation coefficient. The mechanism of the process is the method of expert assessments.
Based on the information obtained from these processes, data analysis takes place. For this process, the same mechanisms are used as for the previous two. The output is the analyzed data. Using the following process, they are written to the database. The process uses the mechanism of working with the database.
The next is the process of forming a comprehensive assessment, which takes place based on information derived from the previous process – a set of relevant assessments. At the exit from the process, we get information about the assessment.
The process of determining the weight of the assessment consists of four subprocesses (Figure 3), three of which formally must occur in parallel, although their order is not important. These first three processes carry out the determination of the status of the appraiser, the establishment of involvement in the topic, the determination of the position of the appraiser in the group.
The processes take place based on information about the appraiser. The mechanism of working with the database is used, but at the exit they give different information: the status of the appraiser, involvement in the topic, position in the group. Based on this information, the process of calculating the coefficient of importance of the assessment takes place, which additionally uses the method of expert assessments and as a result issues an assessment coefficient.
For a better understanding of the diagrams described in the diagrams and the proposed information product, it is necessary to conduct a thorough analysis and carry out the formulation and justification of the problem that needs to be solved on certain key points.
The purpose of the development is to create a module for evaluating scientific publications in scientific libraries of higher education institutions. Consolidation of information about appraisers will allow a qualitative and objective assessment of scientific material, because the method of expert assessments will be used. This, in turn, will simplify the search for quality materials in the system, help identify popular scientific publications, motivate scientists to better prepare materials.
Purpose of the information product. As noted above, the issue of evaluating scientific publications in scientific libraries is relevant and unresolved. Both scientists and students can act as experts. However, it is not enough to derive the arithmetic average of their grades, because the assessment of a student and a scientist with a academic title have a different "weight". The consolidated information obtained as a result of the work of the proposed information resource is intended for further use in determining an objective assessment of a scientific publication, to search for better quality materials.
The place of application of the proposed product is the branch of information technology, which is used in the activities of scientific libraries, in higher education institutions.
Justification of the development of an information product. The growing number of scientific publications, the commercialization of electronic resources of scientific activity leads to the task of assessing the quality of these scientific resources. Of course, the criteria and methods of such an assessment have their own specifics, which in many respects differ significantly from the well-known methods for assessing the quality of printed materials. The task of assessing the quality of scientific publications in the libraries of higher education institutions is becoming more urgent every year. That is why the development of criteria and methods that will solve this problem is an important and necessary task. Therefore, it is necessary to develop an information resource on which publications will be expertly evaluated.
Expected effects from the development of an information product. As a result of the use of consolidated data, the productivity of libraries will increase, and time costs will also decrease – less time will have to be used to search for high-quality content.
The process of interaction in the system begins with the receipt and registration of publication in the library collection, after which the work of the system for evaluating scientific materials begins, which displays materials to the user. Based on several user ratings, the system after some time displays the rating to the author and other users of the system (Figure 4).
It is also advisable to describe the sequence of the system of evaluation of scientific publications. The rater puts a rating/review into the system. The system, using the mechanism for determining the coefficient, establishes the "weight" of the assessment. In the correction system, the score is converted, after which it is recorded in the system. Then the system displays the score in the library catalog (Figure 5). 4. Using the Delphi method to evaluate scientific publications
Each user of the scientific library uses registration data to enter the system, with the help of which user data is determined: scientific direction, position, academic title, academic degree. That is, data to determine the "weight" of the assessment can be taken from the library system.
The scientific content of the library will be evaluated according to the following criteria: relevance, compliance, clarity, completeness, reliability and design.
Appraisers can be called experts, that is, persons who are entrusted with expressing an opinion about a controversial or complex case. Formally, there will be assessments of several groups of experts depending on the degree. According to the appraiser's belonging to one of them, the "weight" of his assessment will be established. The following subgroups are highlighted: a user without an academic title and a scientific degree, with an academic title, with a scientific degree, with an academic title and a scientific degree.
Today, there are many methods of organizing examinations (interviews, questionnaires, Delphi
method, commission method, etc.) and methods for finding out the competence of experts and the
weight of their assessments [
The Delphi method was initially considered solely as a method of forecasting, but later it turned out that it also has quite significant analytical capabilities.
Unlike the commission method, where the coordination of expert opinions is achieved in an open discussion, the Delphi method involves a complete rejection of collective discussions. This can significantly reduce the influence of such psychological factors as the need to join the opinions of reputable specialists, unwillingness to abandon previously expressed opinions, adherence to the opinion of the majority.
The Delphi method is characterized by three features that distinguish it from other methods of collective expert evaluation: • anonymity; • adjustable feedback; • statistical processing of examination data.
Anonymity is achieved by the fact that the members of the group are unknown to each other. In the Delphi method, direct debates are replaced by a carefully designed program of sequential individual surveys conducted in the form of questionnaires. In contrast, the interview method involves a conversation between the organizer of the examination (forecaster) with an expert specialist in a certain field of knowledge, which is carried out in accordance with a pre-developed program. A significant drawback of this method is the lack of time for the expert to prepare answers.
Quantitative assessment of the measurement of the phenomena under study requires the establishment of a gradation, which will allow each expert to put a certain number that characterizes the usefulness (value, importance, period, etc.) of the trait (object, process, phenomenon).
Feedback is one of the main properties of the Delphic procedure. Regulated feedback is achieved through several rounds of the survey. The classic Delphi method involves four rounds of survey. After each round, the experts' answers are summarized, a system of averaged indicators is determined, and together with additional information, the results of the calculations are sent to experts, which allows you to clarify and adjust the initial answers. Such a procedure is repeated until an acceptable convergence of the totality of the judgments made is achieved.
Thus, the use of the results of the previous round of the survey, supplemented by the statistical characteristics of the group response, allows each expert to get acquainted with the opinions of their anonymous colleagues, compare their answers with the generalized conclusions of the entire group of experts.
The processing of examination data consists in determining a system of indicators that allow to assess how the response of each expert corresponds to the point of view of the group of experts.
The method of analytical notes is carried out in writing (questionnaire) by sending the expert questions on interested issues, to which unambiguous answers should be obtained. Questions can be both open and closed. In the latter case, answers should be offered. The questionnaire can be sent by regular or e-mail, but there must be a preliminary agreement with the expert in advance. This method implies a high level of qualification of the organizers of the examination at the stage of asking questions and organizing the survey, as well as in terms of processing the information received.
In contrast to the interview method, the method of analytical notes provides an opportunity for an expert to conduct long and thorough work on the analysis of trends, assess the state and ways of development of the projected object. This method allows the expert to use all the information he needs about the object of forecasting. The expert draws up his reasoning and conclusions in the form of an analytical note.
The main advantages of the considered methods are the possibility of maximizing the potential of experts and the slight psychological pressure exerted on specialists.
A significant disadvantage of the method of individual expert assessments is that not every expert takes the responsibility to independently assess complex phenomena (processes, objects) without considering the opinions of other experts. The lack of scientific connections between experts, the limited knowledge of individual specialists makes the considered methods little suitable for predicting the most complex general strategies.
Information support for conducting research using the Delphi method are questionnaires and tables filled out by experts. The questionnaire is essentially a specially written document, which contains a pre-prepared set of questions focused on achieving the goal of the examination. In case of evaluation of scientific publications of libraries, experts will be invited to evaluate the material according to criteria from 1 to 5.
The examination procedure based on the Delphi method is divided into stages, the sequence and content of which is determined in accordance with the nature and complexity of the object under study (phenomenon, process, problem).
The most typical stages of the examination include [
As a conclusion, we can say that the Delphi method is best suited to solve the problem. To do this, it is advisable to consider an example. The methodology for solving the problem of resource allocation between alternative projects in conditions of systemic uncertainty is based on the application of expert assessment methods. Its essence consists in obtaining quantitative and qualitative expert assessments of the project from a group of experts, aggregating them into a single project assessment (the degree of attractiveness of the project) with an analysis of the coordination of expert assessments, and in the subsequent selection of projects in accordance with their attractiveness and distribution of resources between them.
To determine the position of experts on individual questions, the calculation of certain statistical indicators is used, which are used depending on how the question was formulated, and what answers were offered. If experts used quantitative parameters (a point scale) to express an opinion, then averages are calculated to calculate the generalized opinion. If the composition of experts is homogeneous, then a simple arithmetic mean (1) is calculated. where Vi – individual assessment of each expert; n – number of experts.
If the composition of experts is heterogeneous (as in conducting our expert assessment), then among the expert group are identified leading experts whose opinion on certain issues is more significant. Each of the experts in such cases is assigned a weighting factor (k).
At the same time, to obtain a generalized opinion on a particular issue, the weighted arithmetic mean (2) is calculated. where Vi – individual assessment of each expert; ki – weighting factor; n – number of experts.
An example of a scheme for calculating the assessment of scientific content is presented in Table 1. Calculation of the assessment according to the Delphi method = ∑
, = ∑ ∙
, V 3 2 3 2 3 3 4 V 4 4 4 4 4 4 (1) (2)
Let's start considering the calculation from the bottom of the image. A scientific publication will be evaluated according to 6 criteria: relevance, compliance, clarity, etc. For each group of experts, the weight factor of each of the criteria will be different. For example, for scientists without an academic title and a scientific degree, the clarity of the material is more important than design. The expert for each criterion of the material sets an assessment, which is multiplied by the weighting factor and a balanced assessment of the criterion is obtained. In total, we get a real assessment of the scientific publication by an expert. In each group of experts there are those whose assessment has a higher weighting factor, therefore, multiplying the real estimate by the weighting factor and suturing them, we get the sum of the weighted assessments of experts, which will be a weighted assessment of the scientific publication. Thus, using the Delphi method, a comprehensive assessment is calculated.
The development of a generalized software product with the introduction of such an assessment method into the system of scientific libraries will allow to form a base of weighted assessments of scientific publications in the library funds
The development of an information product for the evaluation of scientific publications will provide library staff of scientific libraries with a tool that would contribute to increasing the relevance, pertinence and reliability of search results. Today, researchers strive for fast and accurate information search results, which in turn places new requirements on the functioning of scientific libraries.
Standard cataloging is no longer enough to meet the information and search queries of library users. Like searching the Internet, searching in library information resources requires rating of available materials. Evaluation of the information content of library information resources helps to implement their ranking with the subsequent formation of the issuance of the result of a search query.
The implementation of the proposed method of evaluating scientific publications in the information system of libraries will solve several problems. The main ones are, improving the relevance of information search to meet the needs of library users, completing library collections with high-quality publications, establishing information support for leading scientists of the educational institution. 6. References