=Paper=
{{Paper
|id=Vol-3241/paper8
|storemode=property
|title=Calculation of the Integral Quality Index of a Scientific Event in the Context of the Interests of a Scientific Institution
|pdfUrl=https://ceur-ws.org/Vol-3241/paper8.pdf
|volume=Vol-3241
|authors=Hryhorii Hnatiienko,Vitaliy Snytyuk,Nataliia Tmienova
|dblpUrl=https://dblp.org/rec/conf/its2/HnatiienkoST21
}}
==Calculation of the Integral Quality Index of a Scientific Event in the Context of the Interests of a Scientific Institution==
https://ceur-ws.org/Vol-3241/paper8.pdf
Calculation of the integral quality index of a scientific event in
the context of the interests of a scientific institution
Hryhorii Hnatiienko 1, Vitaliy Snytyuk 1 and Nataliia Tmienova 1
1
Taras Shevchenko National University of Kyiv, Volodymyrs’ka str., 64/13, Kyiv, 01601, Ukraine
Abstract
The concept of scientific action is introduced and the concept of a scientific event is
proposed. New approaches to assessing the excellence of training and the quality of the
scientific event are considered. A mathematical model for quantifying the performance index
of a scientific event in the context of the interests of the organization is proposed.
Approaches to aggregation of attributes of scientific actions are investigated and types of
convolutions of partial criteria are given. Heuristics to define the mathematical model with
information that is missing when filling the database of scientific actions are introduced. It is
proposed to divide the problem of determining the effectiveness of a scientific event into
three stages. The attributes of each of the three stages of conducting and evaluating the
effectiveness of the scientific event are considered. Prospects for further research on the
problem of evaluating the effectiveness of scientific actions carried out by a particular
organization are presented.
Keywords1
formalization, scientific action, scientific event, aggregation, integral quality of the event,
expert evaluation, decision making, interests of the organization
1. Introduction
Our time is characterized by a wide variety of manifestations of human activity in various fields
and at the same time the desire of different institutions to unify this diversity in order to restore order
and ensure coherence and interaction between different institutions. In particular, to solve the problem
of commensuration, state regulators and international institutions create regulations and formalize the
interaction between teams of people from different fields. The scientific space is also characterized by
the difficulty of measuring results, a wide range of approaches to comparing the activities of scientists
and research teams. In this regard, there are generally accepted agreements on scientometric
databases, metrics of scientific action, criteria of publication activity, citations, impact indices,
quantitative indicators of productivity, impact factors, etc.
Determining the intellectual and scientific level of any scientific event in modern large-scale
information flows is an important scientific problem. The main factor in the scientometric approach to
calculating the level of a scientific event should be its efficiency and effectiveness. The problem of
determining the quality and effectiveness of scientific research is poorly structured, therefore it is
necessary to take into account subjective factors at all stages at its solution. In this case, to adequately
solve this problem, we should choose criteria that reflect the purpose and comprehensively
characterize the quality of the scientific event.
High quality of scientific actions (SA), which ensure a high level of reliability of scientific
information, is a necessary condition for sustainable development of science and technology because
science is based on the principles of presumption of proof [1]. Therefore, adequate evaluation and
even better quantitative evaluation of any scientific event (SE) are especially important and
XXI International Scientific and Practical Conference "Information Technologies and Security" (ITS-2021), December 9, 2021, Kyiv, Ukraine
EMAIL: : g.gna5@ukr.net (H. Hnatiienko); snytyuk@gmail.com (V. Snytyuk); tmyenovox@gmail.com (N. Tmienova);
ORCID: 0000-0002-0465-5018 (H. Hnatiienko); 0000-0002-9954-8767 (V. Snytyuk); 0000-0003-1088-9547 (N. Tmienova)
© 2021 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
79
promising. This will facilitate the possibility of comparing scientific events with each other,
comparing the level or significance of periodic events in different periods.
The growth of the total number of SA, the increase in the volume of articles submitted for review
and presented at particular SA, the spread of scientific content of scientific events as a result of
technology development, the evolution of universities and scientific research have increased the risk
of dissemination of poor quality scientific information [2]. The presence of low-quality publications
in proceedings of scientific events not only lowers the rating and threatens the reputation of individual
authors, SA, publishers, any repository in which they are located, but also weakens the desire of all
participants to ensure high-quality scientific events [ 2].
All this requires the introduction of a mathematical apparatus for adequate comparison of SE in the
scientific space of Ukraine. This is due, in particular, to the fact that the current stage of human
civilization development is defined as the transition to a society of knowledge and is characterized by
qualitatively new requirements for the development of science. Moreover, the scientific space is a
structural element of social space and can be considered as a complex poorly structured system.
2. Classification of scientific activities
Scientific space is a network of cognitive processes within which science operates [3]. The concept
of scientific space outlines the complex configuration of not only the knowledge component of
science but also the scientific infrastructure which includes scientific institutions, scientific
communications, processes of scientists training for professional activities, etc. [4, 5]. SE is a
component of the scientific space, which in turn is a complex poorly structured system, which should
be studied by methods of decision theory. At the same time, the study of scientific space is a difficult
formalized task. Therefore, its formalization can contribute to meaningful and sound decision-making
in this subject area [6].
Note that formal procedures for determining the level of scientific publications are widely used by
scientometric databases. But this is done in order to maintain the high quality of scientific
publications at the world level. At the same time, such an approach may be considered necessary for
further research. After ensuring the necessary conditions of scientific integrity and high quality of the
SA, an additional study of the "sufficiency" of the SE in the context of the interests of the scientific
institution may be conducted. Exactly like this perspective of research is relevant in today's
competitive world, in particular, in the field of research and comprehensive rankings, which are
widespread in the modern world.
Today there is a great variety of SA [7, 8]: congress, symposium, forum, conference, round table,
school, seminar, exhibition, etc. [9, 10]. All these activities after their realization, i.e. their
implementation and achievement of their goal, will be referred to as a scientific event. Depending on
the scope, SE is accepted to differentiate between scientific and theoretical, scientific and technical,
scientific and practical. In addition, depending on the area covered, SE can be international, national,
interregional, regional, etc. But such a classification is too superficial and can not serve a quantitative
study of the importance of SE. Gradation of SE should be more reasonable, detailed, and formalized
[11, 12].
The given list of SE is not exhaustive. In particular, taking into account the scope of this article,
SE can also be considered as a publication of a scientific monograph, the publication of the next issue
of the periodical edition, its indexation in scientometric databases, and more.
SE is a form of organization of scientific action in which researchers present and discuss their
research and their results. There are no clear rules for conducting an SE [7, 13]. They are usually
determined by the organizing committee of SA. There are only some guidelines that are determined
by the criteria for inclusion in the Plan of conducting an SA of the Ministry of Education and Science
of Ukraine [14, 15]. In particular [15, 16], in an international conference, as a rule, there should be at
least five participating countries, and the number of participants should be more than one hundred
people.
SE is an important channel for the exchange of oral information between scientists, as well as the
publication of a written report on the study presented at the SE [2]. It is very important that, as a rule,
80
the SE is important not only for the scientists who take part in it but also for the scientific
organization to which these scientists belong and under the auspices to which the SE is conducted.
This affects the ratings of the organization, the popular indices of influence of scientists and the
organization as a whole, and so on.
A number of requirements are a prerequisite for the publication of SA materials and their indexing
in Scopus and/or WoS. To transform SA into SE, it is advisable to formulate some requirements that
are a necessary prerequisite for determining the rating of SE:
SA policy: editorial policy, review policy, geographical representation of the program
committee, the geographical diversity of SA participants;
content and scientific weight of SA materials;
authority of SA: citation of the participants of SA in Scopus and/or WoS, the authority of the
editorial board and members of the SA Program Committee, number of publications in the
DBLP database, etc.;
he number of years during which SA is organized and becomes SE;
the language of SA, which contributes to its international recognition;
the quality of the SA site, the quality of the SA materials, etc.
It should be noted that the provision of the necessary conditions for the SA conducting should be
comprehensive, all the details and tasks should be provided to prevent a situation that could hinder
success. In particular, the abuse or neglect of publishing ethics is unacceptable and, of course, is also
a necessary condition for the transformation of SA into SE.
3. Statement of the problem
In order to orient in the scientific space, to organize and substantiate the assessment of the SE, as
well as to adequately determine the significance of the SE, it is necessary to choose parameters that
fully and adequately characterize the conducting of the SA. In addition, it is necessary to propose a
uniform scale for measuring SE.
It is clear that the analysis of this information should be preceded by recognition of open data
contained on relevant sites, scientometric databases, etc. that is names of authors, the affiliation of
authors with scientific organizations, links to publications of authors of relevant organizations,
recognition of related attributes, searches for related information, etc.
Let's assume that the number of SA that should be compared and/or determined their integral
effectiveness (efficiency, usefulness, importance, status, weight, quality, significance) in the context
of the interests of the organization, is equal to n and the set of their indices is J 1,..., n.
The problem of determining the effectiveness of the SE should be divided into several stages,
which give SA the status of SE and affect the level of integrated quality of the SE:
1. Defining the policy of organization and conduct of SA.
2. Making a decision on the preparation and publication of SA materials.
3. Determining quantitative indicators of the effectiveness of SE in terms of the interests of the
organization.
In order to prepare a mathematical model to determine the effectiveness of SE, we consider a set of
attributes of SA zi , i I 1,..., k , j J , which can be used to determine quantitative indicators of
j
SE effectiveness and used to compare the effectiveness of different SE.
4. Approaches to determining the quality of a scientific event
As noted above, in the context of the problem publishing the next issue of the scientific journal
and its indexing by scientometric databases are also SE. Therefore, the approaches used by Scopus
and WoS to determine the level of scientific journals can also be applied to identify the integrated
quality of SE.
Approaches to determining the quality of a scientific event may be similar to determining the
impact factor of scientific journals. But, unlike journals, there is much more diversity among SA. In
addition, they are not regular and periodic. They are usually one-off or annually.
81
At the same time, by analogy, we can consider the publishing of the next issue of the scientific
journal as SE. By applying the approach described in this paper, we can also determine the
effectiveness of each issue of the journal for the organization. Prerequisite for the application of this
approach is to ensure preliminary text recognition [17, 18]: names of authors, affiliation of authors,
search for affiliated with organization authors among the cited literature, definition of indexing in
Scopus and/or WoS links presented in the article [19, 20], etc.
Ensuring an appropriate definition of the integrated quality indicator of the SE holding requires the
construction of an adequate model for relevant and sustainable evaluation of the effectiveness of the
SE.
Stage I. Carrying out expert assessments of some parameters and limits of SE attributes change.
Stage II. Introduction of a dynamic approach to determining the quality of SE, clarification of
parameter values and limits of attributes change.
Stage III. Application of a static approach to determining the quality of SE based on the analysis of
the SE database: when filling the database with information is sufficient.
4.1. Expert approach
Attributes and intervals for changing parameters of the model are set by experts. All volumes of
SE materials that are subject to indexing by scientometric databases can be considered as
interdependent in some way. This is due to the fact that all SE materials are included in a higher level
system that is the world scientometric database. In addition, the level of efficiency of the SE is a
relative value and is determined on the basis of comparison with other SEs. In particular, significant
parameters are selected by experts [21, 22]. At the stage of expert approach preference vectors,
weights of parameters, ranking of objects, coefficients of competence of information sources [23, 24],
etc. can be determined in different ways [25, 26].
4.2. Dynamic approach
When applying this approach, the attributes and intervals of the values of the SE characteristics
change as the database is filled. Such changes and clarifications must be made in accordance with the
regulations and may not be permanent. It is necessary to clearly define the time, circumstances, and
criteria for both the implementation of changes and the limits of their change and the actions that
accompany these changes.
In this case, a methodological question arises: do we need to list the values obtained in the initial
stages of the automated system processing for determining the intervals of parameters change, or
leave them as obtained at the beginning of the calculation? The answer to this question may be the
introduction of appropriate heuristics to ensure certainty and unambiguity.
Note that the resulting level (integral index) of SE quality can be determined on different scales
[21, 27]. The representation of the SE effectiveness index can be given by a fixed number, as an
interval or as a function of belonging to a fuzzy set. In turn, these types of integrated effectiveness
index of SE can be measured in different specified ranges [28, 29].
4.3. Static approach
If the SE database is sufficiently full, a static approach can be used to determine the effectiveness
of SE. In this case, the values of the parameters of the mathematical model and the limits of the SE
attributes remain unchanged for different decision-making situations. In this case, all the limits of the
intervals of changing parameters are fixed and do not affect the mathematical model adopted to
determine the quality of SE.
In cases when the static approach is not effective enough, comments and suggestions are
accumulated, new versions of the statically accumulated database and the system of formalization of
SE as a whole are created [30, 31].
82
5. Mathematical model for determining the policy of organizing and
conducting a scientific event
We can consider several evaluation options when it is not possible to select a standard:
measurements in absolute values;
comparison of priority options with each other and formalization of the choice in the form of
a matrix of pairwise comparisons, ranking in the ordinal scale [32] or the vector of weight
coefficients in the cardinal scale [33];
comparison of each variant with a set of variants, if the characteristics of this set, distribution
of values, etc. are known.
Three stages of decision-making are considered:
1. A priori definition of SE policy.
2. Making a decision on the preparation and publication of materials for SE.
3. Preparation for the decision to accept the article for publication and determination of
quantitative indicators of the SE effectiveness in terms of interests of the organization.
In determining the indicators of the SA participants, the Program Committee, the Organizing
Committee should take into account, in particular, their representation in the database DBLP that is
the site of the bibliography of computer science at the University of Trier in Germany [34]. This
service accumulates and provides open bibliographic information on the materials of the main SEs.
To formalize SA and weigh the quality of SE, we assume that in order to study this area it is
considered n SE, which form a set of indices I 1,..., n.
5.1. A priori definition of the policy of the scientific event
A priori constraints, according to the policy of the scientometric base or another resource, are a
necessary condition for the transformation of SA into SE. In addition, at the first stage, the Program
Committee of SA should determine the target values of the parameters:
z11i – representation of the countries of the world among the members of the Program Committee -
the number of represented countries, i I ;
z12i – the relative number of foreign members of the SA Program Committee, i I ;
z31i – geography of the countries represented in the Program Committee, i I ;
z14i – average DBLP index of the Program Committee, i I ;
z51i – lower limits on DBLP indexes of author, i I ;
z16i – use tools like EasyChair or traditional email, i I ;
z17i – official languages of the conference, i I ;
z81i – expert assessment of the quality of SA site, i I ;
z91i – expert assessment of the importance of these factors, i I .
5.2. Making a decision on the preparation and publication of materials for a
scientific action
The main parameters of the second stage are:
z12i – number of foreign authors, i I ;
z 22i – average DBLP index of the authors of the event, i I ;
z32i – limit on the number of citations, i I ;
83
z 42i – the allowable number of citations of Ukrainian authors, i I ;
z52i – lower limits on DBLP indexes of author, i I ;
z 62i – allowable number of self-citations, i I ;
z 72i – allowable number of mutual citations, i I ;
z82i – the number of reports submitted to SE, i I ;
z92i – the number of reports accepted at SE, i I ;
z102i – the ratio of accepted/rejected reports, i I ;
z112i – the presence of a positive history of publishing and indexing in scientometric databases,
iI .
5.3. Making a decision on the acceptance of articles for publication and
determination of quantitative indicators of the effectiveness of SE in terms of
the interests of the organization
The parameters of the third stage include:
z13i – the number of articles of the affiliated organization, i I ;
z 23i – the number of authors of the affiliated organization, i I ;
z33i – the total number of published pages of SE materials by the authors of the affiliated
organization, i I ;
z 43i – the average number of pages per article in the SE materials, i I ;
z53i – the number of mutual citations of the authors of the affiliated organization among the SE
materials, i I .
5.4. Formalization of the problem
The tasks of effectiveness determining of SE will be formalized in the class of problems of
multicriteria optimization. Herewith, given the need to use heuristics in such cases, we will pay
considerable attention to the subjective component of multicriteria problems.
Note that today there are three main approaches to describing the problems of introspective
(internal, deep) analysis: using binary relations, the function of choice and the criterion approach. The
latter approach involves the assumption that each alternative can be evaluated by a specific number,
which is the value of the criterion, so the comparison of alternatives is reduced to comparing the
corresponding numbers. It becomes obvious that in the practice, multicriteria is a way to increase the
goal description adequacy [33, 35, 36].
Let us f1 z is the function of the quality of preparation and conducting of SE, and f 2 z is the
function that reflects the risks of non-publication of SE materials and, accordingly, the impossibility
of considering it as a scientific event. In this case, in the second stage of solving the problem of the
effectiveness of the SE the decision-maker must solve at least a two-criteria problem.
The problem of multicriteria optimization for the case of determining the quality of the SE is
formalized in the following formulation:
f 1 z max,
f 2 z min,
z A, A E k ,
84
where A is the range of allowable values of quality indicators of SE, which are limited by the
requirements of scientometric databases to scientific publications and are characterized in our case by
two parameters, that is belong to the space E 2 ;
y ( z ) ( f1 ( z ), f 2 ( z )) is the vector of assessment of alternatives or criteria determined by the
mapping f : A E 2 .
Additional heuristics can be introduced to determine the problem.
Heuristics E1. The minimum participation of the author in SE is one publication, zero citations of
own articles and zero mutual citations.
Heuristics E2. The maximum participation of the author in SE is a function of the maximum
possible number of publications and a function of the maximum possible number of self-citations and
mutual citations. Moreover, these indicators are determined expertly or statistically.
The use of heuristics E1 and E2 allows to supplement the mathematical model and adequately
apply the monotonic functions of transformations of the values of the model parameters to the
dimensionless form.
6. Tools for determining the effectiveness of SE holding.
To determine the quantitative indicators of the effectiveness of SE holding, all the values of the
parameters z ii , i I 1,..., k , j J 1,..., n are translated into dimensionless space by applying
monotonic transformations: i j zii , i I 1,..., k , j J .
Methods of processing expert information are divided into three main groups, which are currently
well researched and adequately reflect the nature of expert information:
statistical methods,
scaling methods,
algebraic methods.
The essence of algebraic methods is that the distance is given on the set of acceptable estimates
and the resulting estimate is defined as such, the distance of which to the given estimates is minimal
by a certain selected criterion [33, 37].
When solving the problem of comparing the quality of SE special attention should be paid to the
aggregation of group estimates. One of the widespread tasks of expert assessment is the choice in a
fixed in advance class of relations of some resulting (compromise, group, collective) relation, which
is consistent with the given in some way. It is possible to construct a convolution (generalized,
aggregating, integral criterion of quality of an alternative) in many ways and this procedure
necessarily includes an element of subjectivity. The convolution method should be justified only to
the extent that the complete order generated by the convolution must be consistent with the given
partial order. There are known dozens of methods of aggregation, some of which can be used to solve
the problem of aggregation of indicators of integrated quality of SE [37, 38].
6.1. Aggregation of the effectiveness of the SE holding
The lessons learned of expert estimation in numerous fields of human activity shows that any
statistical operations become more useful and justified if the number of features used for analysis is
reduced. Therefore, the problem of aggregating the features that characterize SE to a smaller number
of constructed "factors" (aspects, etc.) occupies a significant place in the tasks of the effectiveness
determining of scientific activities. The analysis of the set of SE estimates by group of parameters is
to determine the level of overall consistency of SE estimates and to identify, if necessary, in the group
a "homogeneous" subgroups that combine SE parameters with agreed estimates. The formulation of
these problems is dictated by the fact that the transition to the aggregation of estimates for different
parameters is possible only after identifying the structure of preferences. For example, if the overall
85
consistency of estimates by parameters is low and the group of parameters is divided into several
subgroups, within which the consistency of estimates is high, then aggregation should be performed
for these subgroups by estimates of parameters.
In the analysis of SE quality estimates and in determining the relative importance of SE
publications, there are problems of presenting these estimates in a systematic way, there are problems
of comparison and aggregation of estimates. The usage of mathematical methods in the analysis of
expert assessments permits to adequately sum up the conclusions of specialists and identify the
information they have in hidden form [38, 39].
6.2. Types of convolutions
Most often on the basis of several conflicting indicators of n SA, "convolution" (aggregation,
integration, generalization, etc.) of indicators with indices i I 1,..., k of each SA in some single
integrated indicator Q j Q z 1j ,..., z kj , j J , is carried out.
To construct a convolution denotes to expand the partial order on the set of SE estimates to the
complete one [33, 37]. This can be done in many mechanisms, and necessarily includes an element of
subjectivity [40]. In this regard, it is sometimes believed that the convolution method should be
justified only to the extent that the complete order generated by the convolution must be consistent
with the natural partial order.
The subset of indicators that are essential for determining the integrated rating of SE is selected
from the general set of indicators, for example, expertly.
Among the most common are the following families of convolutions, adapted to a 100-point scale:
linear convolution
(1)
Q j1 1 i i j * 100 , j J ,
i I
multiplicative convolution
(2)
Q j 2 1 i i j * 100 , j J ,
i I
generalized convolution of indicators, which is also called the principle of "bottleneck"
i I i I
Q j 3 max i i j / max i * 100 , j J , (3)
nonlinear convolution using quadratic metrics
n (4)
Q j4 1 ii *100, j J ,
2
i 1
where i i z i , i I , j J , are the normalized values of the parameters of SE z i j , i I ,
j j j
j J , defined by monotonic transformations;
i , i I , are the weight coefficients of the parameters by which the SEs are estimated.
The approaches described in formulas (1) to (4) reflect the definition of the distance to an ideal
point. In the case of calculating the integral quality of SE, the ideal point is a point with parameters
i 0, i I .
It is also advisable to use additive convolution to determine the integrated rating value of SE:
(5)
Q j 5 i z i j / i max z i j min z i j * 100 , j J ,
i I i I j J j J
In addition to formulas (1) - (5), other types of convolutions can be used and substantiated to
determine the numerical value of the integral quality of SE.
86
When determining the values of the parameters that characterize the effectiveness of SE, it is also
necessary to determine the methods of aggregation and consideration of these indicators. Several
approaches can be used:
averaging the values of parameters of all authors who are included in the published materials
of SE;
combining the values of the parameters of all articles of SE of the organization affiliated to
the scientometric database;
synergetic approach, for example, the value of the h-index of the affiliated organization (for
its unambiguous comparison with other organizations).
7. Computational experiment
Based on the proposed approaches to determining the integrated indicator (index) of SE quality,
several scientific conferences were considered, the materials of which were published in the
international open access archive CEUR Workshop Proceedings.
According to the concept set out in this paper, the importance of these SEs was considered taking
into account the interests of a particular institution - Taras Shevchenko National University of Kyiv.
Additional heuristics should be used.
Heuristics E3. Any participation of a scientist affiliated with the organization in SA is a significant
event and can be taken into account when calculating the rating of SE.
Heuristics E4. All parameters are significant and their weight is determined adequately. For this
purpose, procedures for restoring the values of weight coefficients may be proposed depending on the
subjective preferences of the decision-maker.
Since the software of the mathematical support of the problem of determining the integral quality
of SE described in this paper is at the stage of implementation, it is logical to use an expert approach
to calculate the values of indices of integrated quality of SE. In addition to the heuristics E3 and E4,
we assume that the maximum possible number of scientific publications of authors affiliated with one
organization is 30, and the number of authors is 50. The maximum number of references to literature
the authors of which affiliated with this organization is 100, and the maximum number of DBLP
indexes increased as a result of SE is 50.
We present the intervals of change of parameters and coefficients of their importance for decision
making in the form of Table 1.
Table 1
Intervals of change and weight coefficients of parameters
Parameter name Minimum value Maximum value Weight coefficient
Number of Papers 0 30 0,1
Number of Authors 0 50 0,1
Number of Citations 0 100 0,5
Index DBLP 0 50 0,3
We present some basic situations of decision-making and the values of the parameters near the
basic situations in the form of Table 2.
Table 2
Values of parameters that characterize the basic situations of decision making
Parameter name Situation Situation Situation Situation Situation Situation Situation
1 2 3 4 5 6 7
Number of Paper 30 0 1 1 0 0 0
Number of Authors 50 0 1 0 1 0 0
Number of Citations 100 0 1 0 0 1 0
Index DBLP 50 0 1 0 0 0 1
87
Let's calculate the values of the criterion functions near the basic situations and present them in the
form of Table 3.
Table 3
The value of criteria near basic situations of decision‐making
Criterion name Situation Situation Situation Situation Situation Situation Situation
1 2 3 4 5 6 7
Linear convolution of a
form (1) 100 0 1,63 0,33 0,2 0,5 0,6
Multiplicative con‐
volution of a form (2) 100 0 8,09 3,33 2 1 2
Generalized con‐
volution of a form (3) 100 0 1,2 0,67 0,4 1 1,2
Quadratic convolution
of a form (4) 100 0 1,63 0,33 0,2 0,5 0,6
Additive convolution of
a form (5) 100 0 1,37 0,14 0,14 0,68 0,41
Table 3 illustrates that in basic situations, the values of all individual functions are acceptable and
reflect the content load that was expected to determine the number of values of SE.
To conduct a computational experiment, we take six real SE, which are presented in the
international archive CEUR and present the values of the main parameters of the selected
experiments, in the form of table 4.
Table 4
The values of the parameters that characterize SE, selected from CEUR
Parameter name SE 1 SE 2 SE 3 SE 4 SE 5 SE 6
Number of Paper 3 3 2 14 3 5
Number of Authors 5 4 5 21 7 12
Number of Citations 0 0 9 51 13 8
Index DBLP 0 0 5 18 0 12
Consider the values of the criterion functions of the form (1) - (5), which describe the ratings ofSE,
at the points whose coordinates are given in table 4. These values are presented in the form of a table
5.
Table 5
The values of the parameters that characterize SE, selected from CEUR
Criterion name SE 1 SE 2 SE 3 SE 4 SE 5 SE 6
Linear convolution of a
form (1) 2 1,8 9,17 45,17 8,9 15,27
Multiplicative convolution
of a form (2) 19 17,2 31,2 90,3 32,66 55,72
Generalized convolution of
a form (3) 2 2 9 51 13 14,4
Quadratic convolution of a
form (4) 1,92 1,73 9,16 44,77 8,71 14,93
Additive convolution of a
form (5) 1,1 0,96 9,18 47,12 10,27 12,74
88
It should be noted that the final decision, namely which metric among formulas (1) - (5) should be
used for numerical determination of the integral significance of SE, as always, is made by the decision
maker. Because these formulas reflect different approaches to determining the significance of SE.
Criteria for such a choice may be, for example, which metric is more sensitive, adequate from the
point of view of the decision-maker, and so on.
It is clear that the problem of determining the weights of parameters, the development of adaptive
decision-making procedures require further study. In particular, based on which of the metric graphs
presented in Table 5 is the closest to the subjective perception of the decision-maker about the rating
of SE depending on the achieved indicators.
8. Conclusion
In this paper the techniques to research of efficiency of SE are investigated. The authors assumed
that the consequence of scientific research is a publication that confirms the fact of scientific
achievement, with which the scientist will be able to acquaint not only his colleagues but also the
world community. The scientific work is not completed until it is published and indexed in a
scientometric database. In the modern scientific world, publishing activity is becoming increasingly
important for every scientist, regardless of the scope of his research, and the scientific organization or
the university as a whole.
The following main scientific results were obtained:
theoretical investigation of the problem of creating a system of assessment and control of
effectiveness (efficiency, quality, significance) of the SE is carried out;
the role of the individual component in the decision support system for assessment the
integrated quality of research materials is studied;
techniques to measuring qualimetric indicators of effectiveness of research is developed;
criteria of assessment the quality and effectiveness of scientific activities are explained;
approaches to determining the ratings of SEs are proposed, taking into account the need to
motivate employees of organizations;
interpretation of integral effectiveness of SE is offered and substantiated;
a computational experiment on real data on published SE materials for a specific scientific
organization is conducted.
In the perspective of conducting research in this scope, it is advisable to automate and implement
in the form of DSS tool for automated retrieval of information from the scientometric database and
create software for quality analysis of conducting of SE.
References
[1] Simonov P.V. Selected works: In 2 vols. Vol. 1. Brain, emotions, needs, behavior. - M.: Nauka,
2004. – P. 396.
[2] Kulamer B., Meester W., Salk J., Blair-DeLeon N., MacPherson G., Moses W., Philippidis A.,
Chapman C., Smith D., Stoneham I., Vukmirovic K. Recommended Practices to Ensure
Technical Conference Content Quality // 4th World Conference on Research Integrity Rio De
Janeiro, Brazil, June 2, 2015. URL:
https://www.ieee.org/conferences_events/conferences/publishing/paper_acceptance_criteria.pdf
[3] Moed H., Glanzel W., Schmoch U. Handbook of Quantitative Science and Technology Research.
Springer Netherlands. 2004. 800 p. ISBN 978-1-4020-2755-0.
[4] Ying Ding. Scientific collaboration and endorsement: Network analysis of coauthorship and
citation networks. Journal of Informetrics. 2011. 5 (1). C. 187 – 203. doi: 10.1016/j.joi.2010.10.008
[5] Biloshchytskyi A., Myronov O., Reznik R., Kuchansky A., Andrashko Yu., Paliy S., et al. A
method to evaluate the scientific activity quality of HEIs based on a scientometric subjects
89
presentation model. Eastern-European Journal of Enterprise Technologies. 2017. Vol. 6, Issue 2
(90). – P. 16–2 2. doi: 10.15587/1729-4061.2017.118377
[6] Kuchansky, Alexander. (2020). Components of scientific network analysis. Management of
development of complex systems, 41, 115–126, dx.doi.org\10.32347/2412-9933.2020.41.115-126
[7] Recommendations on how to organize scientific and methodological and initial seminars / Ed.-
style: E.M. Bachinska, N.I. Klokar, V.I. Urus. - Bila Church - Ternopil, 2004. - 48 p.
[8] Rach V.A. Methodology of the systemic approach and scientific reports: Heading guide / V.A.
Rach, O.V. Ignatov. – Luhansk: type of SNU im. V.Dalya, 2010. – 210 p.
[9] Biryukov O. V. Scientific and communicative approach and approbation of scientific results:
nomination, recognition, recognition / O. V. Biryukov, V. V. Zelenska, N. O. Zakharova //
Project management and development of virobnitstva. – 2017. – No. 3. – Pp. 76-96. – Access
mode: http://nbuv.gov.ua/UJRN/Uprv_2017_3_6
[10] Birta G.O. Methodology and organization of scientific research. [text]: textbook way. / G. O.
Birta, Yu.G. Burgu – K .: Center for Educational Literature, 2014. – 142 p.
[11] Robert Day, Barbara Gastel. How to Write and Publish a Scientific Paper. Cambridge:
Cambridge University Press, 2012. – P.9.
[12] Richard Van Noorden. Publishers withdraw more than 120 gibberish papers // Nature. –
doi:10.1038/nature.2014.14763.
[13] Cyril Labbé. Ike Antkare one of the great stars in the scientific firmament (англ.) // International
Society for Scientometrics and Informetrics Newsletter. – 2010. – June (vol. 6, no. 2). – P. 48-52.
[14] Instructional recommendations on proposals from higher education institutions for the formation
of the Plan of the Ministry of Education and Science of Ukraine for international, national
scientific and practical conferences and seminars of young scientists and students // Supplement
to the letter of the Ministry of Education and Science of Ukraine 28.09.2009 №1 / 9-662.
[15] Instructional recommendations on the organization and conduct of international, all-Ukrainian
scientific-practical conferences and seminars of students and young scientists // Supplement to
the letter of the Ministry of Education and Science of Ukraine 19.09.2013 № 1 / 9-649.
[16] On holding international, all-Ukrainian congresses, forums, scientific and practical conferences
and seminars of students and young scientists for 2016 // Letter of the Ministry of Education and
Science of Ukraine 14.09.2015 № 1 / 9-435.
[17] Grant S. Ingersoll, Thomas S. Morton, Andrew L. Farris. Taming text: how to find, organise,
and manipulate it / Shelter Island, NY : Manning. 2013. 298p.
[18] Hillary Mason. The Next Generation of Data Products (2017). http://bit.ly/2GOF894
[19] Rebecca Bilbro, Tony Ojeda, Benjamin Bengfort. Applied Text Analysis with Python: Enabling
Language-Aware Data Products with Machine Learning / O'Reilly Media, Incorporated, 2018. –
332p.
[20] Lane Hobson, Howard Cole, Hapke Hannes Max. Natural Language Processing in Action:
Understanding, analyzing, and generating text with Python/Manning Publications, 2019. – 544 p.
[21] Elkind, E. Distance rationalization of voting rules / E. Elkind, P. Faliszewski, A. Slinko. // Social
Choice and Welfare. 2015. Vol. 45, No. 2. Pp. 345–377.
[22] D’Ambrosio, A. Kemeny’s axiomatic approach to end consensus ranking in tourist satisfaction /
A. D’Ambrosio, V.A. Tutore. // Statistica Applicata. 2008. Vol. 20, No. 1. Pp. 21–32.
[23] Bury, H. Zastosowanie mediany Litvaka do wyznaczania oceny grupowej w przypadku
występowania obiektów równoważnych / H. Bury, D. Wagner. // Studia i Materiały Polskiego
Stowarzyszenia Zarządzania Wiedzą. 2007. Vol. 10. Pp. 19–34.
[24] List, C. Judgement aggregation: a survey / C. List, C. Puppe. // In: Oxford handbook of rational
and social choice. Oxford: Oxford University Press. 2009. Pp. 457–482.
[25] Bury, H. Group Judgement With Ties. Distance-Based Methods / H. Bury, D. Vagner. // In: H.
Aschemann (ed.). New Approaches in Automation and Robotics. Vienna: I-Tech I-Tech
Education and Publishing. 2008. Pp. 153–172.
[26] Cook, W.D. Distance-based and adhoc consensus models in ordinal preference ranking. / W.D.
Cook. // European Journal of Operational Research. 2006. No. 172. Pp. 369–385.
90
[27] Hudry, O. Complexity of computing median linear orders and variants / O. Hungry. // Electronic
Notes in Discrete Mathematics. 2013. Vol. 42. Pp. 57–64.
[28] Amodio, S. Accurate algorithms for identifying the median ranking when dealing with weak and
partial rankings under the Kemeny axiomatic approach // S. Amodio, A. D'Ambrosio, R.
Siciliano. // European Journal of Operational Research. 2015. No. 249. Pp. 667–676.
[29] Alnur, A. Experiments with Kemeny Ranking: What Works When? / A. Alnur, M. Meila. //
Mathematical Social Sciences. 2012. Vol. 64, No. 1. Pp. 28-40.
[30] A. Dodonov, D. Lande, V. Tsyganok, O. Andriichuk, S. Kadenko, A. Graivoronskaya (2019).
Information Operations Recognition. From Nonlinear Analysis to Decision-Making, Lambert
Academic Publishing, 2019, 275 p.
[31] Hnatiienko H., Tmienova N., Kruglov A. (2021) Methods for Determining the Group
Ranking of Alternatives for Incomplete Expert Rankings. In: Shkarlet S., Morozov A.,
Palagin A. (eds) Mathematical Modeling and Simulation of Systems (MODS'2020). MODS
2020. Advances in Intelligent Systems and Computing, vol 1265. Springer, Cham.
https://doi.org/10.1007/978-3-030-58124-4_21. Pp. 217-226.
[32] Hnatiienko H., Snytyuk V. A posteriori determination of expert competence under uncertainty /
Selected Papers of the XIX International Scientific and Practical Conference "Information
Technologies and Security" (ITS 2019), pp. 82–99 (2019).
[33] Voloshin, A.F., Gnatenko, G.N., Drobot, E.V. The method for indirect determination of intervals
of weight coefficients of parameters for metrized relations between objects(Article) // Problemy
Upravleniya I Informatiki (Avtomatika)Issue 2, 2003, Pages 34-41.
[34] Ley, Michael. Maintaining an Online Bibliographical Database: the Problem of Data Quality //
EGC, ser. Revue des Nouvelles Technologies de l'Information: journal. – 2006. – Vol. RNTI–E–
6. – Pp. 5-10.
[35] M. Rakushev,Y. Kravchenko, O. Permiakov, O. Lavrinchuk,V. Bychenkov, V Krainov,
“Modeling of solving stabilized differential equations by differential-Taylor transformations”,
IEEE 2nd International Conference on Advanced Trends in Information Theory, ATIT`2020,
Proceedings, pp. 216–221.
[36] Hnatiienko H. Choice Manipulation in Multicriteria Optimization Problems / Selected Papers of
the XIX International Scientific and Practical Conference "Information Technologies and
Security" (ITS 2019), pp. 234–245 (2019).
[37] Hnatiienko, H., Snytyuk, V., Tmienova, N., Voloshyn, O. Determining the effectiveness of
scientific research of universities staff / CEUR Workshop Proceedings, Volume 2833, 2021,
Pages 164-176 // 7th International Conference ""Information Technology and Interactions"", IT
and I 2020; Kyiv; Ukraine; 2 December 2020 through 3 December 2020; Code 167962
[38] Hnatiienko, H., Kudin, V., Onyshchenko, A., Snytyuk, V. and Kruhlov, A. Greenhouse Gas
Emission Determination Based on the Pseudo-Base Matrix Method for Environmental Pollution
Quotas Between Countries Allocation Problem / 2020 IEEE 2nd International Conference on
System Analysis & Intelligent Computing (SAIC), Kyiv, Ukraine, 2020, pp. 150-157, doi:
10.1109/SAIC51296.2020.9239125.
[39] Hnatiienko H.M., Snytyuk V.Y., Suprun O.O. Application of Decision-Making Methods for
Evaluation of Complex Information System Functioning Quality // Selected Papers of the XVIII
International Scientific and Practical Conference "Information Technologies and Security" (ITS
2018). Kyiv, Ukraine, November 27, 2018. Pp.56-65.
[40] Kraevsky, V., Kostenko, O., Kalivoshko, O., Kiktev, N., Lyutyy, I. Financial Infrastructure of
Telecommunication Space: Accounting Information Attributive of Syntalytical Submission /
IEEE International Scientific-Practical Conference Problems of Infocommunications, Science
and Technology (PIC S&T). DOI: 10.1109/PICST47496.2019.9061494.
91