The article presents a method for predicting the result of submitted applications to scientific tenders, based on the criteria for their assessment. Statistics of applications submitted to external tenders from ITMO University was obtained through the implemented automated system for registering applications. This system performs the functions of forming a register of the organization's applications, and also allows checking applications for compliance with the requirements of the selected tender. For each application, the system records thematic information on the application, information about the its manager, participants and the result obtained. In the article the main groups of criteria for evaluating the application are identified, this groups have the greatest impact on applications efectiveness. To obtain characteristics related to the experience of the project team members, information on the projects already underway was additionally used. These characteristics were considered both separately for the application manager and for the entire team to identify the dependence of the influence of the experience of the manager and the team on the result in various tenders. The presented method can be used to forecast the efectiveness of the organization's applications.
Currently, obtaining funding from scientific grants plays an important role in the academic life of a scientist. This is a kind of recognition of his scientific activity. The receipt of such a grant is influenced by various factors, such as publications in high-ranking journals of the Scopus and Web of Science databases, the availability of intellectual activity results, the level and number of previously supported grants, etc. Having such information about a scientist, one can predict his chances of obtaining scientific grant, provided that the project proposed by him corresponds to the world level. The possibility of obtaining grants allows scientists to implement new projects that contribute to attracting funding to universities and obtaining significant scientific results.
Much attention is paid to the issues of increasing the efectiveness of the scientific activity
of a scientist; various methods are used to evaluate it. One of such methods is a scientometric
assessment of the main results of scientific activity. In [
Similar studies with scientometric indicators for various tenders were carried out in astronomy,
chemistry and mathematics [
In article [
In [
Therefore, the task of creating a favorable environment in the university for the implementation of scientific activities of scientists is very urgent. To solve it, the university has organized a number of business processes aimed at informing university staf about new tenders, assistance and advice in filling out an application for a tender, as well as collecting the necessary signatures and certificates. It is important not only to increase the number of applications submitted by the university, but also to ensure their high-quality content. To solve this problem, an automated system for registering applications has been created at the university, which helps users in making decisions [10, 11].
This system also allows you to streamline the application process and evaluate the applications submitted from the university by the Department for Information Support of Open Tenders for state and municipal needs (DISOT) [12]. The introduction of an automated system for registering applications at ITMO University made it possible to control and record statistical data on submitted applications for various tenders. This system performs the functions of forming a register of the organization’s applications, and also allows checking applications for compliance with the requirements of the selected tender. For each application, the system records thematic information on the application, information about its manager, participants and the result obtained. Also in the developed system the ability to view the characteristics of a group in the form of a network is implemented (Fig. 1). This presentation makes it possible to more clearly assess the characteristics of the research group.
The introduction of the registration system of applications into the information system of the university [13] allowed: • To increase the eficiency of DISOT work through the use of built-in information tools. • Create a knowledge base for pre-project activities, in which information is structured with the possibility of reuse. • To form a base of additional information about applications, such as: publication activity of participants in the application; availability of the results of intellectual activity; the number of participants with academic degrees; experience in project management. • DISOT employees quickly receive various statistical data, which are output from the system and are visible to the university employees interested in this information, including top managers.
For the period from the beginning of 2015 to September 2020, detailed statistical information was collected on the order of a thousand applications. In order to determine the profile of successful project teams, an analysis of applications for which the results of tenders are known was carried out. The analysis made it possible to identify the main groups of criteria for evaluating the application, which have the greatest impact on its efectiveness. These groups of criteria underlie the proposed method for predicting the result of applications submitted to scientific renders from the criteria for their evaluation.
The following criteria were defined for each application: 1. The number of publications over the past 7 years. 2. The number of publications indexed in the Scopus database. 3. The number of publications indexed in the Web of Science (WoS) database. 4. The number of publications in peer-reviewed journals included in the list of Higher
Attestation Commission (HAC). 5. The number of publications indexed in the Russian Science Citation Index (RSCI) database. 6. The number of publications in journals Q1 category. 7. The number of publications in journals Q2 category. 8. The number of publications in journals Q3 category.
9. The number of publications in journals Q4 category. 10. The number of certificates of state registration of computer programs, databases for the past 7 years. 11. The number of patents for inventions and utility models. 12. The number of project participants. 13. The number of bidders with a PhD degree. 14. The number of bidders with a Doctor of Science degree. 15. The numerical assessment of the compliance of the keywords indicated in the application with the topics of the State Rubricator of Scientific and Technical Information (SRSTI). 16. The number of application filed by participants over the past 4 years. 17. Percentage of winning entries from participants over the past 4 years. 18. Maximum Hirsch index (H-index) of the bidder. 19. The number of projects in the application manager performed at the ITMO University.
To obtain characteristics related to the experience of the project team members, in addition to information about applications, information on projects already being implemented was used. These characteristics were considered both separately for the application manager and for all participants to identify the dependence of the influence of the experience of the application manager and the team on the result of the application in various tenders. In total, nineteen main criteria were identified. Characteristics 15, 16, 17 were determined on the basis of accumulated statistics on applications. Other characteristics were determined on the basis of information available in the information system about bidders and their electronic portfolios. To determine characteristic 15, we used data from a survey of university researchers based on information modeling of statistical data on the thematic characteristics of the application and methods of expert assessments.
Based on the results of the survey, the links of the project keywords with the corresponding SRSTI codes are described [13]. In the course of the analysis, the most frequently used SRSTI codes and the most popular keywords were identified, as well as their relationship was implemented in the system. The implementation of the relationship made it possible to automate the process of checking the prepared scientific projects for the correctness of its assignment to the specified SRSTI codes, as well as recommend the SRSTI codes to application managers, by keywords, and vice versa.
To determine the relationship between each criterion for evaluating the application and the result obtained, the Pearson correlation coeficient was calculated. Consider the set of characteristics (criteria) for a group of participants for all applications ( is the total number of applications):
, ∈ , , ≥ 0, where = 0..14, = 1...
Each characteristic is a real number. The value = 0 corresponds to the order result. If 0, = 0, then the -th application is considered not supported, if 0, = 1, then on the contrary supported. The characteristics for the application manager are defined in a similar way:
, where = 0..14, = 1...
To determine the relationship between each criterion for evaluating the application and the result, the Pearson correlation coeficient was calculated. For a group of participants: ∑︀=1 , − × 0, − 0 = √︁∑︀ =1 , − 2 × 0, − 0 2 where = 1..14, is the average value of the -th characteristic.
For application managers:
∑︀=1 , − × 0, − 0 = √︁∑︀=1 , − 2 × 0, − 0 2 where = 1..14, is the average value of the -th characteristic.
Relationship assessment was performed for team and application manager characteristics. Also, a separate calculation of the assessment was carried out for all applications and separately for (1) (2) (3) (4) applications within the framework of tenders for activities of the Federal Target Program (FTP), the Russian Science Foundation (RSF) and the state application. The results are shown in Table 1.
The highest connection between the characteristics of the team and the application manager with the results of the application can be observed on the example of tenders of the Federal Target Program "Research and Development in Priority Areas of Development of the Scientific and Technological Complex of Russia for 2014-2020", according to characteristics: 1, 2, 3, 7, 11, 16, 17, 19 (Fig. 2 and Fig. 3). The following criteria have the greatest influence on the result of the application when assessing the characteristics of the team and the application manager for the RSF tenders: 2, 3, 6, 16, 17 (Fig. 2 and Fig. 3). When evaluating all applications, criteria 16, 17 and 19 have the greatest impact on the result.
Confirmation of the influence of the characteristics of the application manager on applications submitted to tenders for FTP activities is a comparison of the average profiles of the group of managers with supported applications (denote group A) and the group of managers with unsupported applications (denote group B) (Fig. 4). As can be seen from the presented graph, the average indicators of group A are higher than those of group B. This indicates that, in general, according to the FTP tenders, application managers with high indicators win more often than managers with low indicators.
There are also contests where the scientific groundwork of application managers is not so important. This dependence is presented in Table 1, using the example of tenders held within the framework of a state assignment.
For the main groups of application evaluation criteria under consideration, which have the greatest impact on its efectiveness, the values of the evaluation indicators for the profiles of the team and the application manager were calculated, according to the applications available in the system, using the example of FTP and RSF tenders.
The main groups of criteria for evaluating the application that have the greatest impact on its efectiveness: the total number of publications; the number of publications indexed in Scopus; the number of publications indexed in Web of Science; the number of publications from the journals on the HAK list; the number of publications indexed in the RSCI; the number of publications from magazines Q1; the number of publications from magazines Q2; the number of patents; the number of project applications from the application manager; the percentage of applications supported by the application manager; H-index of the application manager; the application manager projects.
For won and unsupported applications, the median profiles of the application manager and the project team were determined. The results are shown in the Figure 5 in the form of the radar charts with normalized data (relative to the maximum value of each characteristic) along the median profiles.
The proposed graphs show a higher importance of the characteristics of the application manager, in comparison with the team. Figure 6 shows an example of the diference between the median profile of the winning manager in the FTP and RSF competitions.
This graph shows that in the FTP tenders there are higher requirements for the application manager in comparison with the Russian Science Foundation. Using information on the statistical profiles of a winning and not a second-hand manager for various tenders, it is possible to develop a methodology for generating recommendations for selecting a tender for a certain manager, and the inverse problem is to determine the competencies that a manager can improve in his profile in order to increase the likelihood of his winning.
The obtained results of the application of the proposed method showed that of the evaluation criteria, the experience of the previous project activity of the application manager and the percentage of applications won by him earlier has the greatest influence on the result of evaluating the application for tenders within the framework of the FTP and RSF activities. Also, an important influence on the result of the application evaluation is exerted by: the number of publications indexed in the Scopus and Web of Science databases by the application manager, the number of patents he received and the total number of applications filed by him.
Currently, a lot of attention is paid to the level of publications, many studies of the influence of scientometric indicators on the assessment of the scientific activity of a scientist are carried out [14, 15, 16]. For tenders held by the Russian Science Foundation in evaluating the publications of the application manager, one article published in the Q1 journal is considered two. Our research has confirmed that when evaluating publications for the Russian Science Foundation, the results of the application are influenced by the articles published in the Q1 journals.
The result obtained is due to the fact that in the indicators for evaluating applications of these tenders, there is a criterion "Qualifications and scientific achievements of the key project executors", the content of which includes academic degrees and titles, the number and level of scientific publications and their citation, results of intellectual activity and the criterion "Experience work of the key performers of the project in the implementation of scientific and technical projects", which includes the experience of key performers in the implementation of projects of a comparable or higher scientific and technical level. The criteria for evaluating a group of participants, with the exception of the number of publications in peer-reviewed journals included in the list of journals of the Higher Attestation Commission, and indexed in the RSCI database, have little efect on the result of evaluating the application.
Application of the proposed method for predicting the result of applications submitted to scientific tenders from the criteria for their assessment will allow: the application manager to make the right accents when describing the scientific and technical backlog of the project and describing the reputation of the participants in the application in a certain scientific field; representatives of the top management of the university to assess the quality of the information provided in the application on the scientific and technical groundwork of the team of project participants and its reputation component, as well as to predict the probability of winning the applications for participation in the tender submitted from the university.
The proposed method can be used to generate a forecast for the efectiveness of the organization’s applications. This topic is also relevant since obtaining funding from scientific grants plays an important role in the academic life of a scientist. To build a forecast, it is planned to study various models used in the banking, scientific and social spheres [ 17? , 19]. The proposed criteria are planned to be used in the future to train a neural network, which will be used to assess the quality of applications and predict their results. It will also analyze the influence of characteristics with a low estimate of the relationship with the result of the application on the quality of the forecast. The information obtained can be used to create an organization profile, which will provide information on the areas in which the university specializes, competencies will be visible in those areas of science in which scientific projects are submitted, which will allow using the information received for internal and external consumers, to which they can include top managers of the organization, customer organizations and industrial partners. [8] Shelton R.D. Scientometric laws connecting publication counts to national research funding.
Scientometrics, 123 (1), 181-206 (2020) [9] Grinev A.V. The Use of Scientometric Indicators to Evaluate Publishing Activity in Modern
Russia. Herald of the Russian Academy of Sciences, 89 (5), 451–459 (2019) [10] Markina G.L., Shley M.D., Kuznetsova O.V., Mazur Ya. V., Markina T.A., Leontieva V.
Assessment of the potential of scientific activity of the team of performers when submitting applications for participation in the competitive selection. Science Science Internet Journal, T. 9, No. 1, 19 (2017) [11] Markina G.L., Shley M.D., Kuznetsova O.V., Tushkanov E.V. Approaches to quality assessment of requests submitted for public tenders in the automated project control systems. 2017 International Conference IT&QM&IS, 436-439 (2017) [12] Shley M.D., Markina G.L. Information System for the Preparation of Requests for Scientific
Foundations. Fundamental Research 5(2), 289-293 (2016) [13] Markina G.L., Shley M.D. Organization of decision-making in the scientific and educational sphere using the example of a project management system. Computer tools in education, 4, 19-31 (2016) [14] Zhang Nan. Wan Shanshan, Wang Peiling. A bibliometric analysis of highly cited papers in the field of Economics and Business based on the Essential Science Indicators database; with co-authors. SCIENTOMETRICS, vol. 116, issue 2, 1039-1053. (2018) [15] Abramo Giovanni. Revisiting the scientometric conceptualization of impact and its measurement. JOURNAL OF INFORMETRICS, vol. 12, Issue 3, 590-597 [16] Bornmann Lutz, Marx Werner. Critical rationalism and the search for standard (fieldnormalized) indicators in bibliometrics. JOURNAL OF INFORMETRICS, vol. 12, Issue 3, 598-604 (2018) [17] Curceac Stelian, Ternynck Camille, Ouarda Taha B.M.J . Short-term air temperature forecasting using Nonparametric Functional Data Analysis and SARMA models. ENVIRONMENTAL MODELING & SOFTWARE, vol. 111, 394-408 (2019) [18] Fouilloy Alexis, Voyant Cyril, Notton Gilles. Solar irradiation prediction with machine learning: Forecasting models selection method depending on weather variability. ENERGY, vol. 165, 620-629 (2018) [19] Wang Zhen, Hu Qinghua, Zhong Qiuzhen. Linear Multistep F10.7 Forecasting Based on Task Correlation and Heteroscedasticity Author. EARTH AND SPACE SCIENCE, vol. 5, Issue 12, 863-874 (2018)