Evaluating the Efectiveness of Research Grants with Journal Bibliometrics

Abstract

We present and exploit WEIG - Wroclaw Efectiveness Indicator for Grants, a scientometric meta-measure that aggregates quality of scientific papers published within a given project normalized with its budget. studies on grants ended between 2014 and 2021 revealed that NSC projects are more efective than ERC ones. The eficiency of Science Branches and Scientific Panels over years are investigated. Some limitations of the proposed approach and observed phenomena are discussed as well.

Keywords

• research eficiency
• grant efectiveness
• funding acknowledgement analysis
• WEIG
• IF
• AIS
• Polish Minis-

-

CEUR ceur-ws.org

1. Introduction

Achievements of research activities or results like grants or scientific papers can be evaluated and compared by means of quantitative bibliometric measures. However, there is no commonly agreed measure for quality of scientific outcome or efectiveness of research grants. Anyway, we still expect a simple and possibly interpretable metric to compare research centres, journals, conferences, or funding schemas. impact factor (IF) appears to fulfill such a function for scientific journals, and h-index for individual scientists. On the other hand, we should be aware that quantitative scientometric applied to evaluaiton of individual scientists, e.g. for the promotion purposes, still remains questionable. This issue was addressed in San Francisco Declaration on Research Assessment [ 1 ] and Leiden Manifesto [ 2 ].

Supporting fundamental research by means of public funds is considered to have positive impact on the country’s economy [ 3 ]. Therefore, public financing of basic studies is necessary in the recent world and “no nation can ‘free-ride’ on the world scientific system” . This paper concentrates on fundamental research, for which the primary expected outcome are scientific LGOBE (M. Dzieżyc) CEUR Workshop Proceedings papers.

To support economical progress and competitiveness of their economy, most developed countries maintain their own dedicated public agencies to boost basic scientific studies. It includes NSF and NIH in USA, EPSRC in the UK, DFG in Germany, ANR in France, NSFC in China, NRF in Korea, JSPS and JST in Japan, as well as some international initiatives like ERC (European Research Centre), which is a part of the European Union’s flagship Research and Innovation programme (Horizon Europe), component of the EU-long-term Multiannual Financial Framework. These public bodies support hundreds and thousands of grants, so they want to monitor the project implementation and validate their results. Therefore, we present a measure that compare the quantity of scientific papers and efectiveness of the projects in a massive manner.

For that purpose Wroclaw Efectiveness Indicator for Grants (WEIG) – a meta-measure was initially proposed in [ 4 ] along with some preliminary analyses of Polish National Science Centre (NSC) grants. Next, it was further developed and applied to compare grants funded by European Research Centre (ERC) and NSC [ 5 ], which ended by 2020. Here, we present some other analyses related to efectiveness of research grants, i.e., we consider an additional journal metric – Polish Ministerial Scoring of Journals (PMSJ), year 2021 is appended, and publications without IF are investigated.

2. Related Work

2.1. Funding Acknowledgement Analysis

The Web of Science (recently by Clarivate Analytics) added two features to the indexed scientific papers in 2008: the funding agency and grant number. These fields were analyzed already in 2011, showing also an important role of non-govermental agencies in Germany and Japan [ 6 ]. Then, it was called ”funding acknowledgement analysis”.

John Rigby showed a weak correlation between the number of declared funding sources and the impact of the publications measured by the number of received citations [ 7 ]. This observation was based on 3,596 articles from the Journal of Biological Chemistry published in 2009. He also found out that Web of Science has a clear advantage over the MEDLINE (PubMed) database when it comes to funding sources. MEDLINE covers only a few sources of funding, mainly from the USA [ 8 ]. He also suggested that the funding text can be used for mapping research outputs and priorities of funding bodies.

In 2017, Nicola Grassano et al. evaluated recall and precision for funding information provided by Web of Science and PubMed. Their study was based on manually annotated data about funding sources from 7,510 paper’s full-texts related to UK cancer research in 2011 [ 9 ]. They received a high recall value for WoS - 93% and 94% for precision. In contrast, recall for PubMed was 42% and precision - 96%. Also, based on the e-mail survey sent to the authors of the articles, they concluded that only 3% of papers did not reveal all funding sources.

Belén Álvarez-Bornstein et al. published a paper with the estimation of the completeness and the accuracy of Web of Science funding data [ 10 ]. In 87.8% of the articles, which included funding text data, both funding agency and grant number fields were correctly extracted. However, as they pointed out, there are problems with the precision of such data: (1) diferent funding bodies can be recognised as one body, (2) there is a lack of the country for the funding body country, and (3) diferent grants from the same funding body can be recognised as one grant.

In 2020, Liu at al. reported improvement in funding information in Web of Science (WoS) [ 11 ]. Authors suggested that the funding acknowledgement data in WoS should be primarily used for the analysis of papers published in English. Also, a case study published in the same year revealed that funding information in WoS is more reliable than in Scopus [ 12 ].

In 2021, Álvarez-Bornstein and Montesi found in their literature review on analysed funding acknowledgements a lack of data normalisation, consistency, and completeness, especially for Pubmed and Scopus databases [ 13 ]. Although Web of Science is thought to possess the best funding acknowledgements data, it still has many problems related to papers in Social Sciences and Humanities.

2.2. Research Assessment

The Declaration on Research Assessment (DORA) was announced at the Annual Meeting of the American Society for Cell Biology in San Francisco in 2012 [ 1 ]. It contains the guidelines for funding agencies, institutions, publishers, organisations that supply metrics and researchers: 1. Be open and transparent by providing data and methods used to calculate all metrics. 2. Provide the data under a licence that allows unrestricted reuse, and provide computational access to data, where possible. 3. Be clear that inappropriate manipulation of metrics will not be tolerated; be explicit about what constitutes inappropriate manipulation and what measures will be taken to combat this. 4. Account for the variation in article types (e.g., reviews versus research articles), and in diferent subject areas when metrics are used, aggregated, or compared.

Polish National Science Centre (NSC) signed DORA in 2018 [ 14 ] and ERC in 2021 [ 15 ]. In 2015, The Leiden Manifesto for research metrics was announced in Nature [ 2 ].

It covers ten principles for research evaluation to avoid abuse of bibliometrics.

2.3. PBRF: Performance-based Research Funding

According to [ 16, 17 ], research funding can be treated as performance-based if: (1) is evaluated ex-post; (2) output or impact of scientific studies are estimated, and the funding or its part is based on such assessment; (3) funding is implemented either at national or regional level.

Implementation of the Performance-based research funding in Australia resulted in the greater number of scientific papers with simultaneous drop in quality of research [ 18 ]. Next, this finding was refuted in 2017 [ 19 ].

A positive efect of PBRF application on the research quantity was concluded from studies on the data from 31 countries and the 1996-2016 period [ 20 ]. The author relied primarly on bibliometric evaluation. Zacharewicz at al. discovered in [ 21 ] that 12 out of 28 EU countries exploited no PBRF, 3 had limited PBRF, 11 applied a quantitative, bibliometric assessment and 5 countries used peer review process.

3. Scientometric Measures for Journals and Grants

3.1. Journal Impact Factor, Derivatives, and Similar Measures

Impact Factor (IF) is a measure for scietific journal as the average number of citations normalized with the number of articles published in recent years. For a given year , its is the number of citations received in year to papers published in two preceding years ( − 1 and − 2 ), divided by the number of citable papers published in years − 1 and − 2 : = −1 + −1 + −2 −2 (1)

Impact Factor is even criticised as a measure of journal quality. IF does not respect the quality of citations, includes unlimited self-citations, is incomparable between domains and concentrates on English journals. Moreover, a simple method of calculation incentives publishers to manipulate this factor by promoting self-citations, preferential for reviews and the decreasing number of Citable Items [ 22, 23 ]. It is also criticised for being unscientific [ 24 ].

The Average Journal Impact Factor Percentile, here refered as IF%, was introduced in 2016.

It is a non-linear transformation of IF, which takes percentile positions of journal’s IF in each scientific category and averages it.

IF% was shown to have a smaller variation than IF and the distribution close to normal [ 25 ].

Article Influence Score (AIS) is defined as eigenfactor (the idea similar to Google Page Rank) divided by the number of citable articles. AIS is comparable to 5-year IF [ 26 ].

3.2. Polish Ministerial Scoring of Journals (PMSJ)

Universities and public scientific bodies in Poland are eveluated by the Ministry of Education and Science separately in 47 scientific disciplines that are grouped into branches of science: humanities (7 disciplines), engineering and technology (9), medical and health sciences (4), agricultural sciences (5), social sciences (11), natural sciences (7), theology (1), the arts (3) [ 27 ]. For the evaluation purpose, scientific entities report among others the papers published by the author afiliated at their organisation. Polish Ministerial Scoring of Journals (PMSJ) is assigned to each paper according to the list announced by the Minister of Education and Science [ 28 ]. Each journal is assigned one of the score: 200 (best), 140, 100, 70, 40, 20 (worst) as well as the list of disciplines a given journal is relevant to. In principle, the scoring is based on IF of the journal, however, it is not the absolute rule. Since PMSJ is not a linear transformation of IF, we cannot claim that sum of IF of two journals with PMSJ=100 is close to IF of one journal with PMSJ=200. To some extent, PMSJ may be considered similar to IF% but with less granularity and some manual corrections.

Results of the evaluation for years 2017-2021 are published in [ 29 ], however, the entire process has not been finished yet due to some appeals against this decision.

3.3. WEIG - Wroclaw Efectiveness Indicator for Grants

Wroclaw Efectiveness Indicator for Grants – () for project is a meta-measure defined as a calculable outcome () of divided by ’s funding () , here expressed in millions of euros:

Assuming the project has a set of related scientific articles , which are the efect of a given project , and which acknowledge . We can use a bibliometric quality measure () for each such paper . By aggregating () for all papers resulting from project , we obtain the general output quality of : () = ∑∈ () . Then, () is:

In this study, as () we analyzed three measures: IF, AIS (Sec. 3.1), and PMSJ (Sec. 3.2), however, also any other measure can be considered like IF% or the number of ’a citations. As a result, we tested WEIG-IF, WEIG-AIS, and WEIG-PMSJ, respectively.

WEIG can also be generalised for the set of projects , i.e., for all grants from a given scientific discipline (panel) or year: ( ) = ( ) ( ) = ∑∈ () ∑∈ () = ∑∈

∑∈ () ∑∈ () where ( ) are aggregated outcomes (sum) of all projects ∈ , and ( ) - their total funding, e.g. in millions of euros.

WEIG was initially proposed in [ 4 ] along with some preliminary analyses of NSC grants and then in-depth investigated for both considered agencies in [ 5 ]. Here, we extend it with additional analyses, a new year – 2021 and a new measure PMSJ. (2) (3) (4)

4. Efectiveness of Research Grants Funded by ERC and NSC

4.1. Research Setup

A detailed description of all processing steps can be found in [ 5 ]. Overall, the process consists of: (1) collecting the data about scientific grants funded by ERC and NSC from their online databases, (2) using grant ids, the outcome papers are identified from Web of Science web service, (3) for each such paper published in the journal, a bibliometric measure is gathered from Journal Citation Report (IF, AIS) or from the online list published by Polish Ministry of Education [ 28 ]. Having bibliometric measures (IF, AIS, PMSJ) for each paper assigned to each grant, appropriate WEIG values are computed, including relative ones, Sec. 3.3. Aggregated WEIG values are calculated for scientific panel, years, and funding schemas.

The project year was defined as the year in which a given grant had ended. We considered all project’s identified publications registered by the data collection date (April 2022) while calculating the WEIG values. We believe that a reliable comparison between agencies or panels should be carried out only between grants completed in the same year.

Year 2014 was treated as a reliable starting point for our studies, since there were relatively few grants ended before this year for both agencies. Their funding schemas (many finished projects) yielded first mature results as late as 2014.

4.2. Efectiveness of Science Branches

Both financing agencies distinguish three separate science branches (see [ 5 ]), which correlate: • Social Sciences and Humanities (SHERC=HSNSC), • Physical Sciences and Engineering (PEERC=STNSC), • Life Sciences (LSERC=NZNSC).

The WEIG values for science branches are relatively stable regardless of the measure taken (see Fig. 1). However, we can observe a general small drop over time for Life Sciences (LS) both for NSC and ERC projects. A sudden drop is characteristic of almost all branches and it is related to the COVID-19 pandemic – loss of efectiveness, which started in 2020, can be explained by the requested projects extensions. A slight increase for Physical Sciences and Engineering projects funded by NSC may be related to growing eficiency of Production and Processes Engineering and Astronomy and Space Science grants. Interestingly, there is one significant diference between WEIG-PMSJ and other metrics: Social Sciences and Humanities (SS&H) efectiveness does not difer from other branches as significantly as in the case of WEIG-IF or WEIG-AIS. On the other hand, a very large part of SS&H scientific papers is published in periodics with no IF (see Fig. 3). Overall IF may not be an adequately good metric to quantify output in this science branch. WEIG-PMSJ may be complementary to other presented metrics.

4.3. Efectiveness of Scientific Panels and Funding Schemes

The values of WEIG-IF, WEIG-AIS, WEIG-PMSJ, and funding for the projects, which ended between 2014 and 2021, were presented for Physical Sciences and Engineering (PE) as well as Life Sciences (LS) in division into individual panels, Fig. 2. Social Sciences and Humanities (SH) were disregarded as there is no correspondence between most of NSC and ERC panels for HS (see [ 5 ]). There is a similarity in the efectiveness of the respective NSC and ERC panels. NSC grants are overall more efective than ERC ones. They are also characterized by significantly smaller funding. The Universe Sciences (PE9) both for NSC and ERC provide the highest overall efectiveness. Simultaneously, Mathematics (PE1) reveals a big diference between WEIG-IF and WEIG calculations based on diferent measures. Computer Science and Informatics and Systems (CS&IS) and Communication Engineering (CE) grants both for NSC and ERC appear to be less efective in comparison to other Physical Sciences and Engineering panels. It may be connected with the specifity of scientific activity. A significant part of the articles for CS&IS and CE have been published in the journals with no IF (see Fig. 3). In fact, they are very often conference proceedings.

WEIG measure can indicate some important diferences between various types of projects. Undoubtedly, larger projects allocate relatively more resources to infrastructure development etc. in comparison with smaller projects. This translates into efectiveness of research grants measured with journal bibliometrics. The results of the comparison of the efectiveness of the small projects for young scientists and larger projects for advanced researchers are significant in this matter (see Fig. 4). Both PRELUDIUM and Starting Grant (AdG) are more efective than bigger projects (OPUS and Advanced Grant (AdG) respectively) in terms of WEIG measure. However, smaller projects have much greater restrictions on financing the purchase or manufacture of scientific and research equipment.

5. Discussion: Other WEIG Variations

We are aware of the limitations and problems associated with the use of journal metrics, e.g. the IF measure [ 30 ]. First of all, journal impact factor could not be simply translated into individual achievements, e.g. actual citations of individual articles. Second, IF is afected by various factors such as research field (e.g. favouring the research fields with literature that rapidly becomes obsolete) or journal and article type (e.g. review articles are heavily cited). Some limitations are related to the database, e.g. its coverage. It is necessary therefore to take into consideration the wider context while analysing and interpreting the results. It is dificult or even impossible to include all of the factors in one measure (e.g. diferent grant results, the characteristics of the discipline etc.). However, some of them have such potential (e.g. exchange rate fluctuations, costs of living etc.). Simple experiments like adding cost of living index (CLI) to the formula may be the way to introduce the relevant determinants into the WEIG measure: () () = (5)

() ⋅

In this case, one would have to consider e.g. that the CLI varies depending on the place of living and the time of the estimation. Referring to the example in the Table 1, after assuming adequate CLI for UK and Poland, the gap between the presented projects would decrease: WEIG-IF (CLI) for grant 2013/09/B/ST6/02317 would be 12490.768 instead of 327.841, WEIG-IF (CLI) for grant 278212 would be 7836.285 instead of 112.752. Thus, the NSC-funded grant would have a score 159.40% better than the ERC grant, instead of 290.76%. Such experiments could nevertheless be a step toward improving the WEIG measure.

6. Conclusions and Future Work

There are three main findings derived from our analyses of the WEIG values for grants funded by two public agencies European Research Council (ERC) and Polish National Science Centre (NSC): 1. Polish NSC grants are commonly more efective than those supported by European Union via ERC (two times for physical and engineering as well as humanities, ca. 50% for life sciences). The diference is particularly noticeable when considering WEIG-PMSJ: three times greater; 2. WEIG calculations using diferent measures provide similar results, i.e., they reveal the same phenomena. However, they are complementary in some aspects; 3. There is a similarity in the efectiveness of the respective NSC and ERC panels for Physical Sciences and Engineering (PE) and Life Sciences (LS). It is connected with the specifity of scientific activity including publishing in conference proceedings.

In many cases, it is not possible to analyze all the data in detail. WEIG, in combination with other project information (such as scientific discipline, subject matter, implementing entity, project size), can be used to filter information on ongoing grants. This can apply to the searching for scientists to collaborate with, identification of promissing research topics, or policy-making in the field of scientific research and technological development. Thus, it could be useful in tackling information overload in various contexts.

Future work will focus on further development and analysis of derived WEIG measures, e.g. staring from the one suggested in Sec. 5. Along with the WEIG-based measures already tested in this paper, they would provide a more copmprehensive insights into research projects and can be used to minimize the impact of some selected factors like local salary level.

Acknowledgments

The authors would like to thank Web of Science Group, Clarivate Analytics for giving consent for usage and aggregation of data acquired from their databases. This work was financed by (1) the National Science Centre, Poland, project no. 2021/41/B/ST6/04471; (2) the Polish Ministry of Education and Science, CLARIN-PL; (3) the European Regional Development Fund as a part of the 2014-2020 Smart Growth Operational Programme, CLARIN – Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19; (4) the statutory funds of the Department of Artificial Intelligence, Wroclaw University of Science and Technology; (5) the Polish Ministry of Education and Science within the programme “International Projects Co-Funded”; (6) the European Union under the Horizon Europe, grant no. 101086321 (OMINO). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor European Research Executive Agency can be held responsible for them.

References

[1] T. A. S. for Cell Biology, Read the declaration - dora, 2012 . URL: https://sfdora.org/read/.

[2] H. Diana , W. Paul, W. Ludo, de Rijcke Sarah, R. Ismael , Bibliometrics: The leiden manifesto for research metrics , Nature 520 ( 2015 ) 429 -- 431 . doi: 10 .1038/520429a.

[3] A. J. Salter , B. R. Martin , The economic benefits of publicly funded basic research: a critical review , Research Policy 30 ( 2001 ) 509 - 532 .

[4] M. Dziezyc , P. Kazienko , Jaka jest efektywność projektów badawczych , Forum Akademickie 07-08 ( 2018 ) 0165551520962775 . URL: https://prenumeruj.forumakademickie.pl/fa/2018/07- 08/jaka-jest -efektywnosc-projektow-badawczych/.

[5] M. Dzieżyc , P. Kazienko , Efectiveness of research grants funded by european research council and polish national science centre , Journal of Informetrics 16 ( 2022 ) 101243 . doi:https://doi.org/10.1016/j.joi. 2021 . 101243 .

[6] J. Wang , P. Shapira , Funding acknowledgement analysis: an enhanced tool to investigate research sponsorship impacts: the case of nanotechnology , Scientometrics 87 ( 2011 ) 563 - 586 . doi: 10 .1007/s11192-011-0362-5.

[7] J. Rigby , Looking for the impact of peer review: does count of funding acknowledgements really predict research impact? , Scientometrics 94 ( 2013 ) 57 - 73 .

[8] PubMed, Grant number information found in the gr field in medline/pubmed, 2021 . URL: https://www.nlm.nih.gov/bsd/grant_acronym.html.

[9] N. Grassano , D. Rotolo , J. Hutton , F. Lang , M. M. Hopkins , Funding data from publication acknowledgments: Coverage, uses, and limitations , Journal of the Association for Information Science and Technology 68 ( 2017 ) 999 - 1017 . doi: 10 .1002/asi.23737.

[10] B. Álvarez-Bornstein , F. Morillo , M. Bordons , Funding acknowledgments in the web of science: completeness and accuracy of collected data , Scientometrics 112 ( 2017 ) 1793 - 1812 .

[11] W. Liu , L. Tang , G. Hu, Funding information in web of science: An updated overview , Scientometrics 122 ( 2020 ) 1509 - 1524 .

[12] W. Liu , Accuracy of funding information in scopus: a comparative case study , Scientometrics 124 ( 2020 ) 803 - 811 .

[13] B. Álvarez Bornstein , M. Montesi , Funding acknowledgements in scientific publications: A literature review , Research Evaluation ( 2021 ). doi: 10 .1093/reseval/rvaa038, rvaa038 .

[14] DoraNsc , National science centre signs the dora declaration , 2018 . URL: https:// www.ncn.gov.pl/aktualnosci/2018-10-23 -ncn-sygnatariuszem-dora?language=en.

[15] DoraErc , Erc plans for 2022 announced, 2021 . URL: https://erc.europa.eu/news/erc-2022 - work-programme.

[16] D. Hicks , Performance-based university research funding systems, Research Policy 41 ( 2012 ) 251 - 261 .

[17] K. Jonkers , T. Zacharewicz , et al., Research performance based funding systems: A comparative assessment, Publications Ofice of the European Union , Luxembourg ( 2016 ).

[18] L. Butler , Explaining australia's increased share of isi publications-the efects of a funding formula based on publication counts , Research Policy 32 ( 2003 ) 143 - 155 .

[19] P. van den Besselaar, U. Heyman, U. Sandström, Perverse efects of output-based research funding? butler's australian case revisited , Journal of Informetrics 11 ( 2017 ) 905 - 918 .

[20] D. Checchi , M. Malgarini , S. Sarlo, Do performance-based research funding systems afect research production and impact? , Higher Education Quarterly 73 ( 2019 ) 45 - 69 .

[21] T. Zacharewicz , B. Lepori , E. Reale , K. Jonkers , Performance-based research funding in EU Member States-a comparative assessment , Science and Public Policy 46 ( 2018 ) 105 - 115 .

[22] M. E. Falagas , V. D. Kouranos , R. Arencibia-Jorge , D. E. Karageorgopoulos , Comparison of scimago journal rank indicator with journal impact factor , The FASEB Journal 22 ( 2008 ) 2623 - 2628 . doi: 10 .1096/fj.08- 107938 , pMID: 18408168 .

[23] S. Ramin , A. S. Shirazi , Comparison between impact factor, scimago journal rank indicator and eigenfactor score of nuclear medicine journals , Nuclear Medicine Review 15 ( 2012 ) 132 - 136 .

[24] K. Moustafa , The disaster of the impact factor , Science and Engineering Ethics 21 ( 2015 ) 139 - 142 . doi: 10 .1007/s11948-014-9517-0.

[25] L. Yu , H. Yu , Does the average jif percentile make a diference? , Scientometrics 109 ( 2016 ) 1979 - 1987 . doi: 10 .1007/s11192-016-2156-2.

[26] C. Bergstrom , Eigenfactor: Measuring the value and prestige of scholarly journals , College & Research Libraries News 68 ( 2007 ) 314 - 316 . doi: 10 .5860/crln.68.5.7804.

[27] Rozporządzenie ministra edukacji i nauki z dnia 11 października 2022 r. w sprawie dziedzin nauki i dyscyplin naukowych oraz dyscyplin artystycznych , 2022 . URL: https: //isap.sejm.gov.pl/isap.nsf/DocDetails.xsp?id= WDU20220002202 .

[28] Nowy, rozszerzony wykaz czasopism naukowych i recenzowanych materiałów z konferencji międzynarodowych , 2021 . URL: https://www.gov.pl/web/edukacja -i-nauka/nowyrozszerzony-wykaz-czasopism-naukowych-i-recenzowanych-materialow-z-konferencjimiedzynarodowych.

[29] Wyniki ewaluacji działalności naukowej za lata 2017- 2021 , 2022 . URL: https://www.gov.pl/ web/edukacja -i-nauka/wyniki-ewaluacji-dzialalnosci-naukowej-za- lata- 2017- 2021 .

[30] P. O. Seglen , Why the impact factor of journals should not be used for evaluating research , Bmj 314 ( 1997 ) 497 .