This article describes a new bibliometric indicator called Impact Factor of a Term (IFT) that helps to predict future impact of scientific works and/or the author. The predictive properties of IFT are proven by two examples of different collections of scientific articles. It is shown that the correlations of the current and future IFT values depending on the trend are practically similar for both collections. The graphs of IFT correlations of the current and future years depending on the number of articles with the word/term are presented. The graphs show that the higher the current frequency of the term and the number of articles with this term, the greater the correlation and stability of IFT. The stability of IFT helps to accurately predict the number of future citations. The list of the most informative words/terms with the largest total values of IFT multiplied by the current frequency is analyzed. It has been shown that the size of collection affects the stability and predictive properties of IFT. The words and terms with high IFT values allow us to judge the future impact of an article and its author based on the prediction of future citations. Such words also help identify promising research directions.
Currently, the influence of the scientist's activities is assessed using metric indicators. Typical indicators are the number of articles written by the author and the total num The reported study was funded by RFBR according to the research projects № 19-07-00857, 18-07-00909 and 20-04-60185. ber of citations received by these publications. Not all publications have equal prestige; it is largely determined by the place of publication of the article.
JIF (Journal Impact Factor), originally designed to help libraries make decisions
about indexing and replenishing their journal collections (Garfield, 2006) [
Nevertheless, a significant increase in the number of publications and the
hypercompetitiveness of scientists over the past decades has led researchers to rely in part
on JIF as an indirect indicator for ranking not only journals, but also articles published
in them (Casadevall & Fang, 2014 [
Author Impact Factor (AIF) [
In the literature there are several indicators for assessing the impact of individual
scientists. The h-index [
The h-index is widely used to quantify the research results of an individual
scientist (Hirsch, 2005 [
This article describes a new bibliometric indicator called Impact Factor of Terms (IFT) that is similar to the impact factors of the author and journal. The empirical predictive properties of IFT are analyzed using two different collections of scientific articles as an example.
The IFT concept combines the advantages of simplicity of JIF calculating with the ability to evaluate the impact of researchers. At the same time, the concept of IFT includes the possibility of forecasting, in contrast to JIF, AIF and h-index. 2
In this study, we compared the predictive properties of the impact factor of the term using two different collections - DBLP and PubMed.
In our previous experiments, we analyzed the DBLP citation network, which is a collection of articles on artificial intelligence from 1936 to 2017, compiled by
Impact factor of a term: a tool for assessing article's future citations and author's influence… 3 aminer.org and called the DBLP collection here. This dataset consists of 3,079,007 articles and 25,166,994 references.
The PubMed collection is much bigger. Over 30 million links from Medline,
science journals and online books have been included in PubMed in the field of
biomedicine [
For this study, we investigated the statistical properties of the terms included in the
headings of the articles in the Medline / PubMed database for all years. It is believed
that the bibliographic data of the article (title, abstract, keywords, etc.) contain a fairly
detailed picture of the subject of the article [
To extract keywords from headings, we used the Medline / PubMed core database for all years. The annual base level is published in December of each year.
Statistical analysis of the collections was carried out using the “Trend +” authors’ program, which built a frequency dictionary of all words and terms in the collections. In addition, for each term with a frequency of more than 5, “Trend +” calculates its own trend indicators (trending situations) including the number of articles with this term per year, the number of links to articles with this term, IFT, indicators for each year.
Below we describe the main stages of the process. 2.1
Before building machine learning models, the text corpus needs to be prepared. Text preprocessing is needed to increase the speed of text processing and reduce the memory spent on texts and the dictionary. Reducing the length of the dictionary leads to a decrease in the total number of internal parameters of the models. Text preprocessing consists of several sequential stages: a) converting all letters to lowercases; b) exclusion of stop words - removal of the most common words in the text, such as pronouns, prepositions, conjunctions, numbers, etc.; c) exclusion of rare words - deleting words that are rarely found in the text; d) text normalization; e) lemmatization of the text (sometimes stemming is used instead of lemmatization); f) additionally, we can connect tokens “no”, “not” with the next token to form one token instead of two. 2.2
The corpus of scientific publications contains the titles of articles, authors and citation links. On the basis of the corpus, a dataset was created for experiments on predicting the citation of an article. The problem of classification was solved: will any given article be cited by any other authors within the next three years (class 1) or not (class 0).
The goal of the experiment was to test how well the SVM machine learning algorithm can predict article citations (distinguish class 1 from class 0). The algorithm was trained on class labels and corresponding publication titles. After preprocessing the titles were converted into vectors using a TF-iDF vectorizer. After training, the algorithm was tested on a lazy sample consisting of 10% of the dataset. The result of testing the algorithm according to the ROC AUC metric was 0.62. We got this a pretty good result using only article titles. After adding the author, the metric was increased to 0.65. 2.3
The experiment was to test how well a deep recurrent neural network with an
attention mechanism [
Variable parameters: n - number of recurrent layers - from one to four (with a different number of neurons), m - number of forward propagation layers - from 1 to 3x with a different number of neurons).
The neural network (Figure 1) was trained on class labels and corresponding titles of publications. To get features from the text, the titles of the articles first went through all stages of preprocessing, and then they were converted into vectors using Word2Vec neural model. After training, the algorithm was tested on a lazy sample consisting of 10% of the dataset. The obtained result for the best neural network architecture was 0.67 according to the ROC AUC metric.
For each word/term, the Trend+ program calculates a number of different statistical indicators in dynamics for different years (trending situations). Further, to predict the future values of these indicators, algorithms of correlation analysis, linear regression and neural network algorithms are used.
Below we describe the results of correlation analysis for a subset of promising words and terms. Calculation of special indicators like IFT expands forecasting capabilities of the Trend+ program. 2.4
It should be noted that Trend+ is capable of processing large amounts of data to collect the specified statistics. For example, in this study, the size of PubMed data in XML format is 248GB.
It is also important that Trend+ is capable of performing a large amount of computation within a few hours. For example, for each word/term W out of 307,000 terms with a frequency of over 100 in PubMed, the following complex computations are performed to calculate the IFT indicator:
- for each year Y, a set A (Y, W) of articles containing the given word W is determined;
- for each set A (Y, W), all references and a set of articles B containing W and quoted from A (Y, W) are determined;
- for each article B, its publication year is compared with Y.
Impact factor of a term: a tool for assessing article's future citations and author's influence… 5 In this paper, we proposed a new numerical indicator of the significance of words and terms in scientific publications, called the impact factor of the term (IFT). The term impact factor is calculated similarly to such well-known numerical indicators as the impact factor of the journal and the impact factor of the author. The impact factor of a term is a numerical indicator of the significance of this term, calculated by the formula (1):
IFT = At
Nt where Аt is the number of citations in articles with the term A published in year t to articles with the term A in the period Δt years to year t; Nt – total number of articles with term A for the time period ∆t + 1.
Just as a journal with a high JIF is considered significant and authoritative, if articles with a certain term are cited high, then this term has a high IFT value and is considered significant and informative. (1)
We understand the significance of a word/term as the ability to reflect important fundamental ideas. For such significant words, IFT values are consistently high. Terms with consistently high IFT indicate important ideas that have been stable in interest for many years.
The information content of the term within the framework of the IFT concept is the property of the term to indicate the "relatedness" of publications that include the term. According to the definition of IFT, if two articles have the same terms with a high IFT, then there is a high probability of a formal bibliographic reference between them.
The PubMed collection has been analyzed in different directions. Of the entire collection of PubMed articles, 179,315 terms were identified that appeared more than 100 times in article headings. The terms were ranked by time of appearance and by the number of appearances in the titles of publications, as well as by total values of IFT multiplied by the current frequency.
In this paper, we compare the results obtained by statistical analysis taking into account the IFT of large document collections of different subject areas. 3 3.1
Impact factor of a term: a tool for assessing article's future citations and author's influence… 7
The main purpose of the statistical analysis of the collections is to study the empirical properties of the Impact Factor Terms (IFT), including the correlation of its current and future values to assess its stability and predict future citations.
The difference between the DBLP and PubMed collections is in the frequency ranges of the terms, since the first collection includes more than 3 million articles, and the second more than 30 million. Therefore, for PubMed, terms with a frequency of 100 or more were considered, and for DBLP, the range was 5 or more.
Figure 2 shows that for completely different collections, the graphs of the correlations of the current and future IFT values almost coincide. The graphs show that the higher the current frequency of the term in any subject area (the number of articles with this term), the greater the correlation and, therefore, the more stable the IFT behaves in time. Stable IFT allows you to accurately predict the average number of citations in the future, since the IFT is exactly equal to the average number of citations of articles with the specified term. The most consistent and predictable terms with high IFT values are significant terms. Significant terms, therefore, have high frequencies and IFT values above a certain threshold.
Therefore, based on the analysis of huge collections of documents, we can conclude that the presence of significant terms in the articles allows a high degree of certainty to judge the future influence of the article on the basis of the prediction of its future citation. This thesis also applies to authors. The more significant terms in the works of this author, the more likely it will be quoted in the coming years.
To calculate the correlation, situations / points were chosen for different words in different years, when the IFT values of the current year were more than zero. There may be several such situations for one word in different years. The selected situations were divided into groups that differ in the number of articles with a word over the past 3 years. Figure 3 shows graphs of the number of situations / points in these groups for calculating correlations. The figure shows that for terms from the PubMed collection there are more points / situations for large trends. The reason is that the size of the PubMed collection is an order of magnitude larger than the DBLP collection, so it has more terms with high frequency. In fig. 2, the Y axis represents the number of points for calculating the correlation of the current and future years. The X axis represents the frequency of terms, i.e. the number of articles with a term over the last 3 years. On the graph for the DBLP collection, the maximum number of points is 54326 at X = 6, and the minimum is 825 at X = 97. On the graph for PubMed, the maximum number of points is 32908 at X = 15, and the minimum is 4603 at X = 100.
Impact factor of a term: a tool for assessing article's future citations and author's influence… 9
The graphs show that the larger the collection (the larger its volume), the higher the correlation and, therefore, the more stable the IFT behaves in time. Figure 3 shows that in 2000, correlations were lower for almost all trends. Consequently, the size of the collection affects the stability of IFT, which improves the accuracy of predicting future citations. 3.3
Among all the terms from the PubMed collection that occur more than 100 times, the most informative words / terms that have the largest total values of IFT multiplied by the current frequency were highlighted.
Of the 179 thousand terms ordered by informational rating, we selected the first thousand terms to analyze their topics.
Surprisingly, the most informative term is “arabidopsis thaliana”. Arabidopsis thaliana is a plant that is a model for studying the processes of plant growth and development. This is a small plant that has a short generation time and grows well in laboratory conditions.
The remaining terms (excluding Arabidopsis, thaliana) we divided into the following groups: diseases (158); enzymes and biomolecules (53); medical genetics (56); methods (84); cell structure (102); body structure, tissues, organs (98); drugs and substances (72); viruses and bacteria (117); other medical terms (119); other general scientific terms (137).
9,73 11,35 7,57 13,24 7,16
21,35 13,78 15,81 diseases viruses and bacteria cell structure body structure, tissues, organs methods
The largest group (excluding commonly used and general medical terms) was the group "Diseases". This group includes 158 terms, or 21.35% of the total number of terms, excluding commonly used and general medical terms. Top 10 include: Parkinson's Disease, Hepatitis B, Gastric Cancer, Human Immunodeficiency, Osteoarthritis, Pancreatic Cancer, Cystic Fibrosis, Kidney Disease, 1 Diabetes, Epstein-Barr.
In second place is the group “Viruses and bacteria” - 15.81% of the total number of terms. Top 10 include: HIV-1, Immunodeficiency Virus, Staphylococcus Aureus, Pseudomonas, C Virus, Helicobacter Pylori, Zika, Clostridium, Salmonella, Plasmodium.
In third place is a group of terms describing the structure of the cell and its components - 13.78%. Examples of terms: Autophagy, Mesenchymal, Macrophages, Dendritic Cells, T-Cell, Mitochondria, Endoplasmic Reticulum, Endothelial Cells.
In fourth place is the group “Structure of the organism, tissues, organs” (13.24%). The following are the “Methods” groups (11.35%), “Medicines and Substances” (9.73%), “Medical Genetics” (7.57%), and “Enzymes and Biomolecules” (7.16%). 4
This article describes the impact factor of the term, which is similar to such bibliometric indicators as journal impact factor (JIF) and author impact factor. Unlike JIF and H-index, the concept of the impact factor of the term includes the ability to predict the impact of scientific papers and / or the author. The empirical predictive properties of IFT were analyzed on the example of two collections of scientific articles that radically differ in volume and topic.
For our experiments we used “Trend+” authors’ program and Medline/PubMed database for all years. The developed IFT-based statistical search strategy extracts the most informative terms from all available headings of biomedical publications. For that purpose 179,315 terms found in article headings in PubMed with frequency over 100 were ranked by total values of IFT multiplied by the current frequency.
It is shown that the correlations of the current and future IFT values depending on the trend are practically similar for both collections (DBLP and PubMed). The higher the current frequency of the term (the number of articles with this term), the more stable the IFT behaves in time. The stability of IFT of a certain term allows you to accurately predict the number of future citations of articles with this term. However, the size of the collection affects the stability of the IFT, the larger the collection, the more stable the IFT.
The presence of significant terms in articles (terms with high IFT values) allows us to judge the future influence of the article and its author based on the prediction of future citations.
Analysis of the first thousand informative terms from the PubMed collection showed that terms related to the names of diseases prevail in it, followed by terms related to viral and bacterial pathogens. Consequently, authors may include specific terms in the titles of their articles to increase the likelihood of reading and citing.
The proposed methodology allows identifying the most significant terms and predicting the probability of citing articles with these terms. The proposed metric can be used to gauge how much researchers are at the forefront of the research trend. 5
The reported study was funded by RFBR according to the research projects № 19-0700857, 18-07-00909 and 20-04-60185. We are grateful to the Russian Foundation for Basic Research for financial support of our projects.