Nowadays the qualitative characteristics of a scientific document are becoming more and more relevant because a large number of conferences, seminars, and journals generate a huge amount of scientific papers. A scientific paper consists of such elements as title, information about the authors, abstract,
The scientific activity assessment is based on the analysis of information flows, which are presented as documents. The quality of these documents affects the evaluation quality, therefore the task of developing the indicator and methods for assessing the quality of scientific documents is very timely and relevant.
This paper discusses how quality is interpreted and how it is measured. Research
quality is a multidimensional concept, where plausibility/soundness, originality,
scientific value, and societal value commonly are perceived as key characteristics [
According to the final report of the European project "European Educational
Research Quality Indicators", a separate project area is the development and testing of
internal and external quality indicators [
In our opinion, the rigor, originality, and integrity indicators depend on the quality
of the scientific sources studied to a certain extent. Therefore, quotes and a reference
list also can be considered as indicators of document quality. The authors of [
A scientific paper consists of such elements as title, information about the authors, abstract, keywords, body and list of references. We focus on the study of the scientific reference list.
Citation is an essential component of any scientific work and one of the important means of scientific communication. In scientific publications, citation can be considered in various aspects. Citation is used to solve problems in many directions. Some of them are presented in Table 1.
All listed directions are important. Bibliometric methods are used widely nowadays. The indicators which are determined on the basis of citation effects the result of academic ratings and the distribution of funding. This contributes to the emergence case of academic fraud and manipulation of indicators in the academic environment. Therefore, ensuring a qualitative assessment of research is relevant.
In practice, the responsibility for the article quality lies with the scientific reviewer, who is an expert in the researched field. He makes decisions based on an analysis of all the substantive and formal properties of a publication, including analyzing a reference list. The authors independently determine the appropriateness and rationality for using a quote.
The authors of this paper aim to investigate and describe the qualitative characteristics of the reference list in already published articles and, based on the mentioned characteristics to propose a software to the reviewer (expert). The proposed software will be recommendatory and will help to reviewer quickly examine the paper and pay attention to certain formal features of the reference list and may help to indicate the unreasonable use of sources. 3
The reviewer's task is to perform a comprehensive evaluation of scientific work. Peer review can be divided into two stages: i) analysis and evaluation of formal features; ii) scientific review of the publication content. Formal features include the following characteristics: total number of citations; time ranges for quotes; number (percentage) of unique source names; quality of sources (indicators and types); a percentage of self-citation; inconsistency of certain citations with the subject of publication; nonspecific sections with excessive citations.
By the example of already published articles that were scientifically reviewed and included in the Web of Science Core Collection (WoS CC) and Scopus, we analyze and characterize some of them of formal features.
We used the 2018 publications on a theme of own scientific work (indexation in WoS CC as of 06.2019) for the analysis of paper formal properties. It is the 83 publications, they were selected from publications included in WoS CC (topic "computational linguistics", 2018 year, categories "computer science", "artificial intelligence", "language linguistics", "information science library science", "computational linguistics").
It should be noted that the average number of sources for publications in various thematic areas is different. The average number of sources in the list for the indicated topic is 39, with the exception of 6 review ones (more than 100 citations).
We determined the percentage of sources included in WoS CC, the year of publication, and the quantity and quality of unique sources for the 20 publications from the list (83 publications).
The citation number in the bibliographic lists of 20 publications is 1087; 506 citations (47%) of them are included in WoS CC. The reliability of publications data included in scientometric databases is not in doubt. The average percentage of links to external sources in the references list is 53%. Possible reasons are certain scientific sources are not included in scientometric databases or scientists have limited access to foreign publications.
The relevance and modernity of the study are evidenced by the use of a significant number of publications in recent years. Review papers are exceptions since a thorough study of the topic is necessary for a qualitative examination of the topic.
The citation structure by years of publication is shown in Fig. 1. Authors often use publications of recent years (2010-2018 – 57%). But older publications (1950-1969) may be considered in review papers.
To ensure the completeness of the study, it is necessary to analyze materials from different sources. The more diverse the list is the better. Therefore, we investigate links from our collection and select unique names of sources for each paper. The total number of unique source names is 245. Fig. 2 shows the percentage of unique sources for 20 publications. The average percentage of unique sources is 48%.
The quality of the reference source is also important. The reviewer, who is an expert on the topic, has information about heavyweight journals in the field and can identify their names on the list. Table 2 presents scientometric indicators of journals which often used in reference lists (TOP-10). More citations (9 out of 20 documents) were identified in the “Computational Linguistics” journal. All journals are wellknown and respected.
An important characteristic of the references is the percentage of self-citation. Authoritative publishers recommend authors to limit the amount of self-citation to 30% of the total number of sources in the citation list. It is believed that this is enough to demonstrate the previous and related works of the authors. In practice, self-citation can be different (from 0% to 100%).
The paper [
In our paper, we investigated the effect of self-citation and obtained a similar to
[
The authors' profiles of Ukrainian scientists distribution by influence self-citation degree on the h-index is shown in Fig. 3. The index increased from 1% to 20% for 56 profiles; for 24 profiles – 21%-30%; for 6 profiles – 31%-40%; for 6 profiles – 41%50%. The h-index in-creased by more than 51% for 8 profiles.
It is determined that self-citation affects the scientometric indices of the authors. It is very difficult to determine the authors' abuse based on the count of self-citation. The authors determine the expediency and justification for the use of quotes, so the problem of self-citation is on an ethical plane. The self-citation percentage of the total number of citations for 100 author profiles is presented in Figure 4.
The use of information tools to automate scientific activity accelerates the scientific process. The reference list, formed in the required format, allows the use of software for data analysis.
We use VOSviewer for quick visualization and analysis of information about authors and the subject of links. It's a software tool for creating visualized scientific landscapes based on textual data. For experiments, we employed the data of Web of Science CC. Through VOSviewer we can quickly analyze such indicators as i) excessive self-citation – a network of author citing; ii) relevance of the quotation topic to the paper topic – keyword analysis from the title and annotation. Fig. 5 shows the author citation network from the reference list for paper1 with using VOSviewer.
The published author is associated with all links and he is present in all 9 citations. In addition to self-citation, unscrupulous authors may cite papers “on order”. Formally, such sources differ in topic and are not related to a specific study. VOSviewer visualizes the semantic relationships between words (title, annotations, keywords). In Fig. 6 we presented the publication titles and annotations from the paper2 citations. Average value of citations 39 196 72% 36% 25%
All words are closely related and demonstrate a certain interdisciplinary interaction (Fig. 6). Thus, we examined the following features and identified them for our data sets. Our results are presented in Table 3. Self-quoting author profile,% of total citations 5% (except for profiling without self-quoting) Increase in the author’s h-index due to self- 5% citation (except for profiling without selfquoting) 5 128 1950 13% 89 390 2019 100% 96% 80%
The described characteristics are not clearly formalized, and therefore the appropriateness of using sources is confirmed by reviewing the content of the publication. Thus, our task is to develop the special software in order to distinguish and present the characteristics of the list for the expert review. 4
The reviewer's task is to perform a comprehensive evaluation of scientific work. In this section, we show 1) a developed application that allows the reviewer to analyze the paper reference list; 2) an approach to the analysis of the bibliography, which allows identifying those sources that may not be relevant to the research topic and, accordingly, artificially increase the performance of other authors. 4.1
In the application development for analyzing the reference list, we tried to take into account not only our research but and the recommendations of conferences and journal’s editorial boards. These main functions we presented below.
Analysis of the publication year. This function helps with the issue of publication relevance, how relevant they are at the time of writing. Our application has a threshold field in which the user can enter the year and the program calculates the number of publications before and after this year. This function allows you to quickly understand whether the author has analyzed the latest research in this area or not (Fig. 7).
As an example, in the field “Results” Fig. 7 a reviewer can see how many publications belong to the 2015 year and higher and a percentage value for these papers towards the total number.
Analysis of self-citation. The user needs to enter the authors of the papers in the “Authors” field and the program gives him the papers of these authors from the reference list and calculates the percentage of the total number (Fig. 8).
As an example, in the field “Results” Fig. 8 a reviewer can see that an author with the surname “Mazov” has 2 papers and a percentage value for self-citation is 33,3%. 4.2
Analyzing the difficulties which reviewers face, we found such a problem as an artificial increase in the citation for a publication. This is realized by citing a source irrelevant to the main topic.
In order to identify such publications, we propose an approach that uses the
methods of computational linguistics and determines the proximity between the sources of
references, and can also take into account paper keywords if it is necessary. Consider
the following example, we have the next reference list (as an example, we take the
reference list from our paper), which consists of 21 sources [
Lefèvre T, Gouagna L-C, Dabiré KR, Elguero E, Fontenille D, Renaud F, et al. (2010) Beer Consumption Increases Human Attractiveness to Malaria Mosquitoes. PLoS ONE 5(3): e9546. https://doi.org/10.1371/journal.pone.0009546
Denote this source as P22. In order to determine the less similar source to the paper topic, we compare the title of each paper with the title of our work “The References Analysis as a Quality Characteristic of a Scientific Article”. The comparison we implement with similarity measure from Spacy library and word embedding models. In the Spacy library, a full sentence word-embedding calculates as an average over all words in the sentence. Before processing, we deleted stopwords in each sentence. As a result, we got Fig. 9.
We received the minimum value of 0,13 for P9. The title of this publication was obtained by transliteration from Russian. Therefore, such proximity coefficient was obtained. However, for the publication with number P22, we received low value and this confirmed our hypothesis. Because this is our "artificial" publication. Other elements from our reference list have very close values from 0,6 to 0,8. It should be noted that this approach does not provide a 100% guarantee for identifying such irrelevant sources, but can help identify candidates for such references. 5
In this work, we have analyzed the qualitative characteristics of a scientific document and focused attention on the paper reference list as an object for research. For research, we selected articles from journals included in the Web of Science Core Collection and data profiles of Ukrainian scientists from Scopus. For analysis, we used the capabilities of Web of Science Core Collection, Scopus, VOSviewer, MS Excel.
According to our research results, we recommend the reviewer, first of all, pay attention to the formalized properties of the citation list. The reference list is researched in this publication and results demonstrate that the quality of the paper content could e defined through the citation list analysis. Due to the allocation and formalization of the citation list properties, it became possible to create a special software tool for reviewers.
We developed an application that allows the reviewer to analyze the reference list of paper and proposed the approach to the analysis of the bibliography, which allows identifying those sources that may not be relevant to the research topic. Our experiments showed that the proposed approach is worked well enough and our next step will be to experiment on the big data sets.