Bibliometric Software: The Most Commonly Used in Research Alejandra Colina Vargas* , Marcos Espinoza-Mina, Diana López Alvarez and Johanna Navarro Espinosa Universidad Ecotec, Samborondón, Ecuador Abstract Scientific research allows for increasing the knowledge of reality and the phenomena that manifest and evolve in it. Today, research is considered a professional competence, which is mainly developed in university institutions. Day by day, the so-called information society is flooded with research works of all types and qualities. At the same time, there is an increase in the number of annual publications that analyze data and measure intellectual works and their contribution to the scientific heritage of each discipline. A series of indicators are generated based on various parameters such as authors, type of document, language, and keywords, among others. The visualization of these numerous indicators, which vary over time, can be achieved effectively through the use of bibliometric software, which is also renewed in its functionality and features. The following is a study, assisted by systematic and bibliometric review techniques, with the aim of identifying the most widely used bibliometric software and those emerging that serve as a guide and good practice in new bibliometric research to obtain notable and relevant knowledge. The applications "VOSviewer", "CiteSpace" and "Bibliometrix" stand out and tools such as "Scopus API R code", "Covidence", "HistCite", "BICOMB", "EndNote X9.6 Statistical", "Ucinet 6.0", "Tools for Innovation Monitoring", "Profiles Research Networking Software", among others, emerge. Keywords Bibliography, Statistics, Metrics 1. Introduction The number of scientific works published today is very large; it is impossible to read all these contributions, even if they are classified as a science or activity. From this point of view, it is difficult for a single person, without methods, techniques, and tools, to judge whether the scientific activity of a country, an institution, or an expert is relevant. In the face of this difficulty, bibliometrics emerges. A bibliometric approach provides, through a detailed analysis of published articles, a qualitative and quantitative result of the contributions made to the scientific community [1]. In bibliometrics, a wide variety of measures are used to evaluate the content of articles and ICAIW 2022: Workshops at the 5th International Conference on Applied Informatics 2022, October 27–29, 2022, Arequipa, Peru * Corresponding author $ acolina@ecotec.edu.ec (A. Colina Vargas); mespinoza@ecotec.edu.ec (M. Espinoza-Mina); dlopez@ecotec.edu.ec (D. López Alvarez); jnavarro@ecotec.edu.ec (J. Navarro Espinosa)  0000-0003-1514-8852 (A. Colina Vargas); 0000-0003-1530-7243 (M. Espinoza-Mina); 0000-0003-2457-7683 (D. López Alvarez); 0000-0002-1741-2234 (J. Navarro Espinosa) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 47 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 the performance of their authors. Researchers, through bibliometrics, try to measure, among others, the evolution of a scientific domain, the impact of scholarly publications, authorship patterns, and the process of scientific knowledge production [2]. In addition to the above, bibliometric analysis is a meta-analytical tool used to measure or access the characteristics of any variable, whether there is an effect, and whether it is positive or negative [3]. It is important that research metrics are defined in an efficient, transparent, and solid way; demonstrating the true impact and influence of the research. Different methods have been developed to seek answers to research questions, using collaborative networks, thematic research groups, patterns of historical evolution, and trends in the topics addressed [4]. Research designs for bibliometrics are structured primarily with the identification of suitable bibliographic databases, the development of search criteria, and the selection of informatics tools for analysis [5]. The development and application of software tools for scientific research can help researchers to accelerate the processes of discovery, innovation, and information exchange [4]. At present, there is software with various approaches, which allow for generating quantitative, qualitative, and graphical bibliometric results that support the study of scientific production. Bibliometrics can assess publications and development trends in a scientific field, and reveal tentative key research directions, by analyzing databases and publication characteristics [6]. Large scientific and electronic databases quickly provide lists of publications and corresponding citation records [7]. Web of Science and Scopus are commercial bibliographic databases, widely used in scientometric cartographic research, with data quality superior to other similar databases [4]. This information, provided by the databases, becomes the input data for the software to generate bibliometric results. In the academic and scientific environment, it is a requirement of science managers and those who define policies to increase bibliometric studies to support the decisions of the developed research [7]. These works are useful for understanding the transformation of the scientific literature or trends in particular segments or geographic areas [2]. As previously analyzed by the authors, few studies have been conducted to explore bibliometric studies and to recognize the technological tools used to increase and improve the quality of scientific production of this type. Given the gap found, and the exponential evolution of information technologies, it was pertinent to carry out an exhaustive review covering the last few years of bibliometric publications [5]. In view of the above and considering the importance of computer tools, which are used in different bibliometric studies, the following is a systematic review and a meta-analysis assisted with bibliometric methods, to collect and summarize evidence on the most used contemporary bibliometric software, in the evaluation of scientific production, from different perspectives that guide researchers in the generation of remarkable and relevant scientific knowledge. 2. Methodology The aim of this systematic review was to identify and evaluate the main software used in bibliometric studies and other emerging tools. To this end, two groups of scientific articles were 48 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 reviewed; the first group included research that used globally recognized bibliometric software, and the second group, also bibliometric studies, did not use any of the bibliometric software identified as the most representative, but worked with other software. For the literature review, the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement was used, published in 2009 and updated in 2020 [8], which proposes a list of twenty-seven verification criteria grouped into seven sections; from which those relevant to this study were taken. 2.1. Eligibility Criteria and Sources Some guidelines were defined for the extraction of the documents to be examined. The main inclusion and exclusion criterion was given by a list of the most used software in bibliometric studies. For this, two contemporary research papers were previously searched and selected, which list the main bibliometric software. From the research works conducted [2, 9]; the following software for bibliometric analy- sis were listed: "Bibexcel", "Bibliometrix", "BiblioTools", "Content Analysis Toolkit for Aca- demic Research (CATAR)", "CiteSpace", "CitNetExporer", "CopalRed", "CRExplorer", "Headstart", "HistCite", "InCite Retrieve", "IN-SPIRE", "Loet Letdesdorff", "Network Workbench", "Publish or Perish", "RobotReviewer", "SAINT", "Sci2 Tool", "SciMAT", "Scientometric Project", "Scopus API R code", "Sitkis", "Structural Dynamics Toolbox", "Utopia Documents", "VantagePoint", "VOSviewer", and "Web of Science API". Of the software listed in the research [2, 9], only those that are active in their development, whether free to use or open source, were evaluated and considered for the present review study. After a detailed characterization, the following were finally selected: "Bibexcel", "Bibliometrix", "BiblioTools", "CATAR", "CiteSpace", "CitNetExporer", "CRExplorer", "Headstart", "Publish or Perish", "RobotReviewer", "SciMAT", "Scopus API R code" and "VOSviewer". Other inclusion criteria determined were that bibliometric studies published between 2018 and 2022 had to be evaluated, and the type of document chosen was an article; in addition, they had to have been written in English, and be freely available for reading, without the need for payment. The Scopus database was used, which contains abstracts and citations of multidisciplinary scientific literature from peer-reviewed journals and web sources. This database ensures that only the highest quality data are indexed through rigorous content selection [10][10]. 2.2. Search, selection and extraction strategy Considering the eligibility criteria, two search strings were structured for the Scopus database, which are presented below, resulting in 1,246 and 210 documents, respectively. String 1: TITLE-ABS- KEY ( ( ( "Bibexcel" OR "Bibliometrix" OR "BiblioTools" OR "CATAR" OR "CiteSpace" OR "CitNetExporer" OR "CRExplorer" OR "Headstart" OR "Publish or Perish" OR "RobotReviewer" OR "SciMAT" OR "Scopus API R code" OR "VOSviewer" ) AND ( ( "bibliometric" OR "scientometric" ) ( "Bibexcel" OR "Bibliometrix" OR "BiblioTools" OR "CATAR" OR "CiteSpace" OR "CitNetExporer" OR "CRExplorer" OR "Headstart" OR "Publish or Perish" 49 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 OR "RobotReviewer" OR "SciMAT" OR "Scopus API R code" OR "VOSviewer" ) AND ( "bibliometric" OR "scientometric" ) ) AND ( ( LIMIT- TO ( OA , "all" ) ) AND ( LIMIT-TO ( PUBYEAR , 2023 ) OR LIMIT-TO ( PUBYEAR , 2022 ) OR LIMIT-TO ( PUBYEAR , 2021 ) OR LIMIT-TO ( PUBYEAR , 2020 ) OR LIMIT-TO ( PUBYEAR , 2019 ) OR LIMIT-TO ( PUBYEAR , 2018 ) ) AND ( LIMIT-TO ( DOCTYPE , "ar" ) ) AND ( LIMIT-TO ( LANGUAGE , "English" ) ) ). String 2: ( TITLE-ABS-KEY ( ( ( "bibliometric" OR "scientometric" ) ) AND TITLE-ABS-KEY ( ( ( "software" ) ) AND NOT TITLE-ABS- KEY ( ( ( "Bibexcel" OR "Bibliometrix" OR "BiblioTools" OR "CATAR" OR "CiteSpace" OR "CitNetExporer" OR "CRExplorer" OR "Headstart" OR "Publish or Perish" OR "RobotReviewer" OR "SciMAT" OR "Scopus API R code" OR "VOSviewer" OR "VOSviewer" ) ) ) AND ( LIMIT-TO ( OA , "all" ) ) AND ( LIMIT-TO ( PUBYEAR , 2023 ) OR LIMIT-TO ( PUBYEAR , 2022 ) OR LIMIT-TO ( PUBYEAR , 2021 ) OR LIMIT-TO ( PUBYEAR , 2020 ) OR LIMIT-TO ( PUBYEAR , 2019 ) OR LIMIT-TO ( PUBYEAR , 2018 ) ) AND ( LIMIT-TO ( DOCTYPE , "ar" ) ) AND ( LIMIT-TO ( LANGUAGE , "English" ) ) ). Once each search string was executed in Scopus, all results were selected. Immediately, for the extraction process, from the same database, the option "CSV export" and "Export document settings" were used. All the information fields provided by the tool were selected. The records were exported to "RIS", "CSV" and "BibTex" formats for further processing. 2.3. Synthesis methods To make a detailed recognition of the data extracted from Scopus, bibliometric techniques were used, supported by functions of the Bibliometrix library, which are executed on the R Studio programming tool. In addition, using the R Studio platform and the R language, algorithms were constructed and applied to the two groups of records (1246 and 210 documents) to process and evaluate the data and summarize them graphically. A meta-analysis was performed with the review; this study made it possible to achieve the statistical synthesis of the data extracted from both groups of scientific document data from Scopus. In order to know where bibliometric studies tend to go within the different fields of application, the fields of education and training of the reference system of the International Standard Classification of Education (ISCED) [11], which is part of the international family of Economic and Social Classifications of the United Nations, were considered. In view of the criteria described in [11], the fields classified as broad, specific and detailed were extracted and unified, leaving the following list as an evaluation tool: "Basic programmes and qualifications"; "Literacy and numeracy"; "Personal skills and development"; "Education science"; "Training for pre-school teachers"; "Teacher training without subject specialisation"; "Teacher training with subject specialisation"; "Audio-visual techniques and media production"; "Fashion, interior and industrial design"; "Fine arts"; "Handicrafts"; "Music and performing arts"; "Religion and theology"; "History and archaeology"; "Philosophy and ethics"; "Language 50 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 acquisition"; "Literature and linguistics"; "Economics"; "Political sciences and civics"; "Psychol- ogy"; "Sociology and cultural studies"; "Journalism and reporting"; "Library, information and archival studies"; "Accounting and taxation"; "Finance, banking and insurance"; "Management and administration"; "Marketing and advertising"; "Secretarial and office work"; "Wholesale and retail sales"; "Work skills"; "Law"; "Biology"; "Biochemistry"; "Environmental sciences"; "Natural environments and wildlife"; "Chemistry"; "Earth sciences"; "Physics"; "Mathematics"; "Statistics"; "Computer use"; "Database and network design and administration"; "Software and applications development and analysis"; "Chemical engineering and processes"; "Environmental protection technology"; "Electricity and energy"; "Electronics and automation"; "Mechanics and metal trades"; "Motor vehicles, ships and aircraft"; "Food processing"; "Materials"; "Textiles"; "Mining and extraction"; "Architecture and town planning"; "Building and civil engineering"; "Crop and livestock production"; "Horticulture"; "Forestry"; "Fisheries"; "Veterinary"; "Dental studies"; "Medicine"; "Nursing and midwifery"; "Medical diagnostic and treatment technol- ogy"; "Therapy and rehabilitation"; "Pharmacy"; "Traditional and complementary medicine and therapy"; "Care of the elderly and of disabled adults"; "Child care and youth services"; "Social work and counselling"; "Domestic services"; "Hair and beauty services"; "Hotel, restaurants and catering"; "Sports"; "Travel, tourism and leisure"; "Community sanitation"; "Occupational health and safety"; "Military and defence"; "Protection of persons and property"; "Transport services"; "Generic programmes and qualifications"; "Education"; "Arts and humanities"; "Arts"; "Hu- manities"; "Languages"; "Social sciences, journalism and information"; "Social and behavioural sciences"; "Journalism and information"; "Business, administration and law"; "Business and administration"; "Natural sciences, mathematics and statistics"; "Biological and related sci- encies"; "Environment"; "Physical sciences"; "Mathematics and statistics"; "Information and Communication Technologies"; "Engineering, manufacturing and construction"; "Engineering and engineering trades"; "Manufacturing and processing"; "Architecture and construction"; "Agriculture, forestry, fisheries and veterinary"; "Agriculture"; "Health and welfare"; "Health"; "Welfare"; "Services"; "Personal services"; "Hygiene and occupational health services"; "Security services". Subsequently, with the 210 records of scientific articles, corresponding to the second group of data extracted, a new group of data was constructed, through a bibliometric process, with the fifty most cited documents. All of them were reviewed by two of the authors of this study. The review was carried out through a content analysis, emphasizing the summary and methodological aspects that were applied to identify and find any emerging software used in bibliometric studies. 3. Results The results obtained from the analysis of the data extracted after applying the two structured search strings with the inclusion and exclusion criteria defined by the authors are presented be- low. The first group of data was used to identify the software currently most used in bibliometric studies; and the second group was used to try to identify other bibliometric software. 51 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 3.1. Most used bibliometric software A total of 1246 documents were extracted from Scopus between 2018 and 2022. All these studies considered the following software: "Bibexcel", "Bibliometrix", "BiblioTools", "CATAR", "CiteSpace", "CitNetExporer", "CRExplorer", "Headstart", "Publish or Perish", "RobotReviewer", "SciMAT", "Scopus API R code", and "VOSviewer". Various bibliometric techniques were applied to the bibliographic information extracted from the database to determine the visibility of scientific production using bibliometric software and the intensity of its use. 3.1.1. Chronological production of bibliometric studies Figure 1 shows how interest in bibliometric research has steadily increased, with an annual increase in the scientific production of 64.71%. In addition, an average number of citations per document of 6.64 was established, showing that this type of research work is highly consulted and referenced. Figure 1: Chronological production 3.1.2. Top countries in bibliometric scientific production and citations China stands out in scientific production considering the use of software and bibliometric techniques. With a much lower number than this country, the following countries stand out: Spain, Malaysia, India, Brazil, Italy, Turkey, Iran, Colombia, and United Kingdom. Of this leading group, only United Kingdom has a very good international collaboration, with an index of 77%. Turkey’s production has been developed only with internal collaboration. In the rest of the countries, although there is both local and international collaboration, when evaluating the contribution in scientific productions, all of them are consolidated mainly with collaboration within the country itself, see Table 1. 52 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 Table 1 Top countries by scientific output in bibliometrics MCP Country Articles Freq SCP MCP Ratio CHINA 510 4.811 424 86 169 SPAIN 70 660 53 17 243 MALAYSIA 34 321 22 12 353 INDIA 32 302 19 13 406 BRAZIL 24 226 14 10 417 ITALY 24 226 14 10 417 TURKEY 21 198 21 0 0 IRAN 20 189 19 1 50 COLOMBIA 18 170 15 3 167 UNITED KINGDOM 18 170 4 14 778 (Freq) Frequency (SCP) Intra-country collaboration index (MCP) Inter-country collaboration index (MCP Ratio) Inter-country relationship Table 2 Top countries by citations in bibliometric studies Country Total Citations Average Article Citations CHINA 2833 5.55 SPAIN 761 10.87 UNITED KINGDOM 399 22.17 ITALY 260 10.83 HONG KONG 232 19.33 AUSTRALIA 195 13.00 INDIA 195 06.09 CANADA 186 14.31 ECUADOR 185 16.82 BRAZIL 164 6.83 Despite the fact that China has a total number of citations much higher than the second most cited country, and by logic a wide gap with the rest of the countries, China has a lower average number of citations of articles, lower than the rest of the leading countries in the group of scientific production evaluated, see Table 2. 3.1.3. Top affiliations with bibliometric scientific production On evaluating the bibliographic records, it was also found that the authors of the scientific production studied have 1793 different affiliations. The following universities stand out: "Central South University", "Sichuan University", "An-Najah National University", "Beijing University of Chinese Medicine" and "Southeast University". See Table 3. 53 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 Table 3 Affiliations Number of articles Proportion CENTRAL SOUTH UNIVERSITY 50 0,01370614 SICHUAN UNIVERSITY 38 0,010416667 AN-NAJAH NATIONAL UNIVERSITY 37 0,010142544 BEIJING UNIVERSITY OF CHINESE MEDICINE 33 0,009046053 SOUTHEAST UNIVERSITY 31 0,008497807 ESPOL POLYTECHNIC UNIVERSITY 30 0,008223684 KING ABDULAZIZ UNIVERSITY 28 0,007675439 AN-NAJAH NATIONAL UNIVERSITY HOSPITAL 25 0,00685307 LANZHOU UNIVERSITY 25 0,00685307 UNIVERSITY OF GRANADA 22 0,006030702 3.1.4. Keyword frequency Table 4 Most frequently occurring keywords Authors’ Keywords Articles Keywords associated by SCOPUS Articles BIBLIOMETRIC ANALYSIS 444 BIBLIOMETRICS 759 VOSVIEWER 313 HUMAN 414 CITESPACE 250 ARTICLE 356 BIBLIOMETRICS 233 UNITED STATES 303 BIBLIOMETRIC 151 HUMANS 268 WEB OF SCIENCE 92 PUBLICATION 221 SCIENTOMETRICS 63 CHINA 174 SCOPUS 62 SOFTWARE 141 COVID-19 59 WEB OF SCIENCE 132 VISUALIZATION 56 MEDICAL RESEARCH 131 Of the bibliometric studies evaluated, the following keywords stand out, reported directly by the authors: "BIBLIOMETRIC ANALYSIS", "VOSVIEWER", "CITESPACE", "BIBLIOMET- RICS" and "BIBLIOMETRIC". Among the keywords associated with the Scopus database itself, the following stand out: "BIBLIOMETRICS", "HUMAN", "ARTICLE", "UNITED STATES" and "HUMANS". See Table 4. 3.1.5. Presence of software in bibliometrics In the abstracts of the 1246 documents, we proceeded to search for the presence of each of the previously selected software, as part of the applied method. As a result of the search, it was found that "VOSviewer" was the most used bibliometric software in this first group of evaluated scientific papers; far behind, and with significant distances, were "CiteSpace" and "Bibliometrix". See Figure 2. 54 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 Figure 2: Presence of software in bibliometrics 3.1.6. Presence of software in bibliometrics The fields with the greatest presence in the abstracts of the 1246 papers evaluated were: "Health" in 245 documents, "Environment" in 190, "Medicine" in 143, "Education" in 105, and "Materials" in 85. With an incidence of less than 50 papers, the fields were: "Biology", "Economics", "Services", "Chemistry", "Statistics", "Agriculture", "Law", "Psychology", "Arts", "Sports", "Biochemistry", "Languages", "Environmental sciences", "Pharmacy", "Physics", "Mathematics", "Forestry", "Vet- erinary", "Humanities", "Fisheries", "Welfare", "Information and Communication Technologies", "Earth sciences", "Arts and humanities", "Food processing", "Textiles", "Nursing and midwifery", "Therapy and rehabilitation", "Occupational health and safety", "Physical sciences". Figure 3 shows a simple and quick representation of the most relevant fields identified. 3.2. Emerging bibliometric software The second group of bibliographic records was extracted from the Scopus database. As a main feature of the search string applied, prior to the extraction, it is noted that previously identified software were excluded, i.e.: "Bibexcel", "Bibliometrix", "BiblioTools", "CATAR", "CiteSpace", "CitNetExporer", "CRExplorer", "Headstart", "Publish or Perish", "RobotReviewer", "SciMAT", "Scopus API R code", and "VOSviewer". The extraction generated 210 bibliographic records between 2018 and 2022. The growth rate of this type of scientific production was 14.42%. The records report the involvement of 878 authors with affiliation to 476 institutions, from 36 different countries. Although the general opinion held among bibliometric experts seems to be that citations represent a good, but not perfect, a measure of impact, the use of citation indicators has increased [12]. From the set of documents obtained in the extraction, i.e. the 210 bibliographic records taken from Scopus, the first fifty, which had the highest number of citations, were taken, see Table 5. 55 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 Figure 3: Word cloud of application fields found in the documents A careful content analysis was carried out, taking into account mainly what was stated in the summary and the methodological procedure declared, for each of the selected articles. This review was carried out with the purpose of recognizing other tools or applications, in addition to those already identified, that the authors rely on to improve bibliometric studies. The articles were evaluated, and as a result, it can be pointed out that 82% of the sample corresponds to research strictly related to bibliometric studies. In addition, 56.09% use software other than those initially identified in this study. Other relevant and useful software for bibliometric analysis and visualization of results emerged, characterized by being open source, free or proprietary. The use of statistical analysis software was found, such as: "Microsoft Excel" (proprietary) and "XLSTAT" (proprietary), "R" (free), "GraphPad Prism" (proprietary), and "Addinsoft" (proprietary). In other cases, some data mining applications, such as "QDA Miner software" (proprietary), "Wordstat software" (proprietary), and "Python’s Pandas library" (free). In addition, we found software specifically cataloged in bibliometrics, such as "Covidence" (proprietary), "HistCite" (proprietary), "BICOMB" (free), "EndNote X9.6 Statistical" (proprietary), "Ucinet 6.0" (propri- etary), "Tools for Innovation Monitoring (TIM)" (proprietary), "Profiles Research Networking Software" (free). Finally, among the visualization programs for bibliomentry, the following stand out: "gCLUTO" (free) and "Pajek software" (free). In addition, a number of variations in the way the names of the tools are spelled were detected, including "VOSviewer", "BICOMB" and "CitNetExplorer", which could limit the results of the identification and selection of suitable software applications for the purposes of other studies. 56 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 Table 5 Most cited articles of the second group of evaluated data TC Title Software used Ref. 397 The State Of OA: A Large-Scale Analysis Of The Preva- does not apply [13] lence And Impact Of Open Access Articles 187 The Pride Database Resources In 2022: A Hub For Mass does not apply [14] Spectrometry-Based Proteomics Evidences QDA Miner software 139 Scientific Development Of Smart Farming Technologies [15] Provalis Research And Their Application In Brazil 72 The Four Dimensions Of Social Network Analysis: An does not apply [16] Overview Of Research Methods, Applications, And Soft- ware Tools 65 A Bibliometric History Of The Journal Of Psychology VOSviewer [1] Between 1936 And 2015 63 Artificial Intelligence In Health Care: Bibliometric Anal- HistCite [17] ysis 48 Current Status And Challenges Of Additive Manufac- does not apply [18] turing In Orthopaedics: An Overview 45 3D Printed Medical Parts With Different Materials Us- does not apply [19] ing Additive Manufacturing 45 The Impact Of Social Media In Business Growth And R [20] Performance: A Scientometrics Analysis 42 Opencitations, An Infrastructure Organization For OpenCitation Index [21] Open Scholarship APIs RESTful of Sco- 38 Pybliometrics: Scriptable Bibliometrics Using A Python [22] pus Interface To Scopus 31 A Systematic Review Of Research On Sustainability In VOSviewer [23] Mergers And Acquisitions 30 Assessment Of Citations Of The Retracted Article By Covidence [24] Wakefield Et Al With Fraudulent Claims Of An Associ- ation Between Vaccination And Autism 30 The Primary Total Knee Arthroplasty: A Global Analy- VOSviewer; R [25] sis 29 Bibliometric Analysis In Motorcycle Accident Research: VOSviewer [26] A Global Overview 57 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 29 Open-Design: A State Of The Art Review R Core Team 2015 [27] 28 Sustainable Approach Of Using Sugarcane Bagasse Ash VOSviewer [28] In Cement-Based Composites: A Systematic Review 28 Classification Of Forecasting Methods In Production VOSviewer [29] Engineering 27 The State Of Exosomes Research: A Global Visualized VOSviewer [30] Analysis 24 Bibliometric Analysis Of Tumor Immunotherapy Stud- BICOMB; gCLUTO [31] ies 23 How New Concepts Become Universal Scientific Ap- R basado en Diderot [32] proaches: Insights From Citation Network Analysis Of Agent-Based Complex Systems Science 22 Corporate Social Responsibility And Supply Chain Man- VOSviewer [33] agement: Framing And Pushing Forward The Debate EndNote X9.6 Statisti- 22 Twenty-Year Span Of Global Coronavirus Research cal; SPSS; CiteSpace; [34] Trends: A Bibliometric Analysis VOSviewer 20 The Relationship Between Innovation And Sustainabil- VOSviewer [35] ity: A Bibliometric Review Of The Literature OpenCitation Index 19 Software Review: COCI, The Opencitations Index Of [36] COCI REST API Crossref Open Doi-To-Doi Citations 19 Identification Of Recent Trends In Research On Vitamin BICOMB; gCLUTO [37] D: A Quantitative And Co-Word Analysis Wordstat Provalis soft- 19 A Bibliometric And Topic Analysis On Future Compe- [38] ware tences At Smart Factories 17 A Comprehensive Overview Of Geopolymer Compos- VOSviewer [39] ites: A Bibliometric Analysis And Literature Review 17 A Large-Scale Analysis Of Bioinformatics Code On does not apply [40] Github 17 Citation Network Analysis Of The Novel Coronavirus CitNetExplorer [41] Disease 2019 (Covid-19) "Tools for Innovation 16 A Bibliometric-Based Technique To Identify Emerging [42] Monitoring" (TIM) Photovoltaic Technologies In A Comparative Assess- ment With Expert Review 58 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 16 Twenty-Five Years Of The Information Systems Journal: VOSviewer [43] A Bibliometric And Ontological Overview 16 Bibliometric Analysis Of Global Research On The Re- HistCite; CiteSpace [44] habilitation Of Spinal Cord Injury In The Past Two Decades 15 Scientific Mapping Of Industry 4.0 Research: A Biblio- VOSviewer [45] metric Analysis 14 Measuring The Impact Of Pharmacoepidemiologic Re- Almetrics [46] search Using Altmetrics: A Case Study Of A Cnodes Drug-Safety Article Profiles Research Net- 14 Trends And Visibility Of "Digital Health" As A Keyword [47] working Software In Articles By Jmir Publications In The New Millennium: Bibliographic-Bibliometric Analysis 14 The Most-Cited Authors Who Published Papers In Jmir Pajek software [48] Mhealth And Uhealth Using The Authorship-Weighted Scheme: Bibliometric Analysis 13 50 Years Of International Journal Of Systems Science: CiteSpace; VOSviewer [49] A Review Of The Past And Trends For The Future Python’s pandas 13 A Comprehensive Overview Of The Covid-19 Litera- [50] library ture: Machine Learning-Based Bibliometric Analysis BICOMB; gCLUTO; 13 Mapping Theme Trends And Knowledge Structures For Ucinet 6.0; GraphPad [51] Human Neural Stem Cells: A Quantitative And Co- Prism Word Biclustering Analysis For The 2013-2018 Period Gender-API; XLSTAT; 12 Author Gender Inequality In Medical Imaging Journals Microsoft Excel, [52] And The Covid-19 Pandemic Addinsoft; R 12 Recognizing The Value Of Software: A Software Cita- does not apply [53] tion Guide 11 Ten Years Of Energy Efficiency: A Bibliometric Analysis VOSviewer [54] 11 Mathematics Anxiety: Mapping The Literature By Bib- VOSviewer [55] liometric Analysis 11 Trends And Future Research In Electronic Marketing: VOSviewer [56] A Bibliometric Analysis Of Twenty Years 11 Research Trends In Career Success: A Bibliometric Re- VOSviewer [57] view 59 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 10 Business Capabilities For Industrial Firms: A Bibliomet- CitNetExplorer [58] ric Analysis Of Research Diffusion And Impact Within And Beyond Industrial Marketing Management 10 Evlncrnas 2.0: An Updated Database Of Manually Cu- does not apply [59] rated Functional Long Non-Coding Rnas Validated By Low-Throughput Experiments 10 Documenting Contributions To Scholarly Articles Us- R library tenzing [60] ing Credit And Tenzing 10 Customer Relationship Management (CRM): A Biblio- does not apply [61] metric Analysis TC: Total citations Ref.: Reference 4. Conclusions This type of study provides information on applications, technological tools, or software that can be used by researchers to improve the efficiency of bibliometric research. It supports the development of bibliometric research, which can be applied to any discipline or science, in the sense that it can be developed through a wide variety of tools, for bibliometric evaluation and mapping in any selected field of research. In the first analysis, a total of 27 software tools used in bibliometric studies were identified. This software were evaluated, and due to their characteristics of being actively developed, free of charge, non-commercial, or open source, the following applications were selected for this study: "Bibexcel", "Bibliometrix", "BiblioTools", "CATAR", "CiteSpace", "CitNetExporer", "CRExplorer", "Headstart", "Publish or Perish", "RobotReviewer", "SciMAT", "Scopus API R code", "VOSviewer". Two sets of bibliographic records were extracted from the Scopus database; a total of 1,246 and 210 items were selected respectively. The first was the result of a search that expressly included the names of the previously identified software; the second was based on the search for papers that had considered software-based techniques and bibliometrics, but excluded the listed software, for the recognition of other bibliometric application alternatives. From the bibliometric evaluation of the first group of data, it is worth highlighting the high annual increase rate of scientific production, which was 64.71%, a sign of the importance of this topic in different fields. As a fact of regional interest, the Latin American countries of Colombia, Ecuador, and Brazil stand out in their scientific production, in the first group of evaluated works related to bibliometrics. The first country for the amount of scientific production and the next two for the number of citations registered. The following institutions stand out for the number of articles published: Central South University, Sichuan University, An-Najah National University. The software with the greatest presence in scientific production related to bibliometrics are "VOSviewer", "CiteSpace" and "Bibliometrix"; when evaluating their frequency, it should be noted that the distance separating each of the three tools is significant. 60 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 The fields detected with the greatest presence in the abstracts evaluated were: "Health", "Environment", "Medicine", "Education" and "Materials". Through the content analysis method, fifty articles were evaluated, selected by their number of citations, from the second group of bibliographic records taken from Scopus; this in order to identify emerging software related to bibliometric techniques. It was found that more than half of the documents reviewed used software other than those already recognized. In addition to the software classified as bibliometric or visualizers of bibliometric results, those that allow statistical analysis or data mining were identified. Statistical programs such as: "Microsoft Excel" and "XLSTAT, R", "GraphPad Prism" and "Addinsoft" are used. In data mining: "QDA Miner software", "Wordstat software" and "Python’s Pandas library". Specific in bibliometrics: "Covidence", "HistCite", "BICOMB", "EndNote X9.6 Statistical", "Ucinet 6.0", "Tools for Innovation Monitoring" and "Profiles Research Networking Software". As visualization software for bibliomentrics, "gCLUTO" and "Pajek" were found. Information technologies are evolving exponentially, and new software alternatives are emerging every day in all areas. It is important to periodically evaluate the scientific production of bibliometric type, to know in a permanent way, the new software that supports this type of activity. In addition, in future research, it would be advisable to review and describe the functional scope of bibliometric software in order to classify and evaluate it. References [1] A. Tur-Porcar, A. Mas-Tur, J. M. Merigó, N. Roig-Tierno, J. Watt, A bibliometric history of the journal of psychology between 1936 and 2015, The Journal of psychology 152 (2018) 199–225. [2] M. N. Muruganandham, Scientometrics, tools and their software’s: A play for vital role in research, Journal of Emerging Technologies and Innovative Research 6 (2019) 8. [3] S. M. Gillani, A. B. A. Senin, J. Bode, S. M. Gillani, et al., Bibliometric analysis of digital entrepreneurial education and student intention; reviewed and analyzed by vosviewer from google scholar, International Journal of Interactive Mobile Technologies (iJIM) 16 (2022) 48–65. [4] J. Li, F. Goerlandt, G. Reniers, An overview of scientometric mapping for the safety science community: Methods, tools, and framework, Safety Science 134 (2021) 105093. [5] Y. Gao, S. L. Wong, N. Noordin, et al., A bibliometric analysis of online faculty professional development in higher education, Research and Practice in Technology Enhanced Learning 17 (2022) 1–19. [6] K. Min, Y. Wu, S. Wang, H. Yang, H. Deng, J. Wei, X. Zhang, H. Zhou, W. Zhu, Y. Gu, et al., Developmental trends and research hotspots in bronchoscopy anesthesia: A bibliometric study, Frontiers in Medicine 9 (2022). [7] R. Karpagam, S. Gopalakrishnan, M. Natarajan, Scientific measures and tools for research literature output, Indian journal of Science and technology 4 (2011) 828–833. [8] M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, L. Shamseer, J. M. Tetzlaff, E. A. Akl, S. E. Brennan, et al., Declaración prisma 2020: 61 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 una guía actualizada para la publicación de revisiones sistemáticas, Revista Española de Cardiología 74 (2021) 790–799. [9] M. E. Bales, D. N. Wright, P. R. Oxley, T. R. Wheeler, Bibliometric visualization and analysis software: State of the art, workflows, and best practices, 2020. [10] J. Baas, M. Schotten, A. Plume, G. Côté, R. Karimi, Scopus as a curated, high-quality bibliometric data source for academic research in quantitative science studies, Quantitative science studies 1 (2020) 377–386. [11] UNESCO, Isced fields of education and training 2013 (isced-f 2013): manual to accompany the international standard classification of education 2011, Montreal: UNESCO IFS (2014). [12] D. W. Aksnes, L. Langfeldt, P. Wouters, Citations, citation indicators, and research quality: An overview of basic concepts and theories, Sage Open 9 (2019) 2158244019829575. [13] H. Piwowar, J. Priem, V. Larivière, J. P. Alperin, L. Matthias, B. Norlander, A. Farley, J. West, S. Haustein, The state of oa: a large-scale analysis of the prevalence and impact of open access articles, PeerJ 6 (2018) e4375. [14] Y. Perez-Riverol, J. Bai, C. Bandla, D. García-Seisdedos, S. Hewapathirana, S. Kamatchi- nathan, D. J. Kundu, A. Prakash, A. Frericks-Zipper, M. Eisenacher, et al., The pride database resources in 2022: a hub for mass spectrometry-based proteomics evidences, Nucleic acids research 50 (2022) D543–D552. [15] D. Pivoto, P. D. Waquil, E. Talamini, C. P. S. Finocchio, V. F. Dalla Corte, G. de Vargas Mores, Scientific development of smart farming technologies and their application in brazil, Information processing in agriculture 5 (2018) 21–32. [16] D. Camacho, Á. Panizo-LLedot, G. Bello-Orgaz, A. Gonzalez-Pardo, E. Cambria, The four dimensions of social network analysis: An overview of research methods, applications, and software tools, Information Fusion 63 (2020) 88–120. [17] Y. Guo, Z. Hao, S. Zhao, J. Gong, F. Yang, et al., Artificial intelligence in health care: bibliometric analysis, Journal of Medical Internet Research 22 (2020) e18228. [18] M. Javaid, A. Haleem, Current status and challenges of additive manufacturing in or- thopaedics: an overview, Journal of clinical orthopaedics and trauma 10 (2019) 380–386. [19] A. Haleem, M. Javaid, 3d printed medical parts with different materials using additive manufacturing, Clinical Epidemiology and Global Health 8 (2020) 215–223. [20] A. Pourkhani, K. Abdipour, B. Baher, M. Moslehpour, The impact of social media in business growth and performance: A scientometrics analysis, International Journal of Data and Network Science 3 (2019) 223–244. [21] S. Peroni, D. Shotton, Opencitations, an infrastructure organization for open scholarship, Quantitative Science Studies 1 (2020) 428–444. [22] M. E. Rose, J. R. Kitchin, pybliometrics: Scriptable bibliometrics using a python interface to scopus, SoftwareX 10 (2019) 100263. [23] T. González-Torres, J.-L. Rodríguez-Sánchez, E. Pelechano-Barahona, F. E. García-Muiña, A systematic review of research on sustainability in mergers and acquisitions, Sustainability 12 (2020) 513. [24] E. M. Suelzer, J. Deal, K. L. Hanus, B. Ruggeri, R. Sieracki, E. Witkowski, Assessment of citations of the retracted article by wakefield et al with fraudulent claims of an association between vaccination and autism, JAMA network open 2 (2019) e1915552–e1915552. [25] J. Gao, D. Xing, S. Dong, J. Lin, The primary total knee arthroplasty: a global analysis, 62 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 Journal of orthopaedic surgery and research 15 (2020) 1–12. [26] H. Ospina-Mateus, L. A. Quintana Jiménez, F. J. Lopez-Valdes, K. Salas-Navarro, Biblio- metric analysis in motorcycle accident research: A global overview, Scientometrics 121 (2019) 793–815. [27] É. Boisseau, J.-F. Omhover, C. Bouchard, Open-design: A state of the art review, Design Science 4 (2018). [28] W. Ahmad, A. Ahmad, K. A. Ostrowski, F. Aslam, P. Joyklad, P. Zajdel, Sustainable approach of using sugarcane bagasse ash in cement-based composites: A systematic review, Case Studies in Construction Materials 15 (2021) e00698. [29] C. Winkowski, Classification of forecasting methods in production engineering, Engineer- ing Management in Production and Services 11 (2019) 23–33. [30] B. Wang, D. Xing, Y. Zhu, S. Dong, B. Zhao, The state of exosomes research: a global visualized analysis, BioMed Research International 2019 (2019). [31] K. Lu, S. Yu, M. Yu, D. Sun, Z. Huang, H. Xing, J. Bi, Z. Li, Z. Li, X. Liu, et al., Bibliometric analysis of tumor immunotherapy studies, Medical Science Monitor 24 (2018) 3405–3414. [32] C. E. Vincenot, How new concepts become universal scientific approaches: insights from citation network analysis of agent-based complex systems science, Proceedings of the Royal Society B: Biological Sciences 285 (2018) 20172360. [33] N. M. Modak, S. Sinha, A. Raj, S. Panda, J. M. Merigó, A. B. L. de Sousa Jabbour, Corporate social responsibility and supply chain management: Framing and pushing forward the debate, Journal of Cleaner Production 273 (2020) 122981. [34] Y. Zhou, L. Chen, Twenty-year span of global coronavirus research trends: A bibliometric analysis, International Journal of Environmental Research and Public Health 17 (2020) 3082. [35] D. Maier, A. Maier, I. As, chilean, L. Anastasiu, O. Gavris, , The relationship between innovation and sustainability: A bibliometric review of the literature, Sustainability 12 (2020) 4083. [36] I. Heibi, S. Peroni, D. Shotton, Software review: Coci, the opencitations index of crossref open doi-to-doi citations, Scientometrics 121 (2019) 1213–1228. [37] A. Yang, Q. Lv, F. Chen, D. Wang, Y. Liu, W. Shi, Identification of recent trends in research on vitamin d: A quantitative and co-word analysis, Medical science monitor: international medical journal of experimental and clinical research 25 (2019) 643. [38] A. Jerman, M. Pejić Bach, A. Bertoncelj, A bibliometric and topic analysis on future competences at smart factories, Machines 6 (2018) 41. [39] H. Yang, L. Liu, W. Yang, H. Liu, W. Ahmad, A. Ahmad, F. Aslam, P. Joyklad, A comprehen- sive overview of geopolymer composites: A bibliometric analysis and literature review, Case Studies in Construction Materials 16 (2022) e00830. [40] P. H. Russell, R. L. Johnson, S. Ananthan, B. Harnke, N. E. Carlson, A large-scale analysis of bioinformatics code on github, PLoS One 13 (2018) e0205898. [41] C. Martinez-Perez, C. Alvarez-Peregrina, C. Villa-Collar, M. Á. Sánchez-Tena, Citation network analysis of the novel coronavirus disease 2019 (covid-19), International Journal of Environmental Research and Public Health 17 (2020) 7690. [42] A. Moro, E. Boelman, G. Joanny, J. L. Garcia, A bibliometric-based technique to identify emerging photovoltaic technologies in a comparative assessment with expert review, 63 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 Renewable Energy 123 (2018) 407–416. [43] A. La Paz, J. M. Merigó, P. Powell, A. Ramaprasad, T. Syn, Twenty-five years of the information systems journal: A bibliometric and ontological overview, Information Systems Journal 30 (2020) 431–457. [44] X. Liu, N. Liu, M. Zhou, Y. Lu, F. Li, Bibliometric analysis of global research on the rehabilitation of spinal cord injury in the past two decades, Therapeutics and clinical risk management 15 (2019) 1. [45] H. Sikandar, Y. Vaicondam, N. Khan, M. I. Qureshi, A. Ullah, Scientific mapping of industry 4.0 research: a bibliometric analysis, International Journal of Interactive Mobile Technologies 15 (2021) 129–147. [46] J.-M. Gamble, R. L. Traynor, A. Gruzd, P. Mai, C. R. Dormuth, I. S. Sketris, Measuring the impact of pharmacoepidemiologic research using altmetrics: A case study of a cnodes drug-safety article, Pharmacoepidemiology and drug safety 29 (2020) 93–102. [47] A. Ahmadvand, D. Kavanagh, M. Clark, J. Drennan, L. Nissen, et al., Trends and visibility of “digital health” as a keyword in articles by jmir publications in the new millennium: Bibliographic-bibliometric analysis, Journal of medical Internet research 21 (2019) e10477. [48] W.-C. Kan, W. Chou, T.-W. Chien, Y.-T. Yeh, P.-H. Chou, et al., The most-cited authors who published papers in jmir mhealth and uhealth using the authorship-weighted scheme: bibliometric analysis, JMIR mHealth and uHealth 8 (2020) e11567. [49] X. Wang, Y. Chang, Z. Xu, Z. Wang, V. Kadirkamanathan, 50 years of international journal of systems science: a review of the past and trends for the future, International Journal of Systems Science 52 (2021) 1515–1538. [50] A. Abd-Alrazaq, J. Schneider, B. Mifsud, T. Alam, M. Househ, M. Hamdi, Z. Shah, et al., A comprehensive overview of the covid-19 literature: machine learning–based bibliometric analysis, Journal of medical Internet research 23 (2021) e23703. [51] W.-J. Wei, B. Shi, X. Guan, J.-Y. Ma, Y.-C. Wang, J. Liu, Mapping theme trends and knowledge structures for human neural stem cells: a quantitative and co-word biclustering analysis for the 2013–2018 period, Neural regeneration research 14 (2019) 1823. [52] E. Quak, G. Girault, M. A. Thenint, K. Weyts, J. Lequesne, C. Lasnon, Author gender inequality in medical imaging journals and the covid-19 pandemic, Radiology (2021). [53] D. S. Katz, N. P. C. Hong, T. Clark, A. Muench, S. Stall, D. Bouquin, M. Cannon, S. Edmunds, T. Faez, P. Feeney, et al., Recognizing the value of software: a software citation guide, F1000Research 9 (2020). [54] A. Trianni, J. M. Merigó, P. Bertoldi, Ten years of energy efficiency: A bibliometric analysis, Energy Efficiency 11 (2018) 1917–1939. [55] Z. Ersozlu, M. Karakus, Mathematics anxiety: mapping the literature by bibliometric analysis, EURASIA Journal of Mathematics, Science and Technology Education 15 (2019) em1673. [56] P. Gao, F. Meng, M. N. Mata, J. M. Martins, S. Iqbal, A. B. Correia, R. M. Dantas, A. Waheed, J. Xavier Rita, M. Farrukh, Trends and future research in electronic marketing: A biblio- metric analysis of twenty years, Journal of Theoretical and Applied Electronic Commerce Research 16 (2021) 1667–1679. [57] R. Pico-Saltos, P. Carrión-Mero, N. Montalván-Burbano, J. Garzás, A. Redchuk, Research trends in career success: A bibliometric review, Sustainability 13 (2021) 4625. 64 Alejandra Colina Vargas et al. CEUR Workshop Proceedings 47–65 [58] Y. Kouropalatis, A. Giudici, O. A. Acar, Business capabilities for industrial firms: A biblio- metric analysis of research diffusion and impact within and beyond industrial marketing management, Industrial Marketing Management 83 (2019) 8–20. [59] B. Zhou, B. Ji, K. Liu, G. Hu, F. Wang, Q. Chen, R. Yu, P. Huang, J. Ren, C. Guo, et al., Evlncrnas 2.0: an updated database of manually curated functional long non-coding rnas validated by low-throughput experiments, Nucleic Acids Research 49 (2021) D86–D91. [60] A. O. Holcombe, M. Kovacs, F. Aust, B. Aczel, Documenting contributions to scholarly articles using credit and tenzing, PLoS One 15 (2020) e0244611. [61] V. Guerola-Navarro, R. Oltra-Badenes, H. Gil-Gomez, J.-A. Gil-Gomez, Customer rela- tionship management (crm): a bibliometric analysis, International Journal of Services Operations and Informatics 10 (2020) 242–268. 65