. BIR 2023 : 13th International Workshop on Bibliometric-enhanced Information Retrieval at ECIR 2023, April 2,
The English language has gradually gained a dominant position in science [1]. At the same
time, the question of the use of diferent languages in publications remains a subject of scientific
debate, which also finds an echo at the political level. It has been shown by Kirillova in 2019 [
In addition, Moskaleva and Akoev [
The work of Smirnova and Lillis in 2022 is based on a corpus comparing research articles
written in Russian with those written in English in the following disciplines : philosophy,
sociology and economics [
In this article we address the problem of multilingualism in publications through a
corpusbased experiment using the Semantic Scholar Open Research Corpus (S2ORC) [
S2ORC is a large dataset of full-text peer-reviewed research articles, mainly written in English.
We chose this dataset for our experiment because it focuses on English research articles and
spans many academic disciplines. S2ORC includes articles from a wide range of scientific
ifelds including medicine, biology, chemistry, engineering, computer science, physics, materials
science, mathematics, psychology, economics, political science, business, geology, sociology,
geography, environmental science, art, history and philosophy. Other large datasets of research
articles exist, such as the thematic dataset COVID19 [
When studying references to foreign language sources, we should take into account the problem of the multiplicity of scripts. Indeed, if we look at an article written in English, it may contain references in other languages that are expressed either in the native alphabet of the foreign language or in Latin characters. Traditionally, two types of operations have been used to romanise languages with non-Latin scripts. On the one hand, transliteration is the operation that consists in replacing each grapheme of one writing system by a grapheme or group of graphemes of another system, regardless of the pronunciation. On the other hand, transcription is the opposite of transliteration. In transcription, each phoneme of a language is replaced by a grapheme or group of graphemes from one writing system to another. Transliteration is based on national and international standards 1.
1. E.g. Armenian : ISO 9985 : 1996 ; Macedonian, Turkish, Russian, Ukrainian, Belarusian, Bulgarian, non-Slavic languages in Cyrillic, based on ISO 9 : 1995 ; Chinese with NF ISO 7098 : 1992 ; Georgian, using ISO 9984 : 1996 ; Greek with ISO 843 : 1997 ; Hebrew, Yiddish or Syriac, which is a Hebrew script based on the NF ISO 259-2 : 1995 standard ; or Thai, which uses the ISO 11940 : 1998 standard.
We use the full S2ORC version 1 dataset, which contains approximately 81 million open access articles published up to 2020. While the dataset is intended to include only English language articles, a closer look reveals that a small proportion of the articles are in other languages. We show this below. The dataset is available in json format. Each article is identified by its paper_id. For our experiment, we extracted the metadata of articles (title and year) and the metadata of the bibliographic references they contain (titles, years, journals, etc.).
The language of each article and bibliographic reference was detected using Google’s Compact Language Detector v3 (gcld3) Python library 2, which implements a neural network model for language identification. We used the title of the reference as input. Several outputs are provided by gcld3, including the code 3 of the most likely language and the likelihood score for that language. The titles of the bibliographic references were used for the language detection.
In order to obtain a good quality sample, we discarded references for which the probability score was lower than 0.95. This is the case, for example, when the title is too short to identify the language with certainty. References with missing metadata, e.g. missing year, were also ignored (less than 1 % of all references). Furthermore, papers and references with a year before 1950 or after 2020 were ignored, as such values are most likely due to typing errors in the dataset (less than 0.5% of all references).
The metadata for the citing paper (title and year) was obtained by using its paper_id in S2ORC. The language of the citing paper was determined in the same way as for the references, using its title.
The quality of the language detection varies for some of the poorly endowed languages. The choice of the gcld3 library was done after testing several other libraries for language detection, e.g. spacy lang-detect. We found that for our task, gcld3 provides better results and is much faster. We manually evaluated language detection by examining a subset of 100 titles for each language. If the observed precision was below 50 % we excluded that language 4 from our experiment. For the majority of the languages that remained the precision is above 75%.
The gcld3 recognises cases of Latin transliteration and assigns a language code followed by -Latn, e.g. ru-Latn stands for Russian text transliterated with Latin characters. In our experiment, we do not make a distinction between references that are transliterated and those that are written in the native script.
2. https://github.com/google/cld3 3. The language codes use the IETF BCP 47 language tags, which combine several standards : ISO 639, ISO 15924, ISO 3166-1 and UN M.49. F. The subtags are maintained by the IANA Language Subtag Registry.
4. This applies to the following languages : af, ar, az, be, bg-Latn, ceb, co, cy, eo, et, eu, fil, fy, ga, gd, gl, ha, haw, hi, ht, ig, jv, kk, ku, ky, lb, mg, mi, mk, mn, ms, mt, ne, ny, sm, sn, so, sq, st, su, sw, uz, vi, xh, yi, yo, zu.
In order to study how the diferent language groups are represented in the dataset, we have classified the languages with their language tags into language groups. The summary is presented in table 1, which contains all language tags identified as present in the dataset. Only English was not associated with its group (the Indo-European Germanic languages), but we have separated it from the other languages because it is the dominant language of the corpus. Languages that were excluded from the experiment because of the poor quality of the language detection are presented in red.
As a result of the language detection, after applying the above criteria, we obtained a subset of S2ORC containing a total of 5.9 million articles with 109.9 million references for which the language was detected with a probability score above 0.95.
Although the dataset is intended to contain only English research articles, we found 35,463 articles that were identified as being written in other languages, out of a total of 5,934,799 articles (0.60 %). Figure 1 shows a bar chart of the number of articles found for each language.
Most of the articles are in Latin, and this result may be biased because titles in the biomedical domain may be misclassified as Latin text. For the remaining languages, we manually sampled and checked the presence of such articles in the dataset. In our study, we have taken this result as an opportunity to work with a subset of multilingual (non-English) research articles and study their references. Overall, we found that the dataset contains articles in 32 languages (including English) and references to sources in 35 languages (including English).
As might be expected, the vast majority of references in the dataset are to English sources. They account for 107,437,865 references out of a total of 109,838,938 references (97.81 %). The remaining 2,401,073 references are to non-English sources. Figure 2 shows a bar chart of the number of references to sources in the diferent languages.
To gain a better understanding of the types of foreign language sources cited, we extracted titles of non-English sources cited in English articles and translated some of them for analysis and discussion. The translation is intended to be informative. The results are shown in the table 2. Languages that were excluded from the experiment because of the poor quality of the language detection are presented in red. We can see that the titles correspond to studies that have local significance and dimension, relate to a particular country or geographical area, and are written in their native language. For example, the title in haw (Hawaiian) represents a study on the history of ”Kahalu’u and Keauhou”. The example in sq (Albanian) is a study of hydrocarbon energy in Albania, and the example in mk (Macedonian) is about geological studies in the villages of Kosel and Pesočan.
We have studied the distribution of citations to non-English sources with respect to the year of reference and with respect to the year of publication of the citing article.
Figure 3 shows the relative proportion of non-English sources cited by articles published between 1950 and 2020. The numbers on the horizontal axis represent the average number of sources cited per article in our dataset. The nine most frequent languages are shown in diferent colours, and the last group contains all other languages. We can see that between 1960 and 1980, the non-English references cited were mostly in French and German. Since 2000, however, their share has been decreasing, with the appearance of a large number of sources cited in Portuguese and Spanish. Overall, the relative share of non-English references has increased since 2000.
To study how foreign language sources are cited in English articles, we looked at the subset of references from English articles to non-English sources. Figure 4 shows the relative proportions of diferent language groups cited in English articles. The left-hand side of the figure shows the linguistic groups of the citing articles, while the right-hand side shows the linguistic groups of the references. The majority of references are from the Indo-European Romance and Germanic groups, which are the closest languages to English. The Indo-European Slavic languages come next, and all the other groups have very small proportions of citations.
Finally, we examined the subset of references from non-English articles. The results are shown in figure 5. Again, the left side of the figure represents the language groups of the citing articles, while the right side represents the language groups of the references. It is interesting to note the dominance of English in this group, as we see that the majority of citations in these articles are to English sources. In addition, articles written in all other language groups cite a majority of English sources. In particular, a significant proportion of publications in Indo-European Romance languages (other than English) cite sources in the same language group, and the same is true for the other Indo-European groups.
The results presented in this study may be biased by several diferent factors. Firstly, the quality of language detection may vary between diferent languages, particularly for poorly endowed languages which may have a low recognition rate. Manual cleaning and evaluation would be required to produce better quality data. Secondly, the use of titles as proxies to detect the language of a publication is not optimal, as titles can be very short or contain technical terms that may wrongly point to Latin or English. However, given the size of the dataset and the data available for references, this was the only way to identify the languages of the sources. Another possibility would be to use titles and abstracts. This study could therefore be improved by harvesting abstracts for the references in the dataset.
The choice of the S2ORC dataset introduces an important bias related to the English language.
As S2ORC is intended to include only English language research articles, the subset of
nonEnglish language articles that we analysed may not be representative of these languages, as it is
made up of publications that are included in S2ORC. The S2ORC dataset was constructed by
retaining only papers identified as English using the cld2 tool run over titles and abstracts [
In general, the choice of sources to be cited in a publication and their languages is a complex process influenced by many factors, such as the languages spoken by co-authors, the subject of the study, and so on. The editorial requirements of some journals may favour English language references or require the titles of foreign language references to be translated into the language of the publication.
We carried out an analysis of the Semantic Scholar Open Research Corpus (S2ORC) and identified the languages of the research articles and their references based on their titles. While the vast majority are in English, we found articles in 44 diferent languages and references in 54 languages. We observed their linguistic groups and their distribution over time between 1950 and 2020. The results allow us to observe the dominance of English in science, where the vast majority of citations in non-English publications are to English sources. We also show that the relative share of non-English citations will increase from 2020 onwards.
Evaluating and improving the quality of language detection can provide us with more reliable results in the future. In the long run, this work aims to propose a typology of foreign language references in order to better understand the way they are used in publications. Indeed, one of the issues that interest bibliometricians today is the reflection on the context of citation. The classification of citation contexts according to multilingual criteria has not been addressed in the literature to our knowledge. To this end, an extension of this approach will focus on national and other multilingual corpora. It also seems relevant to study such references in terms of their place in publications and their linguistic contexts.
This work was supported by grant number ANR-20-CE38-0003-01 and grant number ANR21-CE38-0003-01.
[1] P. S. Rao, The role of english as a global language, Research Journal of English 4 (2019) 65–79.