Coverage is an important criterion when evaluating information systems. This exploratory study investigates this issue by submitting the same query to different databases relevant to the query topic. Information was retrieved f from three databases: ACM Digital Library, WOS (with the Proceedings Citation Index) and Scopus. The search phrase was “information retrieval”, publication years were between 2013 and 2016. The location of the search phrase was limited to title and abstract (and also keywords for WOS) and the subject area was limited to computer science or information science in WOS, computer science or social science in Scopus. From the ACM Digital Library data were retrieved from the more comprehensive ACM Guide to Computer Literature that includes also non-ACM data and also covers the major journals in information science. Altogether 9050 items were retrieved, out of which 5591 (62%) items were retrieved by a single database only, and only 1059 (12%) items were located in all three databases. There are great variations in the citation counts as well.
Cyril Cleverdon [
We only found a few articles that assessed information retrieval research, all
having a different flavor from what is presented here. For example, Ding, Chowdhury
and Foo [
For this study data were collected in May 2017, from three databases, ACM, Scopus and WOS. The search query was identical in all three cases: “information retrieval” as a phrase and so were the publication years, 2013-2016. However, there were slight differences in the search strategies as described below.
The ACM Digital Library allows to search in two sources: the ACM Full Text Collection and the more comprehensive (in terms of meta-data) ACM Guide to Computing Literature. The second option was chosen and we searched for “information retrieval” in the abstract or in the title. After data cleansing (removal of duplicates, items with missing titles or authors), 3849 items remained out of the initially retrieved 4161 items. ACM Digital Library allows to download meta-data, but these do not include citation counts, which had to be added manually.
In Scopus, the searches were also in title and abstract, however in addition to limiting the publication years to 2013-2016, we had to limit the retrieved items to those that were in the area of computer science or social science (to include information science as well). Out of the 5635 items retrieved, 5458 remained after data cleaning.
WOS does not allow to limit the search to abstract only, so we chose topic, which includes title, abstract and keywords. We had to exclude keywords from Scopus because inclusion of keywords added mainly noise (12,931 documents for a keyword search limited to publication years and subject area as above). An examination of a sample of the documents showed that the addition of keywords introduced a lot of noise, while in ACM the keyword search had a huge overlap with the title and abstract search). The search in WOS included the Science Citation Index, the Social Science Citation Index, the Arts & Humanities Citation Index, the Proceedings Citation Indexes and the Emerging Journal Citation Index, the subject areas were limited to computer science and information science and 4265 documents were retrieved.
Next a list of unique documents was created from the items retrieved from the different data sources. This part was rather time consuming, because not all items had DOIs, and occasionally the DOIs were incorrect. Pairwise comparisons were conducted to discover overlap, and to collect the citation counts of the given item from the three databases. Then for items not matched by DOI, title and publication year were compared. These matches were manually checked, as in several cases the items with identical titles and publication years were published in two different venues. It was impossible to automatically match items using the publication source as well, because there are no uniform naming conventions for proceeding titles (e.g. to publication source for papers in SIGIR 2015, appear as: • “Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval” in ACM • “SIGIR 2015 - Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval” in Scopus and WOS and CIKM 2015 appears as • “Proceedings of the 25th ACM International on Conference on Information and Knowledge Management” in ACM • “CIKM'16: PROCEEDINGS OF THE 2016 ACM CONFERENCE ON
INFORMATION AND KNOWLEDGE MANAGEMENT” in WOS WOS retrieved items from this conference series only in 2016, while Scopus indexed only the 2014 proceedings, and ACM retrieved items from all four years, however the source title for 2013 was slightly different, using & instead of and.
Interestingly for conducting the manual check of items that were paired only by title and publication year the start and end page of the items were most useful. Altogether 9050 unique items were identified.
It should be noted that it was not feasible to use Google Scholar or Microsoft Academic Search. In Google Scholar, one can search in the title, but not in the abstract, and appearance of the term “information retrieval” in the full text cannot serve as evidence that the paper is about information retrieval. In any case, even when conducting a title search Google Scholar reports as of May 2017, about 4,240 results published between 2013 and 2017, and for a general search about 45,400 results. Since Google Scholar does not allow to retrieve more than 1000 results, it was not feasible to include Google Scholar. Microsoft Academic Search reports more than 50,000 results for the time period, and 28,700 results for items published in 2013 alone. 2.2
Longitudinal publication trends for the whole set of publications and also for the individual databases was charted both in terms of number of publications and in terms of number of citations. The h—index of the topic in each database was computed. Most cited publications were identified.
Table 2 shows the number of citations publications received from the time of publication until May 2017 per database. Scopus is highest for all years, WOS is second for documents published in 2013 and third for the rest of the years in terms of average number of citations received per paper. Citations accumulate over the years; thus, it is not surprising that both total citation and citations per paper decrease as the time between publications and citations decrease. 3.27 2.04 0.98 0.38 1.57
The most interesting finding of this explorative study is the small overlap between the results retrieved by the databases as can be seen in Fig.2. We found only 1,059 documents (12% out of the total number of retrieved documents – 9050) that were retrieved by all three databases. On the other hand, 5,591 documents (62%) were found in a single database only. The largest overlap was between Scopus and WOS, 58% of the documents found by WOS were retrieved also by Scopus, and the smallest overlap was found between WOS and ACM, only 28% of the publication in WOS were found also by ACM. 3.3
The h-index of the retrieved publications was 24 for ACM, 25 for WOS and 35 for
Scopus. Although Hirsch [
Last, the set of most cited documents retrieved by each of the databases is displayed in order to highlight the differences in terms of citation between them. The top three documents ranked by citation counts are displayed in Table 3 for ACM, Scopus and WOS respectively. Table 3 shows, that the intersection between the three sets is empty! This finding supports the subtitle: “A tale of three databases”. Yuan et al Xiao et al.
Panichella et al.
Deng & Yu Hussein et al.
Brehmer& Munzner Fu et al.
Dit et al. This contribution belongs to the BIR part of BIRNDL, as it explores one of the evaluation criteria of IR systems, coverage, applying bibliometric techniques. It emphasizes the need for searching in multiple databases in order to increase recall. The results highlight the considerable differences between the databases both in terms of the number of results found for the given query, and it terms of the citations these publications receive.
The study is exploratory in its nature and has its limitations. It should be extended to try to understand the meaning of these differences, i.e. why does each database tell us a different story? A single query is not enough for far reaching conclusions, but enough to raise interest to further explore the issue.