Bibliographic information systems need to rely on metadata provided by various sources in various forms and with various quality. The talk gives some insights how the dblp bibliography as an example for such a system is maintained and improved. It shows how metadata can be automatically harvested from publisher websites and how the harvesting process can be steered. It also discusses some open sources of bibliographic metadata and how they can be used to enrich existing bibliographic data, but also how varying their quality is.
Bibliographic information systems are a valuable source for searching, exploring and accessing scienti c publications, but also for assessing scienti c performance and impact of individual researchers, institutions and publication venues. Important examples for systems include commercial providers like Google Scholar and Microsoft Academic, academic initiatives like CiteSeer1, ResearchGate2, and Bibsonomy3, publisher portals like the ACM Digital Library4 and Elsevier Scopus5, and domain-speci c portals like PubMed6, Semantic Scholar7, and dblp8. These systems often di er in the type and volume of metadata provided, in the services they o er on-top of the metadata, and in the scienti c domains they cover. For example, the dblp bibliography focuses on providing metadata provided by publishers or collected from public sources for di erent types of publications in aggregated form, i.e., grouped by authors or venues. dblp does not provide further information like abstracts or citations that can only be extracted from publisher Web sites or even from the full-text of publications. ResearchGate and Bibsonomy provide data contributed by their users; Google Scholar, MS Academic, and Semantic Scholar collect publications and the associated metadata on the Web and extract further information from the full-text; the ACM DL and Elsevier Scopus combine publisher-provided metadata with additional information extracted from the full-text.
Building and maintaining such a bibliographic information system incurs a number of di cult challenges: Publication venues to be included need to be selected and monitored on a regular basis for new publications; information on publications needs to be extracted from their original source, often a Web site; authors need to be disambiguated. Some of these steps can be supported by additional sources of metadata like ORCID. We will now sketch how dblp solves some of these challenges. We will also highlight how citation information may become available in the future. 2
The completeness of a bibliographic information system is clearly an issue. There is a huge number of publication venues today that has clearly outgrown the capacity of metadata providers. As dblp attempts to provide a certain level of data quality that requires at least a partial manual check, it is impossible to cover all published works. Thus, the decision to include a venue in dblp is made based on a certain set of minimal requirements, documented at https://dblp. org/faq/. These requirements include a certain prominence within computer science of people involved in the venue, a certain level of quality control, and a long-time availability of the corresponding publications. The high diversity of publication types makes this decision even more di cult { for example, some technical reports are cited very frequently and thus seem to be very important, however, they usually have neither gone through any academic quality checking nor, especially for commercial technical reports, are not provided in a way that would ensure their long-term availability.
Once a publication venue like a conference or a journal is selected, it is
important that new publications appearing at this venue are included with short
delay. For some of the big publishers, dblp receives regular updates in form of
structured metadata on new publications. For other publishers and even for some
important conferences, this information needs to be crawled and extracted from
the corresponding Web sites. To reduce overhead, monitoring of these sources
is done based on a schedule that takes into account, among other things, the
typical publication frequency; Neumann et al. [
A critical problem for the data quality of a bibliographic information system is
author disambiguation, i.e., deciding which individuals (as opposed to names)
have authored which publications. Common problems include synonyms, i.e.,
di erent individuals with the same name, and homonyms, i.e., di erent names for
the same individual. A typical example for a synonym is the author name \Wei
Wang", which corresponds to more than 130 individuals in dblp9; still, more than
1,000 publications have not yet been assigned to one of these individuals. Many
automated solutions have been proposed in the literature for this problem [
With the recent success of the ORCID initiative10, more and more authors of publications are now annotated with their ORCID, which makes disambiguation a trivial task. In addition, the ORCID author pro les can help to identify individuals and their publications. Adding ORCID information, dblp could identify and correct a few thousand author pro les that were not corresponding to individuals; however, as ORCID feeds its pro les partly from other metadata providers, not all information in a pro le may be correct, thus again manual observation is required. 4
An important class of bibliographic meta information is citations. Citations are an important ingredient of many bibliometric measures, and a large number of potential applications like citation recommendation, collaboration recommendation, reviewer recommendation, and venue recommendation, but also author disambiguation rely or can at least bene t from the availability of citation information. For many years, anyone who wanted to make use of citations had to extract them from the full-text of publications, which in turn was not available for most publications, or had to rely on a manual collection. Well-known examples of citation collections created with automated methods are CiteSeer, the Open Academic Graph11, Microsoft Academic Graph12 (a snapshot of which is included in the Open Academic Graph), or Semantic Scholar. For all automated methods, the quality of the extracted citations is usually not perfect, and often not all citations of a publication can be extracted and mapped to a reference 9 https://dblp.org/pers/hd/w/Wang:Wei 10 https://orcid.org/ 11 https://aminer.org/open-academic-graph 12 https://www.microsoft.com/en-us/research/project/ microsoft-academic-graph/ collection of publications due to errors already present in the original document, introduced by an intermediate OCR process, or errors in the extraction process. Manual citation collection has the potential to provide data with better quality, but does not scale to large volumes of publications. For dblp, an initial e ort was undertaken to manually collect a small number of citations for important publications; however, the manual e ort for this was orders of magnitude larger than collecting other metadata of publications, hence this e ort was not continued.
The recent Initiative for Open Citations13 has changed the picture here. With
the agreement of many important publishers to openly publish citation
information for their publications on Crossref14, there is for the rst time a large bulk
of citation information available for further analysis and usage. I4OC reports
that today more than 50% of all citations on Crossref are freely available. The
OpenCitations initiative, in turn, has made this information available in a
format than can be easily processed by applications [
However, despite these highly useful activities, only a small fraction of citations of computer science publications is freely available today. To better understand this, we made an attempt to collect citations from Crossref for all publications in dblp (as of January 27, 2018). The experiment was done with Crossref data from late February 2018. Of the roughly 4 million publications in dblp, about 3.2 million have a DOI assigned and could thus be looked up with the Crossref API; of these, only about 70,000 were unknown to Crossref. For slightly less than 600,000 publications, Crossref returned citation information, which is slightly less than 20% of the available publications (and less than 15% of all publications in dblp). This is not nearly enough to be useful for an analysis or recommendation. Moreover, of the approximately 16 million citations retrieved, only slightly more than 4 million could be mapped to dblp based on the provided DOI; the other citations either did not provide a DOI or could not be mapped to a paper in dblp. So while the initiative is clearly extremely useful, the citation data available at this time is not yet extensive enough to be useful in a bibliographic information system. Extracting missing citations from the full-text of publications, but also mapping citations without DOI information to a reference collection such as dblp based on their reference text is thus still an important problem.
An interesting research question in this context is estimating when a collection includes enough citation information to provide stable estimates for the impact of authors and publications. Is the citation data now available in Crossref already su cient, or do we need more citations (and probably citations from 13 https://i4oc.org/ 14 https://www.crossref.org/ 15 https://www.springernature.com/de/researchers/scigraph more diverse sources)? Do we need new bibliometric measures that take missing information into account, and that can deal with the inherent uncertainty? 5
A collection of bibliographic needs to be published under a license that is as
open as possible to be useful for scienti c purposes. Many of the collections
mentioned above are available under such licences. The collection of metadata
included in dblp, for example, has been released under the Open Data Commons
ODC-BY 1.0 license. A daily updated XML dump can be found at http://
dblp.org/xml/; the data format is explained in [
Recently, the work on dblp has focused on improving the volume of new publications per year, while at the same time improving overall data quality by cleaning author pro les. A current activity focuses on networking with other data providers, including ORCID and Wikidata. Future activities on the instance level will include increasing the coverage of historic publications and monographs (including PhD theses).
From a broader perspective, integrating validated bibliographic information from the diverse providers and from all scienti c disciplines and publishing this information under an open licence would clearly be a signi cant improvement over the status quo. There clearly is a large overlap of authors between computer science and neighbouring disciplines like mathematics and biology, but also seemingly distant disciplines like psychology. On the modelling side, dblp currently lacks a good representation of conference series and individual conference events. This problem is more complex than it may seem as conferences may change publishers, may pause for some time, or may change name (like the WWW conference), or even may merge with others or split. On the data side, an important aspect of future work is indexing scienti c data sets, which are now often published at providers like DataCite16 and need to be made accessible through information providers; ideally, also their their use in scienti c literature should be tracked and reported as metadata of the publications.
An interesting long-term goal for data providers would be to collect additional metadata on conferences and journals beyond classical bibliographic metadata. This may include, for example, information on the members of the program committee, organizers in di erent roles, keynote speakers, etc. This information would allow for a better estimation of the reputation of scientists in their corresponding scienti c community. Most of this data is already available on the Web, but needs to be collected, cleaned, and completed before it can be useful.