Conducting a scientific literature review is a time consuming activity. This holds for both finding and comparing the related literature. In this paper, we present a workflow and system designed to, among other things, compare research contributions in a scientific knowledge graph. In order to compare contributions, multiple tasks are performed, including finding similar contributions, mapping properties and visualizing the comparison. The presented workflow is implemented in the Open Research Knowledge Graph (ORKG) which enables researchers to find and compare related literature. A preliminary evaluation has been conducted with researchers. Results show that researchers are satisfied with the usability of the user interface, but more importantly, they acknowledge the need and usefulness of contribution comparisons.
When conducting scientific research, finding and comparing
stateof-the-art literature is an important activity. Mainly due to the
unstructured way of publishing scholarly knowledge it is currently
time consuming to find and compare related literature. The Open
Research Knowledge Graph1 (ORKG) [
The benefits of having scholarly knowledge structured include,
among others, the ability to easily find, retrieve but also compare
such knowledge. Comparing resources (scholarly knowledge or
other) can be useful in many contexts [
In this paper, we present a workflow that describes how to select and compare resources describing scholarly knowledge in graph databases. We implement this workflow in the ORKG, which enables the comparison of related literature, including state-of-the-art overviews. In the ORKG, these resources are specifically called research contributions. A research contribution relates the research problem addressed by the contribution, the research method and (at least one) research result. Currently, we do not further constrain the description of these resources. Users can adopt arbitrary third-party vocabularies to describe problems, methods, and results. We thus tackle the following research questions: • RQ1: How to compare research contributions in a graph based system? • RQ2: How to efectively specify and visualize research contribution comparisons in a user interface? 2
Resource (or entity) comparison is a well-known task in a variety of
information systems, for instance in e-commerce or hotel booking
systems. In e-commerce, products can be compared in order to help
customers during the decision process [
The task of comparing research contributions in a graph system
can be decomposed into multiple sub tasks. The first sub task is
ifnding suitable contributions to compare. The most suitable
comparison candidates are similar resources. The actual retrieval of
similar resources can be seen as an information retrieval problem,
with techniques such as TF-IDF [
As suggested above, an efective automated related literature
comparison relies on scholarly knowledge being structured. There
is substantial related work on representing scholarly knowledge in
structured form. Building on the work of numerous philosophers of
science, Hars [
These ontologies are designed to capture primarily metadata
about and structure of scholarly articles, not the actual research
contributions (scholarly knowledge) communicated in articles. In
order to conduct a useful comparison, only comparing article
metadata and structure is oftentimes not suficient. Rather, a
comparison should include (structured descriptions of) problem, materials,
methods, results and perhaps other aspects of scholarly work. The
Open Research Knowledge Graph (ORKG) [
We present a workflow that describes how to perform a comparison of research contributions, thereafter more generally referred to as resources. This workflow consists of four diferent steps: 1) select comparison candidates, 2) select related statements, 3) map properties and 4) visualize comparison. The workflow is depicted in Figure 1. Section 4 presents the implementation for the individual steps. We now discuss each step of the workflow in more detail. 3.1
To perform a comparison, a starting resource is needed. This resource is called main resource and is always manually selected 2http://purl.org/spar/cito 3http://purl.org/spar/c4o 4http://purl.org/spar/fabio 5http://purl.org/spar/biro 6http://purl.org/spar/pro 7http://purl.org/spar/pso 8http://purl.org/spar/pwo 9http://purl.org/spar/doco 10http://purl.org/spar/deo by a user. The main resource is compared against other comparison resources. There are two diferent approaches for selecting the comparison resources. The first approach automatically selects comparison resources based on similarity. The second approach lets users manually select resources.
3.1.1 Find similar resources. Comparing resources makes only sense when resources can sensibly be compared. For example, it does not make (much) sense to compare a city (e.g., dbpedia:berlin) to a car brand (e.g., dbpedia:volkswagen). This of course does not only apply to comparison in knowledge graphs but also applies to comparison in other kinds of databases. We thus argue that it makes only sense to compare resources that are similar. More specifically, resources that share the same (or a similar set of) properties are good comparison candidates. To illustrate this, consider following resources: dbpedia:berlin and dbpedia:new_york_city. Both resources share the property dbo:populationTotal which makes them suitable for comparison. Finding similar resources is therefore based on ifnding resources that share the same or similar properties.
To do so, each comparison resource is converted into a string.
This string is generated by concatenating all properties of the
resource (Algorithm 1). The resulting string is stored. TF-IDF [
Algorithm 1 Paper indexing 1: procedure IndexPaper(paper) 2: for each property in paper do 3: propertyStrinд ← propertyStrinд + property 4: save propertyStrinд 3.1.2 Manual selection. There are scenarios where comparison based on similarity is not suitable. For example, a user may want to compare Germany and France to see which country has the highest GDP. In this case, there is no need to automatically select resources to be compared because they are determined by the user. Therefore, manual selection of resources should also be supported. How to manually select resources is an implementation detail and a proposal for how to do this is presented in Section 4. The result of manual selection is a set of resources used for the comparison. 3.2 This step selects the statements related to the resources to be compared returned in the previous step. Statements are selected transitively to match resources in subject or object position. This search is performed until a predefined maximum transitive depth δ has been reached. The intuition is that the deeper a property is nested the less likely is its relevance for the comparison. As described in the first step, comparisons are built using shared or similar properties of resources. In case the same property has been used between resources, these properties are grouped and form one comparison row. However, often diferent properties are used to describe the same concept. This occurs for various reasons. The most obvious reason is when two diferent ontologies are used to describe the same property. For example, for describing the population of a city, DBpedia uses dbo:populationTotal while WikiData uses WikiData:population (actually the property identifier is P1082; for the purpose here we use the label). When comparing resources, these properties should be considered as equivalent. Especially for community-created knowledge graphs, diferently identified properties likely exist that are, in fact, equivalent.
To overcome this problem, we use FastText [
The similarity matrix γ is generated γ = cos(→p−i , →p−j )i
h with cos(.) as the cosine similarity of vector embeddings for predicate pairs (pi , pj ) ∈ P, whereby P is the set of all resources.
Furthermore, we create a mask matrix Φ that selects predicates of resources ci ∈ C, whereby C is the set of resources to be compared. Formally, Φi, j =
Next, for each selected predicate p we create the matrix φ that slices Φ to include only similar predicates. Formally, φi, j = (Φi, j )
ci ∈ C
pj ∈sim(p)
where sim(p) is the set of predicates with similarity values γ [p] ≥
τ with predicate p. Finally, φ is used to eficiently compute the
common set of predicates [
The final step of the workflow is to visualize the comparison and present the data in a human understandable format. Tabular format is often appropriate for visualizing comparisons. Another aspect of the visualization is determining which properties should be (1) (2) (3)
Algorithm 2 Property mapping 1: procedure MapProperties(properties, threshold) 2: for each property p1 ∈ properties do 3: for each property p2 ∈ properties do 4: similarity ← cos(FastText(p1), FastText(p2)) 5: if similarity > threshold then 6: similar Props ← similar Props ∪ {p1, p2} return similar Props displayed and which ones should be hidden. A property is displayed when it is shared among a predetermined amount τ of papers, where τ mainly depends on comparison use and can be determined based on the total amount of resources in the comparison.
Another aspect of comparison visualization is the possibility to customize the resulting table. This is needed because of the similarity-based matching of properties and the use of predetermined thresholds. For example, users should be able to enable or disable properties. They should also get feedback on property provenance (i.e., the property path). Ultimately, this contributes to a better user experience, with the possibility to manually correct mistakes made by the system. 4
The presented four-step workflow for comparing resources is implemented in the Open Research Knowledge Graph (ORKG), specifically to compare research contributions as a special type of resource. As a companion to the description here, an online video summarizes and demonstrates the ORKG comparison feature11.
The user interface of the comparison feature is seamlessly integrated with the ORKG front end, which is written in JavaScript using the React framework12 and is publicly available online13. The back end of the comparison feature is written as a service separate from the ORKG back end, is written in Python and is available online14.
In the ORKG, each paper consists of at least one research contribution which addresses at least one research problem and is further described with contribution data including for instance materials, methods, implementation, results or other aspects. In the ORKG, it is research contributions that are compared rather than papers.
We will now discuss each step of the presented workflow to illustrate how it is implemented in the ORKG. 4.1
Both approaches presented, namely find similar resources and manual selection, are implemented in the ORKG. The reason for implementing both is that they complement each other. Conducting a comparison based on similarity is useful when a user wants to compare a certain contribution with other (automatically determined similar) contributions (for example, addressing the same problem), while manual contribution selection can be helpful to compare a user defined set of contributions. Figure 2 shows both approaches. As depicted, three similar contributions are suggested 11https://youtu.be/mbe-cVyW_us 12https://reactjs.org/ 13https://gitlab.com/TIBHannover/orkg/orkg-frontend 14https://gitlab.com/TIBHannover/orkg/orkg-similarity to the user. (The corresponding similarity percentage is displayed next to paper title.) These suggested contributions can be directly compared. In contrast, the manual approach works similarly to an online shopping cart. When the “Add to comparison” checkbox is checked, a box is displayed at the bottom of the page. This box shows the manually selected contributions that will be used for the comparison (Figure 3).
To retrieve contributions that are similar to a given contribution, we developed an API endpoint. This endpoint takes the given contribution as input and returns five similar contributions (of which three are displayed). For performance reasons, each contribution is indexed by concatenating the properties to a string (Section 3.1). This string is stored inside a document-oriented database. The indexing happens as soon as a contribution is added. The result of this step is a set of contribution IDs used to perform the comparison. 4.2
An additional API endpoint was developed for the comparison. This endpoint takes the set of contribution IDs as input and returns the data used to display the comparison. The comparison endpoint is responsible for steps two and three of the workflow: selecting the related statements and mapping the properties. For each listed contribution, an ORKG back end query selects all related statements. This is done as described in Section 3.2. The process of selecting statements is repeated until depth δ = 5. This number is chosen to include statements that are not directly related to the resource, but to exclude statements that are less relevant because they are nested too deep. 4.3
Using the API of the previous step, the properties of the selected statements are mapped. As described in the workflow, for each property pair the similarity is calculated using word embeddings. In case the similarity threshold τ ≥ 0.9, the properties are considered to be equivalent and are grouped. The threshold is determined by a trial and error method. Then the results from the API are returned to the UI where they are displayed. Because the comparisons are made for humans, visualizing them efectively is essential and therefore we invested considerable efort on this aspect. Figure 4 displays a comparison for research contributions related to visualization tools published in the literature. In this example, four properties are displayed. Literals are displayed as plain text while resources are displayed as links. When a resource link is selected, a popup is displayed showing the statements related to this resource. By default, only properties that are common to at least two contributions (τ ≥ 2) are displayed. The UI implements some additional features that are particularly useful to compare research contributions. We will now discuss these features is more detail.
4.4.1 Customization. Users can customize comparisons according to their needs. The customization includes transposing the table and customizing the properties. The properties can be enabled/disabled and they can be sorted. Especially the option to disable properties is helpful when resources with many statements are compared. Only properties considered relevant to the user can be selected to display. Customizing the comparison table can be useful before exporting or sharing the comparison.
4.4.2 Sharing and persistence. The comparison can be shared using a link. For sharing the comparison, a persistence mechanism has been built in. Especially when sharing the comparison for research purposes, it is important to share the original comparison. Since resource descriptions may change over time comparisons may also change. To support persistency, the whole state of the comparison is stored in a document-based database.
4.4.3 Export. It is possible to export comparisons in formats PDF, CSV and LATEX. Especially the LATEX export is useful for the ORKG, since the export be directly used in research papers. In addition to the generated LATEX table, a BibTeX file is generated containing the bibliographic information of the papers used in the comparison. Also, a link referring back to the comparison by the ORKG is showed as footnote. Just like the shareable link, this link is persistent and is therefore suitable for use in articles.
In this section we present a preliminary evaluation of the implemented comparison functionality. 5.1
A qualitative evaluation is conducted to determine the usability of
our implementation. Additionally, the evaluation is used to
determine the usefulness of the comparison functionality in general. The
usability is determined using the System Usability Scale (SUS) [
The SUS score ranges from 1 to 100. In our evaluation, the SUS
score is 81, which is considered excellent [
The aim of the contribution comparison functionality is to support literature reviews and make it more eficient. To live up to this aim, the knowledge graph should contain more data. As described in Section 2, structured data is needed to perform an efective and accurate comparison. Currently, such a graph containing research contributions does not exist since most existing initiatives focus solely on document metadata. This is why the ORKG focuses on making the actual research contributions machine-readable. Although the amount of papers in the ORKG is growing, it is currently not suficient for the comparison functionality to be used efectively. The evaluation results suggest that the comparison feature performs well and that users are satisfied with the usability. Additionally, they see the potential of the functionality. Thus, the technical infrastructure is in place for the related literature comparison but more data is needed for an extensive evaluation and real-world use.
In order to evaluate the usability of the interface, a user evaluation is arguably the most suitable method. In total there were only five participants for the user evaluation presented here. While this is not suficient to make any definitive conclusions, it helps to understand what users expect from such a system. The individually provided feedback is also helpful to guide further developments. One of the important outcomes of the evaluation is that all participants agreed on the usefulness of the feature. They saw the potential of conducting literature reviews with the ORKG instead of doing it entirely manually.
Future work will focus on a more extensive evaluation of the individual components of the system. This includes the merging of properties and the similarity functionality. In order to perform such an evaluation, more data should be added to the ORKG. Given that automated related literature comparisons is one of the many advantages of structured scholarly knowledge, more functionalities leveraging this structured data will be developed. An example is faceted search, which provides an alternative to the full-text search commonly used to find related literature. 7
The presented workflow shows how research contributions in a graph database can be compared, which answers our first research question. The workflow consists of four steps in which comparison candidates are selected, related statements are fetched, properties are mapped and finally the comparison is visualized. We presented, evaluated and discussed an implementation of the workflow in the ORKG. The implementation answers our second research question by showing how the comparisons can be efectively visualized in a user interface. The performance evaluation results show that the system scales well. The user evaluation indicates that users see the potential of a related literature comparison functionality, and that the current implementation is user-friendly.
This work was co-funded by the European Research Council for the project ScienceGRAPH (Grant agreement ID: 819536) and the TIB Leibniz Information Centre for Science and Technology. The authors would like to thank the participants of the user evaluation.