ChImp: Visualizing Ontology Changes and their Impact in Protégé Romana Pernisch1[0000−0001−8590−1817] , Mirko Serbak1 , Daniele Dell’Aglio2,1[0000−0003−4904−2511] , and Abraham Bernstein1[0000−0002−0128−4602] 1 Department of Informatics, University of Zurich, Zurich, Switzerland {pernisch,dellaglio,bernstein}@ifi.uzh.ch, mirko.serbak@uzh.ch 2 Department of Computer Science, Aalborg University, Aalborg, Denmark dade@cs.aau.dk Abstract. Today, ontologies are an established part of many applica- tions and research. However, ontologies evolve over time, and ontology editors—engineers and domain experts—need to be aware of the con- sequences of changes while editing. Ontology editors might not be fully aware of how they are influencing consistency, quality, or the structure of the ontology, possibly causing applications to fail. To support editors and increase their sensitivity towards the consequences of their actions, we conducted a user survey to elicit preferences for representing changes, e.g., with ontology metrics such as number of classes and properties. Based on the survey, we developed ChImp—a Protégé plug-in to display information about the impact of changes in real-time. During editing of the ontology, ChImp lists the applied changes, checks and displays the consistency status, and reports measures describing the effect on the structure of the ontology. Akin to software IDEs and integrated testing approaches, we hope that displaying such metrics will help to improve ontology evolution processes in the long run. Keywords: Ontology evolution · Evolution Impact · Change visualiza- tion · Protégé plugin. 1 Introduction In recent years, ontologies and controlled vocabularies have gained popularity. They can be viewed as graphs that contain concepts and individuals, which are put into various relations with each other using properties. They can represent portions of the real world, e.g., products, people, cities, and emotions. Various communities invested time, money, and resources into building such graphs and using them they can improve search engines, recommenders, and research. The management of ontologies has inspired several studies, which analyze different aspects of them, such as creation, usage, and contents. However, most studies either focus on the usage, disregarding its evolution, or vice versa. When c 2020 for this paper by its authors. Use permitted under Creative Com- Copyright � mons License Attribution 4.0 International (CC BY 4.0). 47 ChImp: Visualizing Ontology Changes and their Impact in Protégé studies only consider the usage of ontologies and how to improve upon exist- ing applications, they often assume a static knowledge graph. This assumption does not correspond to reality since those graphs are continuously evolving, e.g., by adding new gene interactions or updating the composition of national gov- ernments over time. Only a small number of studies consider both usage and evolution at the same time [12, 20, 11, 22]. Ontology engineers are often unaware of all the consequences they cause while editing an ontology. Their main focus is on the knowledge being integrated. Respective changes do, however, affect not only the ontology itself (e.g., its consistency and quality), but also the services built on top of it. Inexperienced engineers may lack the expertise to fully grasp all the consequences of their actions. Moreover, experienced editors are likely to work with ontologies they do not know well and might, therefore, be unaware of the effect of changes. We argue that editors need to be made aware of the effect of their changes using multiple perspectives, including the change’s semantic and structural consequences while they are changing the ontology. In this article, we propose ChImp (Change Impact), a Protégé [18] plug-in to display information related to the changes. ChImp’s goal is to increase the understanding of ontology changes, making editors more aware of consequences and impact of their changes. To collect requirements for ChImp, we conducted a survey asking ontology editors about ontology evolution and its consequences. The survey contained mock-ups of change visualizations for rating, opportunities to provide detailed explanations of preferences, and general questions about demographics, already established practices, and the topic itself. Based on the responses we received, we built a plug-in offering three perspectives on changes: a summary of the performed changes, the consistency of the ontology, and changes to ontology measures (such as number of classes or properties). Therefore, our contributions are: (1) a list of requirements for visualizing and informing about changes and (2) the ChImp plug-in for Protégé providing real-time information to the ontology engineer about the applied changes. In the next section, we present related research. Section 3 introduces the survey, shows its results, and summarizes the requirements for the plug-in. In Section 4, we present our implementation in detail. We address conclusions and future work in the last section. 2 Related Research This section addresses two aspects of the related research. The first is research on the impact of ontology evolution, which is fairly young. As artefact evolution is a known phenomenon, we take a step further and address research on the con- sequences of their evolution. Second, we discuss visualizations of ontologies with a focus on change visualization. We also acknowledge the many published Pro- tégé plug-ins dealing with visualization of differences between ontology versions. However, none of them focus on the visualization of changes applied during a users current Protégé session. 48 ChImp: Visualizing Ontology Changes and their Impact in Protégé Evolution Impact. Noy and Klein [19] already made clear that ontology evo- lution is not the same as database schema evolution. They point out that evo- lution’s consequences are difficult to foresee because of the decentralisation of ontologies. Gonçalves et al. [10] propose a categorization of changes based on a logical impact. They investigate whether changes affect the set of entailed axioms in the next version and distinguish between effectual and ineffectual changes. Gross et al. [12] examine how changes in an ontology impact previ- ously conducted functional analysis. Also Gottron and Gottron [11] investigate the impact of knowledge graph evolution using Linked Open Data: they im- plement twelve different indexing methods and evaluate how respective indices are affected by the evolution of the data using three different measures. Os- borne et al. [20] present the pragmatic ontology evolution, in which they analyse the selection of concepts for a new version by evaluating the performance of four different tasks. Pernischová et al. [22] investigate and predict the impact of knowledge graph evolution on embeddings by comparing neighbourhoods. Un- fortunately, most of these approaches are too complex and calculation-intensive to be implemented as an interactive Protégé plug-in. Analog to Pernischová et al. [22] using embeddings, we will calculate the materialization and, therefore, check the consistency of the ontology. The consistency status is considered the impact of changes and can be determined after each edit. Change Visualization. Katifori et al. [15] and Dudás et al. [4] are two im- portant contributions to ontology visualization. The former covers a range of ontology visualization methods and techniques; it discusses the strengths and weaknesses of each method and addresses the issue of visualizing time-related data. The latter presents the current state of the art in ontology visualization. It states that there is no de-facto standard of visualization which has been accepted by the Semantic Web community, due to the fact that there is no single solution fits all applications. Visualizations are not new in Protégé [18]. Nonetheless, none of the available plug-ins deal with the direct visualization of changes at editing time. Calcula- tion of impact is also a recent research, and no plugin has addressed it so far. We specifically focus on plugins that visualizes the changes. ChangeAnalysisTab, addition to the Change Management Plugin used in Falconer et al. [6], enables the exploration of changes and annotations using different aspects, such as au- thors or terms. The Logical Difference Visualizer (LogDiffViz) is noteworthy in terms of its capabilities in comparing ontology versions [8]. Unfortunately, this plug-in does not update visualization based on changes applied during the Pro- tégé session. One can only compare two ontologies and visualize the differences afterwards. Further, OWLDiff [16] does not provide much of a visualization at all. This practical tool serves the comparison and merging of ontologies rather than investigating changes. Lastly, Change View [3] provides a straightforward list of changes but does not visualize them beyond a simple grouping. 49 ChImp: Visualizing Ontology Changes and their Impact in Protégé 3 Requirements Survey The first step of the ChImp design process was the elicitation of requirements. While we defined the ones related to the behaviour of ChImp through an internal design process, we collected the requirements about the visualization through a survey. We first present our survey, focusing on three specific questions which we present below. Subsequently, we formulated the related requirements that drove the development of ChImp. Survey structure. We conducted a survey, which consists of four main sec- tions.3 The first section contains questions to collect demographic information. We use it to weigh the responses based on the self-declared expertise of the participants. The second section, titled “Changing an Ontology”, collects partic- ipants’ experience on editing ontologies. It asks questions about different change types, to determine which are the most common. It also inquiries about tools (plug-ins or visualizations) related to changes that participants may already use. This part of the survey collects participants’ preferences on the information they are interested in monitoring while developing ontologies. The third section, ti- tled “Mock-ups of a Prototype”, collects opinions on visualizing the changes and their impact. It presents mock-ups visualizing Boolean metrics (such as consis- tency), numerical values (impact of changes, primitive, and composite ontology measures), and categorical variables (e.g., change type). The last section col- lects feedback and provides a wrap-up. It asks participants about their interest and opinion on ontology evolution and tools to monitor it. Additionally, it in- quires about availability to participate in follow-up studies. The collection of the requirements mainly rely on three questions, one in the second section of the survey and two in the third one. Figure 1 shows the first question we analyze, which we label the helpfulness question. It investigates the degree of helpfulness (from not helpful to very help- ful) of individual features describing ontology changes and how to visualize them. The information about changes includes number and types of changes, a varia- tion of primitive measures (e.g., numbers of classes, properties, or annotations), composite measures (e.g., class to property ratio or the number of annotations per class), and consequences of change (e.g., ontology consistency). We propose two visualization styles: textual and graphical. As the names suggest, the former consists of descriptions, numbers and tables, while the latter includes plots and charts. We decided not to provide any visual aids to avoid driving participants towards specific types of plots or text. For each type of information and visual- ization style, the participants express its helpfulness using a drop-down menu. All the answers are mandatory to nudge participants to consider each option, in- stead of simply skipping certain ones. Among the possible answers, participants can pick "don’t care/know", to capture the cases where they do not have any opinion or interest in the metrics. 3 We have published a static version of the survey on our web site at https://files.ifi.uzh.ch/ddis/chimp/reqsurvey.pdf 50 ChImp: Visualizing Ontology Changes and their Impact in Protégé Fig. 1: Last question from the second part of the survey “Changing an Ontology.” Each drop-down shows the same selection options and each requires an answer. The second question is the mock-up question, where participants observed five mock-ups. Figure 2 shows five mock-ups, each showing a different aspect, such as impact, consistency, changes, and measures. The participants judge each mock-up with a score from 1 (not at all informative) to 5 (very informative). The participants could also choose to not assign any score. Participants Demographics. We invited semantic web practitioners to an- swer the survey. We distributed the survey among the authors’ contacts and asked them to share it with colleagues who edit ontologies. 20 people signed up, out of which 12 completed the survey. The remaining eight did not complete the survey. The average age is 38.33 with standard deviation (sd) 7.1. Participants claim to have worked with ontologies for 10 years on average (sd: 5.29). 42% of partici- pants are working on professorial level, 33% on PhD and Post-Doc level, and the remaining 25% on other research positions, most of which in industry. Only one participant works in engineering, the others in research, and they all participants either still change ontologies regularly or did it in the past. Ten out of twelve participants have used Protégé to change ontologies. When asking about previously used tools and measures to communicate or visualize changes, only a few participants answered. Specifically, one participant specified that they add an informal description of what has been changed in the README document when releasing a new ontology version. Others mentioned using Protégé to check consistency and other requirement compliance before a re- lease. One participant uses an UML-like graphical representation of the changes to inform users about the new version. Survey analysis. We report the results of the three questions—helpfulness, mock-up, and plot—in Figure 3. Figure 3a shows the mean rating for the help- fulness question. We assign values between 0 and 4 to the answers of this ques- 51 ChImp: Visualizing Ontology Changes and their Impact in Protégé All changes Modified: ex:patient rdfs:subclassOf ex:patientRole Added: ex:patientRole rdf:type ex:role (a) List of changes Metrics Select impact measure Primitive Graph Distance i Axioms � 26 486 +31 0.5384 i Classes 6� 256 −12 I(Mi )−I(Mi+1 ) 2 i Subclasses i −( ) 6� 863 +29 1−e |δi | i Object Properties 54 −3 with I(M ) being a topological index: i Datatype Properties 15 +1 � I(M ) = �1 ... ... u∈V (M ) d(u) V (M ) are all nodes in M and Composite d(u) is the degree of node u i Property Class Ratio 0.01 −0.0002 (d) Impact measure [1] i Inheritance Richness 1.1 +0.008 i Attribute Richness 0.002 +0.0001 i Average Population 0 ... ... (b) Table of measures Check consistency Ontology is NOT consistent (c) Consistency (e) Plots Fig. 2: Mock-ups used in the requirements survey. Participants rated each mock- up from 1 (not at all informative) to 5 (very informative). tion. Values 1 to 4 map to the answers from “not helpful” to “very helpful”, while the value 0 maps to “don’t know/care”. The information about impact/- consequences received the highest ratings, followed by the one about the number and types of changes. For these information types, participants prefer textual summaries more than visualizations. We observe the opposite situation for the primitive and composite measures; in these cases, participants prefer visualiza- tion over textual summaries. Due to the participants’ preference for the number and types of changes, we formulate the following requirement: [R1] ChImp should list the applied changes. ChImp will report all changes 52 ChImp: Visualizing Ontology Changes and their Impact in Protégé Textual Summary List of changes Graphical Visualization Consistency Some impact/consequences on the applied changes Impact measure Number and types of changes Change in primitive measures Table of measures Change in composite measures Plot of measures 0 1 2 3 4 0 1 2 3 4 5 (a) Results of the helpfulness ques- (b) Results of the mock-ups question tion. Participants rated the options where mock-ups of different visualiza- from "very helpful (4)" to "not helpful tions were presented to the partici- (1)" along two axis: textual summary pants. They rated them from "very and graphical visualization. The option informative" (5) to "not informative" "Do not know/care" was rated as 0. (1). Fig. 3: Results of the three questions on helpfulness, mock-ups and plots of the requirements survey showing average rating. applied during the current session, highlighting the most recent one. Protégé changes should be grouped based on the action the user has taken, e.g., deleting a class. In the background, Protégé might execute more changes triggered by the action of the user. Such subsequent changes should be displayed and grouped with the action of the user. To decide on further requirements about displaying impact/consequences as well as primitive and composite measures, we consider the results of the mock- up question as well. The ratings of the answers in the mock-up question already range from 1 to 5. We do not consider the empty answers in these questions. Fig- ure 3b shows the answers to the mock-up question. We observe that participants prefer the answers: consistency, list of changes, and table with measures. R1 already covers the list of changes. Thus, we derive a second requirement about consistency: [R2] ChImp should inform the user about the consistency of the loaded ontology. The consistency needs to be checked after every change. The rating of the impact mock-up is the lowest. We believe that this is because there is not much research on ontology evolution impact. In the helpfulness question, we ask the participants about two specific op- tions: a table showing numbers and plots of the metrics. The participants clearly preferred a graphical visualization. However, while answering the mock-up ques- tion, participants do not perceive the plots showing ontology measures as in- formative and favor the tabular visualization. Seamingly, the results from the helpfulness and mock-up questions contradict each other. We assume this to be due to lack of contextual information when displaying the plot mock-ups. We have decided to only use a table to visualize the change in numbers since this answer was rated higher in the mock-up question of the survey: 53 ChImp: Visualizing Ontology Changes and their Impact in Protégé [R3] ChImp should show primitive and composite measures in a table, visualiz- ing the new value and its difference to the old value based on the applied changes. Section 4 discusses the implemented measures in more detail. Two participants commented on the choice of colours in the table mock-up. They pointed out that colours are suitable for awareness, but the choice of colour is essential. Therefore, we formulate the following requirement: [R4] ChImp should use colors to indicate changes. However, the choice of colour should not imply additional meaning, e.g., "good" or "bad". Also to accommo- date colorblindness, we will avoid colours like red and green. In general, all participants found that the topic of impact and consequences of ontology editing is important, yet there is no universal way of representing consequences. Impact can be very domain-dependent. In the last part of the survey, participants elaborated on this with detailed comments. With biomedical ontologies, engineers might be interested in the impact on the class hierarchy. Where prediction models make use of ontologies, the engineer might want to know when the model needs to be re-learned because the ontology has changed significantly and thus would produce inaccurate predictions. Some participants find ontology consistency to be sufficient where others suggested to include the number of other ontologies and systems which will be (specifically) affected by the changes. In the open questions, survey participants also suggest adding change logs into versioning systems. These logs could also include changes in primitive mea- sures as well as some indication of impact. We therefore formulate the following requirements: [R5] Ontology release notes should include the number and types of changes. These can include the number of additions and deletions of axioms and an- notations. They could also be more specific and indicate additions of classes or hierarchy changes. Further, ontology measures such as the number of classes, properties, annotations, or individuals could be reported together with the num- ber of changes. Release notes should also include impact or consequences: [R6] Ontology release notes should include the result of a consistency check. [R7] Ontology release notes should report changes to the materialization, as in- dication of consequences. As we explain in Section 4, R5, R6 and R7 are not available in the current version of ChImp, but they will be addressed in future work. Other requirements. We also formulate requirements regarding users’ inter- actions with ChImp as well as its responsiveness. They are based on the authors’ experience and best practices. [R8] ChImp should allow the user to chose between the presentation of metrics either in absolute values or as percentages. Engineers have different preferences in the presentation of numbers. Using percentages has advantages, just like ab- solute numbers do as well. The particular ontology size can also influence this preference. Therefore, we want to leave the choice of presentation of numbers up to the users. 54 ChImp: Visualizing Ontology Changes and their Impact in Protégé [R9] ChImp should let the user choose between using only the last change or all changes for the calculation of primitive and composite measures. While it makes sense to display impact measures cumulatively, we see the potential for both types of calculations regarding ontology measures. The user should be able to make this choice on the fly. While engineers change the ontology, ChImp executes many calculations in the background. We need to ensure that ChImp does not block Protégé while calculating and waiting to display new numbers. Responsiveness is essential for good user experience, particularly when working with large ontologies. Therefore, we capture the following requirement: [R10] ChImp should be responsive, and should not block usage of Protégé while calculating the inference, consistency, or measures. Using Protégé and OWL native calls will help in achieving this requirement. 4 The ChImp Plug-in ChImp is distributed under the Apache 2.0 license an can be be downloaded from the project web site.4 We aimed to leverage existing code and libraries, following good software engineering practices. Where possible, we based our calculations on already available methods from Protégé [18]. 4.1 Plug-in Implementation ChImp is a Protégé plug-in implemented as a view component, which is a build- ing block for workspace tabs. Any tab can display ChImp, which provides with three views: Change Display, Impact Display, and Metrics Table, as shown in Figure 4. The Change Display is split into two parts: the last change and previous changes. The former, situated right at the top, reports the most recent change, e.g., deletion of a class, and all the dependent automatic changes executed by Protégé, e.g., deletion of type axioms for individuals of the removed class. The latter, below, lists all the previous changes performed in the current session. The grouping is retained the same as within the last change part. When the engineer applies a new change, the last change is updated and the former one is pushed into the list of previous changes. This display acts as a stack and addresses R1. To address R2, ChImp reports the consistency status in the Impact Dis- play. ChImp uses the internal reasoner, in this case HermiT [9], to check for consistency. Consistency is not checked automatically, but has to be synchro- nized using Protégé’s reasoning menu. This display will also alert the user if the reasoner has not been started. The Metrics Table has two parts, primitive and composite metrics, to ad- dress R3. The current version of ChImp shows the metrics listed in Table 1. The top section of the table explains primitive measures. For each of them, Ta- ble 1 reports the Protégé methods we used to retrieve respective values. The 4 https://gitlab.ifi.uzh.ch/DDIS-Public/chimp-protege-plugin 55 ChImp: Visualizing Ontology Changes and their Impact in Protégé (a) ChImp after initialization. (b) ChImp after starting the reasoner and executing some changes. Fig. 4: Screenshots of ChImp, loaded in the ontology overview tab. composite measures, in the bottom part, are combinations of primitive measures and capture structural aspects of the ontology. We opted for only five measures since some participants commented that more metrics do not necessarily provide additional information. ChImp uses colors to display the number and the delta when a measure is affected by the changes and, therefore, satisfies R4. To fulfil R8, the user can chose to display the change in metrics using absolute numbers or percentages also through a drop-down menu. Moreover, the user can access either the last change or all changes, according to R9. The former only shows the difference in metrics for the last change, while the latter is cumulative and displays the changes in metrics since the start of the session. 4.2 Architecture The plug-in is implemented for and based on Protégé 5.1. The application con- sists of two main parts: The display panels and the calculation logic of the met- rics. All metrics—primitive, composite, and impact—extend the abstract Metric class that enforces the implementation of the method calculateMetric(). This abstraction functions as a strategy and enables an implementation-independent interaction with the individual metrics. Additionally, it implicitly enforces pri- vate fields by requiring constructor arguments that define the name and descrip- tion of the metric. Figure 5 shows a diagram of the application. 56 ChImp: Visualizing Ontology Changes and their Impact in Protégé Description Implementation c number of classes o.getClassesInSignature().size() i number of individuals o.getIndividualsInSignature().size() o.getObjectPropertiesInSignature().size() p number of properties + o.getDataPropertiesInSignature().size() h number of subclasses o.getAxioms(AxiomType.SUBCLASS_OF).size() a number of annotations o.getAnnotations().size() o.getAxioms(AxiomType. INVERSE_FUNCTIONAL_OBJECT_PROPERTY).size() inv number of inverse + o.getAxioms(AxiomType. relations INVERSE_OBJECT_PROPERTIES).size() average population i/c [5, 7, 23] inheritance richness h/c [2, 17, 23] annotation richness a/c [5, 23] property class ratio p/c [5, 24, 7, 23] inverse property ratio inv/p [2, 7, 23] Table 1: Description and implementation of the primitive and composite metrics in ChImp. o is the instance of the ontology within each metric implementation. Fig. 5: UML Class diagram of the ChImp implementation, showing its architec- ture. The main class is ChimpPlugin. It extends AbstractOWLViewComponent so that it can be displayed as a view component in the Protégé editor. This rela- tionship allows access to the OWLModelManager and the internal ontology. Three individual classes, one for each panel, implement the user interface. The LastChangePanel holds all implementation of displaying and managing the change stack. The ImpactMetricsPanel takes care of the reasoner and consistency checking. If impact metrics were available, this panel would in- stantiate and display them. Primitive and composite metrics hold all informa- 57 ChImp: Visualizing Ontology Changes and their Impact in Protégé tion in their respective implementation of the metric interface. Therefore, the StandardMetricsPanel only needs to create instances of the metrics for the dis- play. As the ontology is changed, each panel within the interface is updated by using a change listener on the OWLModelManager. This means that the plug-in can react to all change events fired by the main Protégé editor. Within the class ChimpPlugin, there is also a change listener that listens to changing ontologies. It is configured to reload the plug-in if the user switches the ontology. The OWLModelManager also enables access to a reasoner if one is loaded in the Protégé editor. All reasoner plug-ins implement the OWLReasoner interface provided by the OWL API [14], and can therefore be used interchangeably. How- ever, since their individual implementations and capabilities differ widely, they vary regarding results as well as performance. The impact panel leverages such a reasoner if it is available. Currently, it displays whether or not the underlying ontology is consistent. In the future, impact measures would also make use of the reasoner. For ChImp to track changes, the user needs to load the plug-in into Protégé and open the view once. After that, it starts recording the applied changes and displaying the changes, even if it is not in focus or visible. Protégé’s change listener is used for this purpose. 5 Conclusions and Future Work This work aims at closing the gap between ontology engineers and the informa- tion about the changes they apply to an ontology. With a requirements survey, we asked practitioners about their opinion and preferences on visualizing changes within Protégé. We formulated ten requirements and were able to implement six of them directly. The ChImp plug-in is the result of this implementation. R10 addresses responsiveness. Even though, we used only Protégé-native calls and did not make use of additional libraries, responsiveness requires a separate evalua- tion. As future work, we will evaluate our implementation in a test environment with different ontologies and various change types and sizes. R5 through R7 require ontology release notes to include information such as number of changes, consistency, and ontology measures. We plan to extend ChImp with a feature which allows exporting this information when saving the ontology for release purposes. There are various extensions already planned for ChImp. In the future, we would like to give the engineer the option to select the metrics that are displayed in the Metrics Table and also offer more quality- and semantic-oriented measures. In addition, listed changes are currently based on the change listener within Protégé. The inclusion of more complex aggregation and classification of changes, e.g., COnto-Diff [13], is one possible extension. We will investigate the possibility of impact measures using the materialization but also other applications like embeddings or functional enrichment analysis [21]. Depending on the application, impact measures can either be computed directly, approximated, or learnt and 58 ChImp: Visualizing Ontology Changes and their Impact in Protégé then predicted. The latter case would entail the loading of a learnt model for impact prediction [21]. Lastly, we are planning to use ChImp conduct a detailed user study. We will investigate the awareness of engineers concerning the consequences of changes. Participants will be asked to solve specific tasks, during which they will change an ontology and either have access to ChImp or not. With a post-task survey, we will assess their awareness and compare their editing behaviour in detail. ChImp will enable studies about awareness of consequences of ontology changes. Additionally, we can potentially improve the ontology evolution processes and make both the editing of and transitioning to new versions of ontologies eas- ier and more accessible for ontology users. In the long run, we hope that these kinds of studies will lead to insights that have a comparable impact on the field as software evolution research had on software engineering. References 1. Dehmer, M., Emmert-Streib, F., Shi, Y.: Interrelations of Graph Distance Mea- sures Based on Topological Indices. PLOS ONE 9(4), e94985 (Apr 2014). https://doi.org/10.1371/journal.pone.0094985 2. Djedidi, R., Aufaure, M.A.: ONTO-EVO A L an ontology evolution approach guided by pattern modeling and quality evaluation. In: International symposium on foundations of information and knowledge systems. pp. 286–305 (2010) 3. Drummond, N.: ChangeView (Mar 2011), https://code.google.com/archive/p/co- ode-owl-plugins/wikis/ChangeView.wiki 4. Dudás, M., Lohmann, S., Svátek, V., Pavlov, D.: Ontology visualization methods and tools: a survey of the state of the art. Knowledge Eng. Review 33, e10 (2018) 5. Duque-Ramos, A., Fernández-Breis, J.T., Iniesta, M., Dumontier, M., Aranguren, M.E., Schulz, S., Aussenac-Gilles, N., Stevens, R.: Evaluation of the OQuaRE framework for ontology quality. Expert Systems with Applications 40(7), 2696– 2703 (2013) 6. Falconer, S.M., Tudorache, T., Noy, N.F.: An analysis of collaborative patterns in large-scale ontology development projects. In: K-cap. pp. 25–32. ACM (2011) 7. Gangemi, A., Catenacci, C., Ciaramita, M., Lehmann, J.: Modelling ontology eval- uation and validation. In: European semantic web conference. pp. 140–154 (2006) 8. Gatens, W., Konev, B., Ludwig, M., Wolter, F.: Versioning based on logical differ- ence for lightweight description logic terminologies. Proc. of ARCOE-11 (2011) 9. Glimm, B., Horrocks, I., Motik, B., Stoilos, G., Wang, Z.: HermiT: an OWL 2 rea- soner. Journal of Automated Reasoning 53(3), 245–269 (2014), publisher: Springer 10. Gonçalves, R.S., Parsia, B., Sattler, U.: Categorising logical differences between OWL ontologies. In: CIKM. pp. 1541–1546. ACM (2011) 11. Gottron, T., Gottron, C.: Perplexity of Index Models over Evolving Linked Data. In: ESWC. vol. 8465, pp. 161–175. Springer (2014) 12. Gross, A., Hartung, M., Prüfer, K., Kelso, J., Rahm, E.: Impact of ontology evo- lution on functional analyses. Bioinformatics 28(20), 2671–2677 (2012) 13. Hartung, M., Gross, A., Rahm, E.: COnto-Diff: Generation of complex evolution mappings for life science ontologies. Journal of Biomedical Informatics 46(1), 15–32 (Feb 2013) 59 ChImp: Visualizing Ontology Changes and their Impact in Protégé 14. Horridge, M., Bechhofer, S.: The owl api: A java api for owl ontologies. Semantic Web 2(1), 11–21 (Jan 2011) 15. Katifori, A., Halatsis, C., Lepouras, G., Vassilakis, C., Giannopoulou, E.G.: Ontol- ogy visualization methods - a survey. ACM Computing Surveys 39(4), 10 (2007) 16. Kremen, P., Smid, M., Kouba, Z.: OWLDiff: A practical tool for comparison and merge of OWL ontologies. In: DEXA workshops. pp. 229–233. IEEE Computer Society (2011) 17. Lantow, B., Sandkuhl, K.: An analysis of applicability using quality metrics for ontologies on ontology design patterns. Intelligent Systems in Accounting, Finance and Management 22(1), 81–99 (2015) 18. Musen, M.A.: The protégé project: a look back and a look forward. AI Matters 1(4), 4–12 (2015) 19. Noy, N.F., Klein, M.C.A.: Ontology evolution: Not the same as schema evolution. Knowl. Inf. Syst. 6(4), 428–440 (2004) 20. Osborne, F., Motta, E.: Pragmatic Ontology Evolution: Reconciling User Require- ments and Application Performance. In: ISWC. LNCS, vol. 11136, pp. 495–512. Springer (2018) 21. Pernischová, R.: The Butterfly Effect in Knowledge Graphs: Predicting the Impact of Changes in the Evolving Web of Data. In: ISWC: Doctoral Consortium. CEUR- WS.org, Aukland, NZ (Oct 2019) 22. Pernischová, R., Dell’Aglio, D., Horridge, M., Baumgartner, M., Bernstein, A.: Toward predicting impact of changes in evolving knowledge graphs. In: ISWC satellites. CEUR workshop proceedings, vol. 2456, pp. 137–140. CEUR-WS.org, Aukland, NZ (Oct 2019) 23. Tartir, S., Arpinar, I.B., Sheth, A.P.: Ontological evaluation and validation. In: Theory and applications of ontology: Computer applications, pp. 115–130. Springer (2010) 24. Tempich, C., Volz, R.: Towards a benchmark for Semantic Web reasoners-an anal- ysis of the DAML ontology library. In: EON. vol. 87 (2003) 60