Investigating tool usage in ontology engineering: a survey Clarete A. Diniz Gomes1 , Glaice Kelly Q. Monfardini1,2 , Jordana S. Salamon1 , Rosiane da Silva Sangali3 , Isauflânia S. Ribeiro Timoteo1 , Monalessa P. Barcellos1 and Vítor E. Silva Souza1,* 1 Ontology & Conceptual Modeling Research Group (NEMO), Federal University of Espírito Santo (UFES), Brazil 2 Federal Institute of Espírito Santo (IFES), Brazil 3 Optimization Laboratory (LabOtim), Federal University of Espírito Santo (UFES), Brazil Abstract The use of ontologies has been acknowledged as a successful approach to solving interoperability and knowledge- related problems. They have also been recognized as valuable for explainability in AI-based systems. Developing ontologies involves a lot of tacit knowledge and requires tool support. Several tools are available and can be used for different purposes to aid ontology engineering (OE) activities. The use of a suitable set of tools contributes to increasing development process performance and improving ontology quality. Aiming at getting evidence about the use of tools to support OE, we conducted a survey with 74 ontology engineers. We investigated which tools have been used, their main characteristics, benefits and difficulties perceived when using them, and properties considered important in OE tools. The results indicate that tools have been widely adopted, but some issues, such as integration problems and scope limitations, still need to be addressed. This paper presents the survey, its results, and discusses the main findings. Keywords Tools, Ontology, Ontology Engineering, Survey 1. Introduction Ontologies allow the formal representation of fundamental concepts interpreted by users, in a way that facilitates communication and understanding of a given domain [1]. They have been recognized as conceptual tools of great importance in Computer Science since the end of the 1960s. In the last fifteen years, there has been an increase in works related to ontologies in various segments of Computer Science [2], mainly in areas such as Data Modeling (conceptual modeling) [3, 4] and Artificial Intelligence [5, 6], and ontologies have become the predominant way to deal with semantic aspects in semantic integration initiatives [7]. Developing ontologies is not trivial. Several ontology engineering (OE) methods have been proposed to aid ontology engineers in developing high-quality ontologies (e.g., [8, 9, 10, 11, 12, 13]). They decompose OE into other processes (or phases) and recommend activities for each one. Although there are differences among OE methods, in general, they consider that OE core activities involve defining the ontology requirements, capturing and developing the ontology conceptualization, implementing the ontology, and evaluating it. OE also relies on tools. They emerged as a form of automation to optimize and accelerate the ontology development process [14] and are crucial to formalizing ontological conceptual models, disseminating knowledge, implementing machine-readable ontologies, and ensuring ontology quality [15]. Currently, there is a wide range of tools to support OE. Some are devoted to OE activities (e.g., Proceedings of the 17th Seminar on Ontology Research in Brazil (ONTOBRAS 2024) and 8th Doctoral and Masters Consortium on Ontologies (WTDO 2024), Vitória, Brazil, October 07-10, 2024. * Corresponding author. $ clarete.gomes@edu.ufes.br (C. A. D. Gomes); monalessa@inf.ufes.br (M. P. Barcellos); vitor.souza@ufes.br (V. E. S. Souza)  0009-0004-1657-7557 (C. A. D. Gomes); 0000000215596201 (G. K. Q. Monfardini); 0000000232095273 (J. S. Salamon); 0009-0003-9618-2039 (R. d. S. Sangali); 0009-0001-4415-6264 (I. S. R. Timoteo); 0000000262259478 (M. P. Barcellos); 0000-0003-1869-5704 (V. E. S. Souza) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Protégé1 ), while others were not conceived for OE but can be used in this context (e.g., Visual Paradigm2 ). Selecting the more suitable tools that provide the necessary features is an important decision in the OE process and it may not be easy [16]. Several factors should be taken into account, such as ontology scope, team characteristics (e.g., size, experience), organization context (e.g., policies, preferences), and ontology’s intended use. Considering the tool characteristics is also relevant (e.g., availability, features). Some OE methods help tool selection by suggesting some to support their activities [17, 13]. Although there has been an evolution in the functionalities of tools to support ontology development and some tools have become very popular (e.g., Protégé) [18], there is still a need to understand what (combination of) tools offer the best support for the OE process [19]. For that, it is necessary to understand which tools have been used, their characteristics, perceived benefits, and faced difficulties. To take a step in this direction and gather evidence on this subject, we carried out a survey with 74 ontology engineers to investigate tools used to support OE. The results provide a preliminary picture of the state of practice. With that, we intend to share data and perceptions with other ontology engineers and shine a light on opportunities to advance the research topic. In summary, the results reinforce the importance of using tools to aid OE. Tools have been widely used, with a predominance of tools for conceptual model development and ontology implementation. Both, OE-specific (e.g., ontology editors) and generic (e.g., text editors) tools have been used and there has been a preference for free tools. Functional suitability, reliability, usability, and availability have been considered the more relevant properties. Ontology development acceleration and ontology quality improvement have been perceived as the main benefits, corroborating the literature [20, 21, 4]. On the other hand, integration problems and scope limitations have been the main obstacles to overcome. In this paper, we provide an overview of our study, summarize the main findings, and discuss the results. The remainder of the paper is organized as follows: Section 2 presents the theoretical background for the paper; Section 3 presents the study protocol; Section 4 synthesizes the results; Section 5 discusses the results; Section 6 addresses the study limitations; Section 7 discusses related works; and Section 8 presents our final considerations. 2. Background An ontology is a formal and explicit specification of a shared conceptualization [22]. The conceptu- alization is an abstract and simplified view of the portion of the world intended to be represented. An important distinction sets apart ontologies as conceptual models, called reference ontologies, from ontologies as computational artifacts, called operational ontologies [23]. A reference ontology is built aiming at making the best possible description of the domain in reality, representing a model of con- sensus within a community, regardless of its computational properties. Operational ontologies, in turn, are machine-readable artifacts designed with the focus on guaranteeing desirable computational properties [9]. Engineering ontologies is a complex and collaborative task, which is challenging for both newcomer and experienced engineers [24, 2]. It requires adequate methodological approaches and appropriate tooling support [25]. The literature presents several OE methods to help in this matter (e.g., Methon- tology [26], Ontology development 101 [27], eXtreme Design [8], SAMOD [10], LOT [13]). In general, developing ontologies involves management, development, and support activities. The first covers the organizational setup of the overall process (e.g., managing resources, controlling the project schedule, and the quality of the produced artifacts). The second refers to the ontology development itself and includes activities such as ontology specification, conceptualization, formalization, and implementa- tion. The third involves activities related to knowledge acquisition, documentation, and configuration management, among others, which are carried out in parallel with development activities to support them [28]. 1 https://protege.stanford.edu/ 2 https://www.visual-paradigm.com/ Several tools have been developed specifically to support ontology development, such as Protégé, OLED [29], and Onto4AllEditor [4]. Given the similarity of some OE and Software Engineering (SE) activities (e.g., conceptual modeling), SE supporting tools, such as Astah3 and Visual Paradigm, have also been used to aid ontology development tasks. Such tools can be used off-the-shelf or adapted to suit OE needs through plugins, like the OntoUML Visual Paradigm plugin presented in [30]. According to Gómez-Pérez et al. [31] OE supporting tools can be grouped in six categories: develop- ment tools, including tools and integrated suites used to build a new ontology from scratch (usually support documentation, export and import to/from different formats and languages, graphical edition, library management); evaluation tools, used to evaluate the ontologies and their related technologies; merge and alignment tools support merging and aligning different ontologies in the same domain; annotation tools help add instances of concepts and relations and maintain (semi)automatically ontology- based markups in Web pages; querying tools and inference engines, allowing querying and performing inferences; and learning tools, which derive ontologies (semi)automatically from natural language texts, as well as semi-structured sources and databases, by means of machine learning and natural language analysis techniques. 3. Study Design The study consisted of a survey whose goal was to investigate the use of tools in OE to get an overview of this topic in practice. A survey aims to identify the characteristics of a broad population by generalizing data collected from a representative sample of individuals [32]. Surveys are conducted to produce a snapshot of the situation to capture the current status [33]. We chose this method to reach many individuals, which would be harder or unfeasible through interviews or case studies, for example. We followed the process defined in [33], which comprises five activities: scoping, when we delimit the scope of the study and establish its goals; planning, when the study design is determined; operation, which consists of data collection; analysis and interpretation, which involves analyzing data to draw conclusions about the research topic; and presentation and package, when the results are communicated. Aligned with the study goal, we defined three main research questions: (RQ1) Which tools have been used and what are their characteristics?; (RQ2) What benefits and difficulties have been perceived when using the tools?; and (RQ3) Which properties have been considered important in OE tools?. The instrument used in the study was a form created in Google Forms. It is composed of a consent term for participation in the study and three sections of questions. The first section has eight closed questions to characterize the participants. The second has 11 closed questions related to RQ1 and RQ2. One question allows the participant to complement their answer by providing information in text format. The third section is related to RQ3 and has one closed question plus two open questions to collect further information, comments, and suggestions. In some questions we used ordinal agreement scales but excluded the neutral option to capture meaningful information. The form is available in the study package [34]. The survey participants must be a sample of the target population. Therefore, we targeted ontology engineers with knowledge of and experience with ontology development and OE tooling support. The procedure followed in the study consisted of three steps. In the first step, we carried out a small pilot to evaluate the form and the study protocol. We asked an ontology engineer to answer the questionnaire and report response time, problems, and suggestions. Taking his feedback into account, we made minor adjustments to the form. In the second step, we sent messages inviting people to take part in the study. We sent messages to research groups working with ontologies, mailing lists involving researchers and professionals in the field, contact networks at universities, public and private organizations, and authors of the articles selected in an ongoing systematic literature mapping we are performing on the topic. We also requested that those invited forward the invitation to other individuals whom they believed could participate in the study. The final step involved collecting data from the answered questionnaires, representing collected data in tables and graphs, and analyzing it. 3 https://astah.net 4. Study Execution and Data Synthesis Invitations to participate in the study were sent in two batches, the first in late December 2023 and the second in late January 2024. We contacted 227 people and received 74 answers until March 1st, 2023, which amounts to a response rate of 32.6%. In this section, we summarize data collected in the study. For questions in which the participants could choose more than one answer, the sum of the absolute values is higher than 74 and, thus, the sum of the rates exceeds 100%. For simplification reasons, we rounded the percentage values to the first decimal place. The complete set of collected data plus tables and graphs representing them is available in the study package [34]. The participants’ profile was identified through questions regarding their level and area of education, knowledge of ontologies, and experience in OE. To better contextualize the results, we also asked about the context in which they have worked with ontologies (academia or productive sector), the ontology types they have developed (reference or operational), and how often they have used OE systematic approaches (e.g., SABiO [9], NeOn [11], Methontology [26]). Most participants are Brazilians (50; 67.6%). Others are from Spain (8; 10.8%), Russia (4; 5.4%), Netherlands (3; 4.1%), Argentina (3; 4.1%), France (2; 2.7%), United Kingdom (1; 1.4%), South Africa (1; 1.4%), and Belarus (1; 1.4%). One participant did not provide this information. Most participants have a background in IT, including Computer Science (73%), Computer Engineering (8.1%), Information Systems (10.8%), and Information Science (6.8%) plus one respondent who has his highest level of training in Social Work, but who also has a degree in Computer Science. Most of the participants work in the academic context (70.3%), have worked with OE for five or more years (62.2%), and have high experience in OE (58.1%). 20.3% of the participants have developed only operational ontologies, 32.4% only reference ontologies, and 47.3% both. Most of the participants (60.8%) often use OE systematic approaches and 21.6% always use them. Figure 1 illustrates data about the participants. Figure 1: Participants’ profile. In the following, we synthesize collected data by grouping the questionnaire questions into four topics: Use of tools in OE, OE tool characteristics, Perceptions of using OE tools, and Properties of OE tools. The first two topics are related to RQ1, the third to RQ2, and the last one relates to RQ3. Use of tools in OE: In this topic, we asked which tools the participants have used, how often they have used them, how they found the tools, and in which OE activities they have used OE-specific (i.e., tools addressing OE tasks, such as ontology editors) and generic (i.e., domain-independent tools that address general tasks, such as text editors) tools. When asked about how often they use tools to support OE, the majority of participants (55.4%) informed that they always use tools, 39.2% indicated they use tools most of the time, and 5.4% rarely use them. Concerning the used tools, OE-specific tools have been predominant. Protégé (85.1%) has been the most used tool, followed by Visual Paradigm (43.2%), Astah (41.9%), and OntoUML-vp-plugin (36.5%). Protégé is a free, open-source platform that provides a graphical interface for developing ontologies and an architecture with a suite of tools to construct domain models and knowledge-based applications with ontologies [35]. Visual Paradigm and Astah are tools that support creating UML conceptual models and generate code from the models. OntoUML-vp-plugin,4 in turn, is a plugin designed for OntoUML modelers that adds features to Visual Paradigm to enable the use of OntoUML stereotypes in class diagrams, model checking, and transformation. Other OE-specific tools used by the participants have been OLED [29] (20.3%) and Menthor [36] (16.2%). Both are OntoUML editors that support model building and formal verification, incorporating support for a range of methodological resources for developing, evaluating, and implementing ontologies. Generic tools have also been used, mainly Electronic Spreadsheets (27%), Text Editors (27%), and Slide/Presentation Tools (24.3%). In addition to the tools listed in the figure, others were mentioned by the participants, such as OntoTrack, Onto4AllEditor, Diagram.io, Miro, Widoco, Astrea, OnToology, Hermit, Themis, CmapTools, ONTOFOX, Enterprise Architect, PlantUML, Coggle.it, and yEd Graph Editor. As for how the participants discovered the tools, most of them (82.1%) received recommendations from other people (e.g., friends, work/study colleagues, teachers). 39.2% discovered the tools in courses or classes; 37.8% by searching the Internet; 36.5% have used tools employed where they work, 35.1% have used tools developed by the organization they work; and 17.6% declared to have used tools indicated in OE systematic approaches. To investigate tooling support provided to ontology development phases, we asked the participants to indicate the phases that have been supported by OE tools and the type of used tools (OE-specific or generic). Requirements Specification has been supported mainly by generic tools (89.2% of the participants indicated that have used that kind of tool in this phase). The other phases have had a predominance of OE-specific tools: 64.9% in Ontology Conceptualization and Formalization; 71.2% in Ontology Design, 73% in Ontology Implementation, and 58.1% in Verification, Validation, and Testing. Figure 2 summarizes data about the use of tools in OE. Figure 2: Use of Tools in OE. 4 https://github.com/OntoUML/ontouml-vp-plugin OE tools characteristics: In this topic, we asked the participants about the tools’ availability, features, and integrated support for OE. Figure 3 summarizes the results. Regarding availability, most participants (63.5%) declared that they have used only free tools, 33.8% informed that some of the tools they have used are free, and 2.7% of participants have not used free tools. Concerning features, we provided a list of options for the participants to indicate which ones have been offered by the tools they have used. The features most indicated were ontology’s concept hierarchy representation (86.5%), ontology visualization (85.1%), diagram creation (82.4%), ontology verification (59.5%), operational ontology implementation (56.8%), and ontology validation (50%). We also asked the participants if the tools have provided integrated support for the OE process. 47.3% of participants declared that the tools partially provide integrated support; 41.9% informed that the tools do not provide integrated support, and 10.8% stated that the tools have provided integrated support for the OE process. Figure 3: Characteristics of the tools. Perceptions of using OE tools: In this topic, we asked the participants about their perceptions of using OE tools regarding how useful the tools have been, provided benefits, and faced difficulties. Concerning tools usefulness, 43.2% of the participants stated that they have helped a lot; 39.2% answered that they have helped a little; 16.2% declared that they have helped very little; and one participant answered that tools have not helped at all. The participants were asked to justify their answers. We grouped the answers with the same meaning and categorized them. Regarding the reasons why participants have considered tools useful, they highlighted that the tools have helped simplify developing complex ontologies (13.5%), evaluate the ontology (12.2%), automate repetitive tasks (10.8%), visualize and understand the ontology as a whole (8.1%), increase productivity (8.1%), communicate and interact with stakeholders (2.7%), manage knowledge (2.7%) and reuse (2.7%). As for the reasons why participants have not considered the tools useful, they pointed out that the tools are too “fragmented” and the entire OE process cannot be supported by a single tool or a set of integrated tools (29.7%). Additionally, there has been a lack of support, documentation, tutorials, or other educational resources to help them use the tools (17.6%), automation has been limited (5.4%), there have been performance problems (5.4%), lack of free access (1.4%), and complex user interface (1.4%). We also asked the participants to indicate the benefits and difficulties they have perceived when using the tools. As for benefits, they highlighted that the tools accelerate the ontology development process (90.5%) and improve the quality of the produced ontologies (85.1%). Facilitating reuse (66.2%) and Figure 4: Perceptions of using OE tools. Figure 5: Relevance of OE tools properties. improving work management (64.9%) have also been acknowledged, among others. As for difficulties, the ones more prominent were scope limitations (e.g., the tools support a limited set of OE activities or the provided functionalities have limitations) (62.2%) and integration problems (e.g., the tools do not allow integration with others, the result produced in one tool cannot be exported to another) (56.8%). Identifying and selecting the tools has been difficult for half of the participants. Figure 4 summarizes the participants perceptions. Properties of OE tool: To investigate which properties ontology engineers expect to find in an OE supporting tool, we presented 10 properties (Functional Suitability, Efficiency, Usability, Reliability, Security, Maintainability, Portability, Availability of use, Collaboration, and Coverage) and asked the participants to indicate the level of relevance of each of them. The properties are based on ISO 25010 [37] (we adapted some of them to better fit OE tools). Figure 5 presents the results. Functional Suitability, Reliability, Usability, and Availability of Use were considered by the participants as the most relevant properties. In contrast, Maintainability and Security were considered less relevant (the number of “neutral” plus “irrelevant” plus “very irrelevant” answers is higher than in the other properties). The participants were also asked to indicate other properties they consider important in a tool to support OE. 31 participants answered this question. We grouped the equivalent properties. The main cited properties were Interoperability, Documentation & Support, Version Control, and Reuse. 5. Discussion Next, we discuss the results by considering the topics identified in Section 4. Regarding the use of tools in OE, around 94.6% of the participants declared that have systematically used tools, which demonstrates the key role tools play in OE. The most used tools were Protégé (85.1%), Visual Paradigm (41.9%), and Astah (36.5%). They support implementing operational ontologies (Protégé) and creating ontology conceptual models (Visual Paradigm and Astah). These are the OE most specific activities and tooling support is crucial to perform them. Thus, it is not a surprise that the most used tools relate to these activities. By analyzing the ontology types, we noticed that 79.7% of the participants have developed reference ontologies and 67.6% operational ontologies. Although there has been a predominance of reference ontologies, the number of participants that have used Protégé (85.1%) is higher than the ones using Visual Paradigm or Astah (59.5% of the participants have used only one of them or both). We believe that this is because Protégé is free, very popular among ontology engineers, and supported by a strong community of academic, government, and corporate users. In contrast, although Astah and Visual Paradigm are popular, they have similar features, which may divide users (e.g., 23% of the participants have used Visual Paradigm while 17.6% have preferred Astah), and are not free (Astah has a student version and Visual Paradigm has a community version with fewer features than the paid version). OntoUML-vp- plugin was also cited by many participants (36.5%). It helps validate conceptual models created in Visual Paradigm and represented in OntoUML and generates code that implements the corresponding operational ontology. Thus, it provides integrated support to modeling and implementation activities and bridges the ontology conceptual model and its operational version. We believe that tools like this contribute to built-in quality and should be further explored by the OE community. Four participants (5.4%) declared that have used tools rarely. Given that developing ontologies without tool support can be unfeasible, we hypothesize that the participants may have thought that the question referred to OE-specific tools only. That was the first question about tools presented for the participants, thus they were not advised that both, generic and OE-specific tools, should be considered. By looking at the 10 most cited tools, we notice that six of them are specific tools supporting conceptual modeling and ontology implementation, while four are generic tools. By analyzing in which OE phases the different types of tools have been used, we notice that the more specific the task is, the more specific the tool needs to be. Hence, there has been a predominance of generic tools to support Requirements Specification activities. Given that requirements are usually recorded in text format, generic tools such as text editors and spreadsheets can be helpful. Ontology conceptualization and formalization, design, implementation, and evaluation, in turn, have depended mainly on OE-specific tools. However, the use of generic tools to support ontology conceptualization and formalization (e.g., record concepts definitions and axioms) has also been expressive. Although generic tools have been useful, it may be hard to map information recorded in such tools (e.g., text editors) to the ontology stored in OE-specific tools (e.g., ontology editors). Thus, advances in OE-specific tools to support OE initial phases and connect them to other tools would be welcome. By analyzing tool support for each OE phase disregarding the types of tools, we noticed that around 92% of the participants have used tools to support Requirements Specification, 86% to support Con- ceptualization and Formalization, 84% to support Design, 76% to aid in Implementation, and 66% in Verification, Validation, and Testing. It is possible to notice higher values in the three first phases when compared to the last two phases, even though tools are essential to implement operational ontologies. These results are probably because many participants (32.4%) have developed only reference ontologies and, thus, do not implement or test operational ontologies. There has not been a common structured source of tools (e.g., a catalog, a repository). Most ontology engineers (more than 80%) have discovered the tools mainly through recommendations from other people. This suggests a lack of knowledge of the panorama of existing tools, their main characteristics, features, strengths, weaknesses, and so on. Some ontology engineers (around 17%) have used tools indicated by OE systematic approaches. Given that around 80% of the participants declared that have used such approaches, this result suggests that OE methods have not suggested tools or have suggested limited or unavailable ones, which is in line with observations made in [9]. Concerning OE tool characteristics, free tools have been predominant. This result corroborates the literature, which states that free access to the source code and cost savings are among the main characteristics desired in software tools [38, 39, 40]. The features offered by the tools have been diverse. The six most cited features are directly related to OE core phases: diagram creation, concepts hierarchy representation, ontology visualization, ontology implementation, ontology verification, and validation. This demonstrates that the tools have been concerned with providing features to address core OE tasks. Other features have covered support activities, such as reuse, automatic documentation, and configuration management. Features related to requirements (competency questions definition and management) have been the ones less covered by the tools. This is consistent with the findings previously discussed, indicating a lack of support of OE-specific tools for requirements specification. Providing advances to support requirement specification activities and connecting the produced artifacts to outcomes of subsequent phases would be beneficial for the OE community. Only around 10% of the participants informed that the tools they have used provide integrated support to the OE process. This suggests that despite the range of existing OE tools, they still have not supported the ontology lifecycle in a comprehensive and integrated way. In fact, most OE tools focus on a few activities of the ontology lifecycle, mainly design and implementation [20]. Moreover, the tools often work in an isolated way or with little integration with others. Thus, ontology engineers run them individually, switching between different tools and putting effort into integrating results from different tools [41]. This makes OE activities cumbersome and time-consuming [42], hampering tool support efficiency. Even some widespread OE tools only cover some OE tasks and cannot be used in the sense of a fully-fledged integrated development environment (IDE) [43, 20]. Therefore, there is a lack of integrated suites of tools that address the entire OE process. Analogous to tools that bridge the ontology conceptual model and its operational version, it is important to produce solutions that offer integrated OE support environments or other alternatives (e.g., plugins, APIs) to facilitate the integration of existing tools. As for the perceptions of using OE tools, around 82% of the participants perceive OE tools as helpful. Although this result was in some way expected, almost half of these participants claimed that the tools help just a little. Moreover, around 18% of the participants indicated that the tools help very little or are not helpful. Participants declared that the main problems hindering tool usefulness have been related to integration, reinforcing the issue discussed above, and documentation. Without a proper integration among the several tools used to support OE activities and clear documentation and support to use the tools, the advantages of using them are hampered [42, 19]. On the other hand, the support of the tools in developing complex ontologies, visualizing and evaluating ontologies, as well as automating repetitive tasks have contributed to ontology engineers perceiving the tools as helpful. The reasons why ontology engineers perceive OE tools as helpful (or not) were declared by the participants in an open question. After that, they were asked about the benefits and difficulties of using OE tools. In general, the results were consistent with the reasons presented by them, providing an overview of the pros and cons of using OE tools. Accelerating the ontology development process and improving the quality of the produced ontologies were the benefits perceived by more participants. These benefits have also been advocated in the literature (e.g., [4, 20, 21]). Hence, the results reinforce them and provide evidence of tools usefulness. Although ontology engineers have several OE tools available, they still have faced difficulties when using them, mainly due to scope limitations and integration problems. Moreover, selecting the tools has been challenging and the tools have presented many bugs. These problems have also been pointed out in the literature (e.g., [19, 44, 45, 20, 43, 46, 42, 41]), revealing that despite the advances in OE tools, there is a need for further improvements. Concerning integration problems, which is a prominent issue revealed in the survey results, it is important to advance works aiming at creating a common workbench for OE that facilitates ontology development during the whole ontology life cycle, also including support to configuration management and ontology evolution [20]. Regarding properties of OE tools the four considered more relevant by the participants (Functional Suitability, Reliability, Usability, and Availability) show that ontology engineers expect tools with the capability of doing what they are set out to do (i.e., have the necessary functions to meet the target tasks and objectives [47]), producing correct and reliable results, communicating clearly with the users, and being easily accessed (e.g., free tools). In other words, the tools should correctly do what they are intended to do and it should be easy to access and use them. These properties are perceived mainly from the external perspective (i.e., when ontology engineers are the users of the tools) and are consistent with other results. For example, bugs, limited access, and usability problems were pointed out as difficulties faced by the participants when using the tools, indicating the need for improvements in reliability, availability, and usability and the relevance of such properties. Despite scope limitations having been pointed out by most of the participants as an obstacle to overcome when using the tools, Coverage was not among the top five properties. Security and Maintainability were the properties considered less relevant. Security refers to the ability of the tool to protect data and information. We believe that the participants perceived this property as less relevant because, in general, when developing ontologies, we do not record confidential or sensitive information in the tools. However, protecting the ontology models, concepts, instances, etc. is still necessary, which was recognized by 50% of the participants, who considered this property relevant or very relevant. Maintainability, in turn, concerns the internal perspective, i.e., from who develops the tool. Thus, its relevance would be perceived only by ontology engineers who interact with the tools by playing different roles, user (ontology engineer) and developer (software engineer). The results suggest that this is not the case for most of the participants. The list of properties we provided to the participants was not exhaustive. To capture other relevant properties, we asked them to cite others they consider relevant. Interoperability, Documentation & Support, Version Control, and (support for) Reuse stand out. All these properties are related to aspects pointed out by the participants in other questions, which reinforces their relevance. Integration problems, lack of documentation & support, and version control issues have been indicated as difficulties faced when using the tools. Reuse, in turn, helps accelerate the development process, which is the main benefit of using tools perceived by the participants. To properly meet ontology engineers expectations and needs, tools should manifest properties referring to desirable quality attributes. The survey results indicate that some of them have not been adequately addressed by existing tools. Therefore, there is an opportunity to explore how these properties should be manifested and develop strategies to achieve them. 6. Threats to Validity The validity of a study denotes the trustworthiness of the results. Every study has threats that should be addressed as much as possible and considered together with the results. We discuss threats using the classification presented in [33]. Threats related to the constructs involved in the study can affect the results and impact Construct Validity. The main threat is the participants misinterpreting the questions. To minimize this threat, after producing the questionnaire, we carried out an internal test in which one of the authors, who did not participate in the questionnaire creation, answered it and pointed out aspects to be improved. After improving the questionnaire, we ran a pilot with an ontology engineer external to our research group to evaluate the questionnaire and response time. Moreover, in the questionnaire, we provided examples and definitions for the participants to better understand how to answer the questions. Another threat is related to the scale used in some questions. Given that we did not provide any common grounds, different participants may have interpreted terms (e.g., a lot, a little) in different ways. Moreover, the results might have been different if we had not excluded the neutral alternative in some questions. The alternatives of answers provided in each question also represent a threat because they may not cover all the relevant alternatives. To address this threat, when defining the questions and the respective alternatives of answers, we considered results from the systematic literature mapping about tools we are carrying out and, in addition, we included “Others” as an alternative the participants could choose and provide further information. Still regarding construct validity, it is important to be aware that the results reflect the participants’ personal experiences, interpretations, and beliefs, and the answers embed subjectivity that could not be captured through the questionnaire. Concerning Internal Validity, which refers to the ability of a new study to repeat the behavior with the same participants and objects, the main threat is participants provide inaccurate answers because they think they could be evaluated, since we asked them to inform their email to participate in the study. To mitigate this threat, we informed participants that data would not be evaluated individually and the participants’ identities would not be revealed. Regarding External Validity, i.e., to what extent it is possible to generalize the study results, the main threat is the study sample, which comprises 74 ontology engineers and most of them are Brazilians. Ideally, the sample should be larger and the geographic distribution of the study participants more diverse. To minimize this threat, we invited people from different countries and organizations and also the authors of papers selected in an ongoing systematic literature mapping we are carrying out. Moreover, we asked people to invite other people freely. Even so, the sample presents limitations. Given that the participants’ profile directs influence the results, new studies involving participants with different profiles might produce different results. Finally, the decisions and data interpretations made by the researchers affect Reliability Validity, which regards the influence of the researchers on the results and conclusions. The classification schemas for categorizing data collected in open questions and the classification of data in such categories involve a lot of judgment, as well as analyzing data and getting conclusions. Therefore, other researchers could obtain different results. To minimize this threat, the collected data was initially discussed and analyzed by five authors. Data and analysis results were then presented to the other two authors (more experienced researchers) who reviewed them and discussed the results with the other researchers to refine the conclusions and reach a consensus. 7. Related Work In the literature there are several works addressing tools for OE. Some of them propose tools, such as WIDOCO [48], a wizard for documenting ontologies, ROBOT [49], a tool to automate workflows, and Chowlk [50], a drawing library and a converter to draw conceptual models and convert them into operational ontologies. Other works compare tools. For example, in [16], Abburu and Babu compared tools (Ontolingua Server, OntoSaurus, OilEd, WebOnto, Protégé, Swoop, TopBraid Composer, WebODE, OntoEdit, and Neon Toolkit) considering a set of features (e.g., versioning, collaborative functions, extensibility, support for large ontologies). Vigo et al. [19], in turn, compared Protégé to its web version, WebProtégé, TopBraid Composer and Swoop against criteria such as situational awareness, ontology overview, search engine, filtering and linking features, ontology retrieval and reuse, among others. Alatrish [51] studied Apollo, OntoEdit, Protégé, Swoop and Top Braid Composer regarding aspects such as software architecture, tool evolution, interoperability, inference services, and usability. Slimani [52] used these same criteria to compare other tools. A few works have reported surveys investigating OE tools. We highlight the work by Khondoker and Mueller [53], who conducted a survey with 32 practitioners who were asked to choose a tool and perform a modeling task using it. After that, they provide feedback regarding the tool used, considering their perceptions of the tool’s user-friendliness, learning time, problems, and their level of satisfaction with the tool, among others. The results showed that although there was a wide range of free tools, the participants chose only a few of them, most OE-specific tools, with Protégé being the predominant. From the works we found in the literature, we consider [53] the one most related to ours because it consists of a survey with OE practitioners aiming at getting their perceptions about OE supporting tools. In both works, Protégé is the predominant tool and there has been a preference for using OE-specific tools. However, there are some important differences between the work by Khondoker and Mueller [53] and ours. The survey presented in [53] focused on the use of the tools in the conceptual modeling task at hand instead of on the aspects that led the participants to choose the used tools and how they use them. Moreover, in that work, only aspects relevant to the performed modeling task were investigated. In contrast, our work investigates a more comprehensive set of aspects, provides an overview of the use of tools to support OE practice, gathers evidence on this subject, and raises some issues that can be addressed in future research. 8. Final Considerations This paper presented a survey that investigated the use of tools to support OE. The survey aimed to answer three research questions: (RQ1) Which tools have been used and which are their characteristics?; (RQ2) What benefits and difficulties have been perceived when using the tools?; and (RQ3) Which properties have been considered important in OE tools? In summary, concerning RQ1, there has been a predominance of OE-specific tools, mainly Protégé, Visual Paradigm, and Astah, which support modeling and implementation activities. Generic tools have been used mainly to support OE initial phases, mostly Requirements Specification. The tools have been discovered mainly through recommendations from other people and there has been a preference for free tools. Tools have not provided adequate integrated support for the OE process. As for RQ2, increasing development process performance and improving ontology quality have been acknowledged as the main benefits. In contrast, integration problems and scope limitations have been the main difficulties faced by ontology engineers. Regarding RQ3, Functional Suitability, Reliability, Usability, and Availability have been considered the most relevant properties. These results provide a panorama of tool usage in OE and raise some issues that can be addressed in future research. By looking at the pros and cons of using the tools, their weaknesses, and strengths we can set a path to provide advances in the topic. For example, improving the support for ontology requirements specification activities and connecting the produced artifacts to other tools would benefit OE practice. This study can be replicated in other countries, allowing for data comparison and evaluation of its applicability in broader contexts. It is also necessary to produce solutions to enable integrated support for the OE process and deal with scope limitations. Moreover, there is a need to improve the quality of the tools by considering the properties ontology engineers expect to find in them. Therefore, investigating such properties and defining strategies to achieve them would be welcome. Currently, we are carrying out a systematic mapping of the literature to provide a panorama of the state of the art about OE tools. We intend to analyze the survey and systematic mapping results to obtain a multivocal overview of tool usage in OE and shine a light on the road ahead. Considering the studies’ results, we intend to produce a solution (e.g., a catalog, a repository, guidelines) to help ontology engineers search and use tools. Acknowledgments We would like to thank everyone who took part in the survey and spread our questionnaire to help us with this study. This research is supported by Coordination for the Improvement of Higher Education Personnel - Brazil (CAPES) - Finance Code 001, and Espírito Santo Research and Innovation Support Foundation (FAPES) - Processes 2023-5L1FC, 2021-GL60J, 2022-NGKM5, and T.O. 1022/2022. References [1] M. Uschold, M. Gruninger, Ontologies and semantics for seamless connectivity, ACM SIGMod Record 33 (2004) 58–64. [2] C. C. Reginato, J. S. Salamon, G. G. Nogueira, M. P. Barcellos, V. E. S. Souza, M. E. Monteiro, R. Guizzardi, A goal-oriented framework for ontology reuse, Applied ontology 17 (2022) 365–399. [3] D. Oberle, How ontologies benefit enterprise applications 5 (6)(2014), 473–491. doi: 10.3233, SW-130114 (????). [4] F. M. Mendonça, J. L. Emygdio, L. P. de Castro, E. R. Felipe, Onto4alleditor: a graphic web ontology editor for information science: a graphic web ontology editor for information science, Frontiers of Knowlegde Representation 1 (2021) 70–94. [5] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins, et al., Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai, Information fusion 58 (2020) 82–115. [6] R. Confalonieri, G. Guizzardi, On the multiple roles of ontologies in explainable ai, arXiv preprint arXiv:2311.04778 (2023). [7] P. S. S. Júnior, M. P. Barcellos, R. de Almeida Falbo, J. P. A. Almeida, From a scrum reference ontology to the integration of applications for data-driven software development, Information and Software Technology 136 (2021) 106570. [8] E. Blomqvist, K. Hammar, V. Presutti, Engineering ontologies with patterns-the extreme design methodology., Ontology Engineering with Ontology Design Patterns 25 (2016) 23–50. doi:10. 3233/978-1-61499-676-7-23. [9] R. A. Falbo, SABiO: Systematic Approach for Building Ontologies, in: G. Guizzardi, O. Pastor, Y. Wand, S. de Cesare, F. Gailly, M. Lycett, C. Partridge (Eds.), Proc. of the Proceedings of the 1st Joint Workshop ONTO.COM / ODISE on Ontologies in Conceptual Modeling and Information Systems Engineering, CEUR, 2014. [10] S. Peroni, A simplified agile methodology for ontology development, in: OWL: Experiences and Directions–Reasoner Evaluation: 13th International Workshop, OWLED 2016, and 5th Inter- national Workshop, ORE 2016, Bologna, Italy, November 20, 2016, Revised Selected Papers 13, Springer, 2017, pp. 55–69. [11] M. C. Suárez-Figueroa, A. Gómez-Pérez, M. Fernández-López, The neon methodology for ontology engineering, in: Ontology engineering in a networked world, Springer, 2011, pp. 9–34. [12] D. Spoladore, E. Pessot, Collaborative ontology engineering methodologies for the development of decision support systems: Case studies in the healthcare domain, Electronics 10 (2021) 1060. [13] M. Poveda-Villalón, A. Fernández-Izquierdo, M. Fernández-López, R. García-Castro, Lot: An industrial oriented ontology engineering framework, Engineering Applications of Artificial Intelligence 111 (2022) 104755. [14] T. Tudorache, Ontology engineering: Current state, challenges, and future directions, Semantic Web 11 (2020) 125–138. [15] S. Rautenberg, J. L. Todesco, A. V. Steil, F. A. Gauthier, Uma metodologia para o desenvolvimento de ontologias, RECEN-Revista Ciências Exatas e Naturais 10 (2008) 237–262. [16] S. Abburu, G. S. Babu, Survey on ontology construction tools, International Journal of Scientific & Engineering Research 4 (2013) 1748–1752. [17] R. Iqbal, M. A. A. Murad, A. Mustapha, N. M. Sharef, et al., An analysis of ontology engineer- ing methodologies: A literature review, Research journal of applied sciences, engineering and technology 6 (2013) 2993–3000. [18] J. Cardoso, The semantic web vision: Where are we?, IEEE Intelligent systems 22 (2007) 84–88. [19] M. Vigo, S. Bail, C. Jay, R. Stevens, Overcoming the pitfalls of ontology authoring: Strategies and implications for tool design, International Journal of Human-Computer Studies 72 (2014) 835–845. [20] O. Corcho, M. Fernández-López, A. Gómez-Pérez, Methodologies, tools and languages for building ontologies. where is their meeting point?, Data & knowledge engineering 46 (2003) 41–64. [21] K. I. Kotis, G. A. Vouros, D. Spiliotopoulos, Ontology engineering methodologies for the evolution of living and reused ontologies: status, trends, findings and recommendations, The Knowledge Engineering Review 35 (2020) e4. doi:10.1017/S0269888920000065. [22] R. Studer, V. R. Benjamins, D. Fensel, Knowledge engineering: Principles and methods, Data & knowledge engineering 25 (1998) 161–197. [23] G. Guizzardi, Conceptualizations, modeling languages, and (meta) models, in: Databases and Infor- mation Systems IV: Selected Papers from the Seventh International Baltic Conference, DB&IS’2006, volume 155, IOS Press, 2007, p. 18. [24] O. Noppens, T. Liebig, Ontology patterns and beyond: towards a universal pattern language, in: Proceedings of the 2009 International Conference on Ontology Patterns-Volume 516, 2009, pp. 179–186. [25] A. Stadnicki, F. F. Pietroń, P. Burek, Towards a modern ontology development environment, Procedia Computer Science 176 (2020) 753–762. [26] M. Fernández-López, A. Gómez-Pérez, N. Juristo, Methontology: from ontological art towards ontological engineering, American Asociation for Artificial Intelligence (1997). [27] N. F. Noy, D. L. McGuinness, et al., Ontology development 101: A guide to creating your first ontology, 2001. [28] O. Corcho, M. Fernandez-Lopez, A. Gomez-Perez, Ontological engineering: what are ontologies and how can we build them?, in: Semantic web services: Theory, tools and applications, IGI Global, 2007, pp. 44–70. [29] J. Guerson, T. P. Sales, G. Guizzardi, J. P. A. Almeida, Ontouml lightweight editor: a model- based environment to build, evaluate and implement reference ontologies, in: 2015 IEEE 19th international enterprise distributed object computing workshop, IEEE, 2015, pp. 144–147. [30] C. M. Fonseca, T. P. Sales, V. Viola, L. B. R. da Fonseca, G. Guizzardi, J. P. A. Almeida, Ontology- driven conceptual modeling as a service., in: JOWO, 2021. [31] A. Gomez-Perez, J. Angele, M. Fernandez-Lopez, V. Christophides, A. Stutt, Y. Sure, et al., A survey on ontology tools, OntoWeb deliverable 1 (2002). [32] S. Easterbrook, J. Singer, M.-A. Storey, D. Damian, Selecting empirical methods for software engineering research, Guide to advanced empirical software engineering (2008) 285–311. [33] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, A. Wesslén, Experimentation in software engineering, Springer Science & Business Media, 2012. [34] C. A. D. Gomes, J. S. Monfardini, Glaice Kelly Q.and Salamon, R. d. S. Sangali1, I. S. R. Timoteo, M. P. Barcellos, V. E. S. Souza, Investigating tool usage in ontology engineering: a survey - protocol & data extraction (2024). doi:10.6084/m9.figshare.25927438. [35] M. Musen, The protégé project: A look back and a look forward, AI Matters 1 (2015). doi:10. 1145/2557001.25757003. [36] J. L. Moreira, T. P. Sales, J. Guerson, B. F. Braga, F. Brasileiro, V. Sobral, Menthor editor: an ontology driven conceptual modeling platform (2016). [37] I. O. for Standardization (ISO), Iso/iec 25010:2011 - systems and software engineering – systems and software quality requirements and evaluation (square) – system and software quality models, ISO Standards (2011). URL: https://iso25000.com/index.php/en/iso-25000-standards/iso-25010. [38] J. Sanders, Linux, open source, and software’s future, IEEE software 15 (1998) 88–91. [39] J. Feller, Perspectives on free and open source software, MIT Press, 2005. [40] B. Fitzgerald, Has open source software a future? (2005). [41] L. Zhang, Y. Yu, J. Lu, C. Lin, K. Tu, M. Guo, Z. Zhang, G. Xie, Z. Su, Y. Pan, Orient: Integrate ontology engineering into industry tooling environment, in: The Semantic Web–ISWC 2004: Third International Semantic Web Conference, Hiroshima, Japan, November 7-11, 2004. Proceedings 3, Springer, 2004, pp. 823–837. [42] A. Alobaid, D. Garijo, M. Poveda-Villalón, I. Santana-Perez, A. Fernández-Izquierdo, O. Corcho, Automating ontology engineering support activities with ontoology, Journal of Web Semantics 57 (2019) 100472. [43] G. Dziwis, L. Wenige, L.-P. Meyer, M. Martin, Ontoflow: A user-friendly ontology development workflow., in: SemIIM, 2022. [44] A. Katifori, C. Halatsis, G. Lepouras, C. Vassilakis, E. Giannopoulou, Ontology visualization methods—a survey, ACM Computing Surveys (CSUR) 39 (2007) 10–es. [45] M. Dzbor, E. Motta, C. Buil, J. Gomez, O. Görlitz, H. Lewen, Developing ontologies in owl: An observational study (2006). [46] A. Zouaq, R. Nkambou, A survey of domain ontology engineering: Methods and tools, in: Advances in intelligent tutoring systems, Springer, 2010, pp. 103–119. [47] A. S. Puspaningrum, S. Rochimah, R. J. Akbar, Functional suitability measurement using goal- oriented approach based on iso/iec 25010 for academics information system, Journal of Information Systems Engineering and Business Intelligence 3 (2017) 68–74. [48] D. Garijo, Widoco: a wizard for documenting ontologies, in: The Semantic Web–ISWC 2017: 16th International Semantic Web Conference, Vienna, Austria, October 21-25, 2017, Proceedings, Part II 16, Springer, 2017, pp. 94–102. [49] R. C. Jackson, J. P. Balhoff, E. Douglass, N. L. Harris, C. J. Mungall, J. A. Overton, Robot: a tool for automating ontology workflows, BMC bioinformatics 20 (2019) 1–10. [50] S. Chávez-Feria, R. García-Castro, M. Poveda-Villalón, Chowlk: from uml-based ontology concep- tualizations to owl, in: European Semantic Web Conference, Springer, 2022, pp. 338–352. [51] E. Alatrish, Comparison some of ontology, Journal of Management Information Systems 8 (2013) 018–024. [52] T. Slimani, Ontology development: A comparing study on tools, languages and formalisms, Indian Journal of Science and Technology 8 (2015). [53] M. R. Khondoker, P. Mueller, Comparing ontology development tools based on an online survey, World Congress on Engineering 2010 (WCE 2010), London, UK, 2010.