Effective and efficient information sharing for reuse and analysis of scientific data from published research papers is an important challenge for researchers working within the empirical software engineering (EMSE) domain. Currently, there is only limited support for storing empirical research data and results in a way that is easy to access and reuse for other researchers. In this paper, we propose the Systematic Knowledge Engineering Tool (SKET), an ontology-based tool to provide researchers in the EMSE domain with capabilities for storing, sharing, and verifying results within their research community. The initial evaluation results show that SKET can address relevant needs in the EMSE community and can be considered as a foundation for advanced tool capabilities.
Publication databases (PDs) are large collections of scientific publications such as research papers and books. PDs help editors represent and publish research-related books and papers and have become the main source of information for researchers to discover and survey recent research results in their respective areas. The current approach to structure and represent a PD (e.g., IEEEXplore1, Google Scholar and Sco1 http://ieeexplore.org/; http://scholar.google.at/; http://www.elsevier.com/online-tools/scopus pus) mostly focuses on syntactic search in generic publication meta-data elements e.g., title, keywords (with the notable exception of MeSH2 that also provides termbased search). Therefore, these approaches provide only limited support for effective reuse and analysis of scientific data from research papers. This hinders the users in reenacting the analysis reported in a paper or conducting other advanced analysis.
Empirical software engineering (EMSE) is one research area affected by the
limited support of the current PD approach. A typical example of EMSE research data
that is currently difficult to access in traditional PDs includes: the research
hypotheses, key result variables, the outcomes of the experiment, and the raw experiment data
/ materials from EMSE experiments. With the available tools, users usually need
significant expertise, resources, and time to extract detailed information from a published
paper. This typically cost several person/hours every time a user wants to extract
different elements of research data from a paper [
Since the number of EMSE studies is considerably growing, it is needed to adopt
systematic approaches in order to provide objective evidences on a particular EMSE
topic in an efficient way [
Key requirements of EMSE researchers for a good solution are: • An effective and efficient process for data import, including quality assurance, to enable the incremental building and reuse of knowledge. In a context that needs high-quality data (e.g., EMSE), humans need to be involved to assure the quality of extracted data and relationships between data elements. • Concept-level search capabilities on EMSE research documentation, e.g., result variables of an experiment. • An effective and efficient process for querying and data exports from the PD to enable advanced data analysis processes based on the collected knowledge.
Within this paper, we report on the design and implementation of the Systematic
Knowledge Engineering Tool (SKET). SKET is a PD that combines the power of an
ontology-based approach [
Two distinct SKET prototypes covering different EMSE topics are available online3,4. We evaluated these prototypes based on the most relevant requirements and queries from EMSE researchers with data from more than 40 EMSE experiments. The initial evaluation shows that SKET can help to effectively and efficiently find detailed
3 http://cdlflex.org/prototypes/ske/ 4 http://cdlflex.org/prototypes/ske/theory/ experimental data for meta-analysis or reuse purposes, thus, improving on the functionalities offered by mainstream PDs.
The remainder of this paper is organized as follows. Section 2 explains the EMSE domain and clarifies the typical users and queries that SKET supports. Section 3 describes the design of the SKET tool. Section 4 describes the concrete SKET prototypes and their initial evaluation. Section 5 summarizes related work. Section 6 concludes and discusses the future work. 2
This section introduces the Empirical Software Engineering (EMSE) domain, user roles, and typical questions as context for designing and evaluating a tailored Publication Database (PD). 2.1
Researchers in EMSE collaborate on research topics, such as software quality
assurance, to build up a body of knowledge (BoK). An EMSE BoK includes theory models
[
A major goal of researchers in EMSE is to investigate the effects of software engineering concepts, methods, and tools, e.g., whether a new method on software inspection performs differently from established methods, and the influence of factors, such as the level of expertise of the users of a software engineering method. The researchers design and conduct experiments in controlled environments, case studies in academic and industrial environments, and surveys to collect data for analysis from literature and from domain experts. The analysis results of the collected data help to provide practitioners with information on the strengths and limitations of methods and tools as well as on success and risk factors in order to support their informed decision on which methods and tools best to use in and adapt to their software engineering environment. In addition, researchers test hypotheses to develop theories on the effects of software engineering concepts, methods, and tools and drive the planning of the most relevant future experiments, case, studies, and surveys. 2.2
In our previous research [
EMSE researchers typically use a PD to conduct three major tasks: Meta analysis research, replication research, and research network analysis.
Meta analysis research is conducted by posing a set of questions that focus on the aggregation of research results analysis, for example, “Which are the hypotheses investigated in experiments on <inspection method Perspective Based Reading> and <theory construct ‘effectiveness’> and its synonyms?”
Replication research questions aim at retrieving information that could help users to replicate an experiment. While this kind of information could be retrieved from reading the whole paper, we are aiming to simplify this process by providing users with the predefined queries with parameters to reduce the number of papers to be analyzed. One example question here is “Which are the reported findings of experiments on <a certain inspection method>?”
Research network analysis focuses on a community-level understanding of the EMSE research, through questions such as “Which research groups are working on topics with response variables similar to <a certain domain concept>?” 3
The Systematic Knowledge Engineering Tool was created as part of a toolset that
supports the Systematic Knowledge Engineering (SKE) process approach [
SKET automates a significant part of the SKE process, mainly in the third and fourth stages, which will be explained in the Section 3.1 to 3.3. The SKET workflows and module implementation will be explained in Section 3.4. 3.1
The data input to SKET consists of data extracted from EMSE research publications. These research publications typically contain implicit and explicit information related to the experiments conducted as the basis for the publication. Several typical data elements extracted from the experiments are: hypotheses, threats to validity, measurements, and experiment results.
EMSE domain experts extract experiment related information from these publications as part of the SKE process and store this information in a predefined but customizable data format, such as excel spreadsheets. EMSE experts prefer spreadsheets as the main data storage format due to their easy handling during data collection and verification. Therefore SKET accepts spreadsheets as its primary input data format, while being designed to support a wide variety of data import formats from a range of heterogeneous data sources. 3.2
As foundation for the storing process of the raw data, we designed an ontology model based on the EMSE domain context. The high level abstraction of the data model is depicted in Figure 1.
Figure 1 illustrates that researchers and publications contribute the source information, which contains the typical metadata of a publication database (PD), e.g., publication title, authors, and publication events. The light blue parts of the figure depict the data elements relevant for the EMSE BoK and BoK topics. Typical information in this area is the categorization of research inside their respective BoK, e.g., an EMSE BoK may be “Software Quality research” and a specific BoK topic “software inspection”. Empirical study information (dark blue) and empirical study data/artifacts (yellow) consist of detailed, experiment-level information, which is typical for SKET and therefore sets the tool apart from mainstream generic PDs.
The data model used for storing the EMSE information is depicted in Figure 2. The main concepts in this data model are: (1) Publication, which holds the common bibliographic metadata, and (2) Experiment, which glues together all concepts and properties related to the setup of the empirical research experiment, and (3) Experiment Run, which stores the experiment execution details. Currently the ontology data model is not yet re-using any existing ontology, e.g., publication ontology5 and PRO6 (Publishing Roles Ontology). Such an alignment is part of future work.
The storage of the publication metadata extracted by experts into the ontology is currently performed automatically based on a manually crafted mapping between the two data structures 7. To enable both lay-users and domain experts to use SKET, we developed a userfriendly interface for the tool as shown in Figure 3. To access the queries, users click the Queries tab in the top-left part of the SKET prototype page and choose the desired query. The list of queries can be further extended based on user requests.
This user interface mainly consists of a landing page, which explains the contexts of the domain problem, a set of predefined SPARQL queries that reflect the main EMSE expert questions, and a link to a glossary9 of terms related to the domain (in this case the EMSE BoK of software inspection). The result of queries is visualized in a tabular view, which is simple yet sufficient to cater for the needs of the users and can be extended with a graphical visualization of complex relationships. 3.4
The SKET workflow and modules are shown in Figure 4. The arrows show process
steps within SKET and its interaction with users. SKET consists of three modules,
which corresponds with the number in the figure.
1. SKET data import module (DIM). To enable the data import from the
metaresearchers, we provided a spreadsheet import tool to convert their gathered data into
ontology-based metadata. There are already several open source implementations of
spreadsheet data importers, see, e.g., the tool survey in Kovalenko et al. [
2. SKET data storage module (DSM). The DSM enables the usage of ontology data within our environment. For this module, the availability of an inference engine is very important, since it will enable inference result enhancement (2a) based on predefined rules, implemented as generic Jena rules in the current system.
3. SKET ontology-querying tool (OQT). The OQT provides the user interface for
lay users and domain experts. We already proposed a framework for analyzing OQT
for different kinds of users [
For evaluation, we tested SKET with empirical software engineering experts. Figure 5 shows how SKET addresses the challenges posed in the EMSE domain by introducing a Knowledge Base (KB) and the role of a knowledge engineer (KE). In this context, researchers extract data from EMSE studies published in digital libraries (depicted with (1) in Figure 5) and the KE integrates the extracted data into the SKET Publication Database KB. Therefore, the knowledge collected is available for semantic querying also to the general readership, including other researchers and practitioners.
To identify the most relevant query candidates to be answered in the context of an EMSE Body of Knowledge (BoK) on software inspection, we focused on EMSE researchers as main stakeholders, who conduct meta-analyses on study reports or conduct empirical studies and need to be aware of relevant research in their area. Based on an informal survey with software inspection researchers in six research groups (located in Austria, Brazil, Chile, Ecuador, and Spain), we identified a set of query candidates.
For the evaluation of SKET we observed two concrete instances of the SKET
prototype: SKET prototype P13 for general EMSE BoK domain [
P1 contains data from the 30 most recent papers in the specific area of software
inspection in EMSE out of 102 total papers that were found in the area in a Systematic
Literature Review (SLR) [
Since the data from P1 and P2 could be merged together without affecting the result of the queries, we decided to update P1 and P2 ontology with the aggregated version of extracted data from both. Hence, the current versions of the prototypes P1 and P2 use the same ontology. Table 1 provides more detailed information on both prototypes and the effort for publication analysis. The size of the semantic data shows the growing data instance within the ontology, while effort for publication analysis and data extraction reflect the effort needed by experts within the approach.
Initial evaluations of processes using P1 and P2 show that SKET can successfully answer the queries of the researchers in the EMSE domain. However, we are aware that these initial evaluations can provide only limited insight and a future larger scale evaluation is likely to bring up new needs regarding the usability and usefulness of SKET.
The SKET evaluation showed the feasibility of building a software inspection EMSE PD from controlled experiments. A major result of the SKET evaluation was that the effort for data extraction and storing is similar to the effort for data extraction in a traditional SLR process. Therefore, the benefits of SKET can be achieved with little extra cost for data extraction. The resulting KB may be used as input to a future SLR protocol to identify relevant research on a specific topic or for exploring semantic search facilities usually not available in current PDs. SLR extraction sheets, on the other hand, can be used as input for extending the knowledge contained in the SKET.
The SKET prototypes support users in locating empirical evidence and reusing it according to their specific needs. However, it is important to state that support for meta-analyses and for applying specific research syntheses methods are beyond the scope of SKET, which provides exports of relevant data as input to user-specific tool chains for advanced data analysis. Moreover, the relevant queries of our evaluation were chosen focusing on a survey with a specific type of stakeholder, the EMSE researchers in software inspection. Other stakeholders may have different needs.
The straightforward approach for data importing and querying enables researchers to contribute towards building up additional knowledge, and to query for knowledge with low effort. Using ontologies facilitates extensions of the underlying KB common data model and semantic search. The querying capabilities were found efficient in the evaluation on answering the EMSE researchers’ most relevant queries.
The main lessons learned relate to success factors for SKET. A major success
factor is properly involving a knowledge engineer. The overhead of this new role is
likely to be offset soon by benefits obtained by the established KB. Another success
factor is getting the EMSE research community involved for long-term collection and
use of data. Therefore, incentives and benefits of contributing should be clarified.
More detail regarding these evaluation could be found in [
The use of Semantic Web technologies for representing, publishing and making sense
of research related data is a broad and active field of research. CS AKTive space, for
example, was one of the first projects, which facilitated access to research data by
using semantic techniques [
Another line of research focuses on using semantic web technologies to represent
and make use of experimental data, including experimental workflows [
Our work focuses on research data from academic publications, as many of the works above. However, we are interested in representing fine-grained knowledge in terms of used research data and results in the EMSE domain, which brings specific challenges as discussed, and, to our knowledge, has not yet been considered as an application area for semantic publishing. 6
In this paper we introduced the Systematic Knowledge Engineering Tool (SKET), which supports building publication databases (PDs) from empirical studies. SKET supports the Systematic Knowledge Engineering process and provides a Knowledge Base (KB) as storage for the extracted data. SKET allows the research community to find gathered knowledge and reuse it with semantic search facilities, as building blocks for a variety of analyses. An important aspect of SKET is to provide highquality data, which is precise and correct within the scope of the target research area. Therefore, humans need to be involved in quality assurance of the imported data and to provide semantically correct relationships between data elements.
SKET was evaluated by building a PD for the software inspection topic from the knowledge acquired through controlled experiments. Information was extracted from more than 40 research papers and integrated into the KB. The resulting PD on software inspection is available online being implemented in two prototype systems3,4. The main evaluation results were: • Data extraction of inspection experiments into spreadsheets based on the common data model was successful. However, support for flexible mapping should be provided to reuse the data model for different research communities. • The SKET KB was effective and efficient in answering the most relevant stakeholder queries. The standardized query support and the flexible data model provided a stable background for this result. However, a better data storage should be provided for better scalability. • SKET enables knowledge reuse (by applying queries) for analysis and metaanalysis purposes. Moreover, new knowledge, i.e., new data from literature, can be included in the KB as a foundation for a growing EMSE PD.
SKET showed promising results in the software inspection context and should also be evaluated in other contexts. The overall effort of data extraction and import to SKET is comparable to the effort of conducting traditional systematic literature reviews (SLRs) and, for the specific purpose of building an EMSE PD, several advantages of using SKET can be identified, such as analysis of specific topic from EMSE research area and EMSE theory identification support.
As future work we propose: (a) investigating additional EMSE research stakeholder needs; (b) extending the set of empirical studies on software inspections in the SKET KB; (c) instantiating SKET in other research domains beyond EMSE; (d) extending SKET with a platform to allow building on the collective intelligence of the research community for quality assurance and recommendation; (e) Ontology Querying Tool extension to provide users a graphical visualization of complex relationships; (f) reuse of currently existing publication ontologies and (g) providing support for data instance versioning and model evolution for the SKET back-end. This work was supported by the Christian Doppler Forschungsgesellschaft, the Federal Ministry of Economy, Family and Youth, Österreichischer Austauschdienst (ÖAD) and the National Foundation for Research, Technology and Development - Austria. We also wanted to thank the reviewers for their valuable inputs to the paper.