[Context & motivation:] Software traceability links which are created during the development process are subsequently underutilized because project stakeholders lack the skills they need to compose trace queries. TiQi addresses this problem by accepting natural language trace queries and transforming them to executable SQL. [Question/problem:] The TiQi engine depends on the presence of a domain model. This can best be constructed through collecting samples from potential users. We are interested to learn what trace queries would be of interest to potential stakeholders and what terminology they would choose to express those queries. [Principal ideas/results:]In this live study we will demonstrate TiQi in action, and lead the participants through a series of carefully crafted 'what-if' scenarios designed to capture a variety of sample queries. [Contribution:] The study is expected to lead to a more extensive domain model which will improve the accuracy of TiQi's query transformation process.
In safety-critical domains traceability is prescribed by regulators [
In our previous paper presented at the International Requirements
Engineering Conference, entitled “TiQi: Towards Natural Language Trace Queries” [
Hazard ‒ ID ‒ Hazard ‒ Classification ‒ Probability causes
FaultFault ‒ ID ‒ Contributing Fault ‒ Severity Environmental Assumption general NL database query language and customize it for the traceability domain; however, an extensive search of available open-source solutions was unsuccessful. We therefore constructed a generic NL database query mechanism and then customized it with vocabulary and concepts from the traceability domain. The result of this work is TiQi, which transforms spoken or written natural language trace queries into executable SQL statements. In a series of controlled experiments, TiQi has transformed NL queries from di↵erent projects at accuracy rates ranging from 50% to 89%.
TiQi prompts the user for a NL query by displaying a Traceability Information Model (TIM), which as depicted in Figure 1 models artifact types, their attributes, and semantically typed links.
In order to interpret queries, TiQi requires an underlying domain model describing the terminology and concepts of the traceability and product domain. This includes project-specific terms, question terms and junk. Project-specific terms describe artifact types and attribute names, and can be extracted from the raw data of the project. Question terms form the ‘glue’ which holds the pieces of the query together. For example terms such as show me, list all, or I’d like to see are all synonyms for the SQL construct SELECT. Similarly, terms such as associated with, related to, or with are synonyms for various forms of JOIN.
Our research goal is to collect sample trace queries from which we can extend TiQi’s domain model. There are two specific research questions. First, “What queries are of interest to project stakeholders?” and second “What terminology do project stakeholders use to express their queries?”
The study will be performed interactively as a large group with both a
paperbased and online option. The session will open with 10 minutes of training in
which the notion of trace queries and TIMs will be presented. First, demographic
data about each participant will be collected. Then each participant will be given
a TIM similar to the one shown in Figure 1. Half of the participants will work
on the Isolette System and half on the EasyClinic system [
In the first part of the study specific scenarios will be used as prompts. The participant will be asked to address the scenario by creating a NL query – using words of their own choosing. There are nine scenarios of which four are shown below. Each participant will be asked to select and create a query for five of them. Participants will also be asked to add 1-2 of their own queries that would support a Software Engineering task which they perform regularly. Sample scenarios include: 1. The safety ocer is worried that an important requirement R136 is not correctly implemented. The developer tells him that it is not only implemented but has also passed its acceptance tests. The security ocer runs a trace query to confirm this. What might his query be? 2. Piotr discovered that one of the high risk hazards have been neglected during system specification. He cannot find any related requirements. Now he is worried that other hazards might have been neglected too. Can you help him issue a trace query to overcome his concern? 3. Last week, Jin modified some functions in order to improve the eciency of the system. The system has not functioned well since then. She is not sure if all the test cases have passed. What trace query should she issue in order to verify this? 4. While testing the system, Paxton becomes suspicious that the test cases aren’t sucient to show that all requirements have been addressed. What trace query could he issue to help him investigate this problem? To add some fun to the session – once all queries have been collected, we will randomly select three of them and feed them to TiQi. The participants will experience TiQi’s capabilities and/or failures in real time. In terms of benefits, all participants will learn more about natural language trace queries and will be given access to our online TiQi tool (not yet released publicly).
Because our goal is to capture sample queries from a broad range of potential project stakeholders, we have only basic prerequisites. We require all participants to hold an MS degree in Computer Science, Information Systems, Software Engineering or a related field, and/or to have at least one year of IT experience. We will capture this information during the session through the meta-questions and will later discard any responses from people who do not meet these criteria.
To analyze the data we will use data mining tools to generate a list of terms, phrases, and their frequencies. We will then automatically filter the list to exclude project specific terms and to remove terms that are already known to TiQi. This will leave the new question terms and the non-useful junk terms. Researchers on our team will then go through this list and categorize the terms according to the equivalent SQL expression, and add them to the model. “Junk“ terms that are neither question terms nor project specific, will be identified and removed. 3
The end result of this study is expected to be a greatly expanded list of synonyms and definitions for each of the SQL expressions in the domain model. This expanded set of domain terminology will enable TiQi to service future NL queries more accurately.