[Context and Motivation:]Software project artifacts such as regulatory codes, requirements, design, code, and test cases facilitate many requirements-related engineering tasks including change impact analysis, safety assessment, and coverage analysis. Unfortunately, users often lack technical expertise in query languages such as SQL or XQuery and therefore have di culties constructing meaningful queries. The problem can be alleviated when users express their queries using natural language (NL). [Question/problem:] NL interfaces depend upon domain models that capture the concepts and terminology of the domain. While prior studies have explored the types of queries, and associated terminology, used by software developers and maintainers, no such systematic study has been conducted in the requirements domain. We therefore do no fully understand the queries, terminology, and user interfaces needed to interactively support NL queries for requirements engineers. [Principal ideas/results:] This dissertation proposes a series of empirical studies to discover the information needs, and related queries, pertinent to requirements engineers. A domain model capturing concepts and terminology of the domain will be extracted from the sample queries. The work will also include a series of user-studies focusing on improving query representation techniques for displaying and communicating the interpreted NL Queries back to the requirements engineer. [Contribution:]Prior work has developed a prototype NL interface for supporting Software Engineering queries but has not studied its use in practice. The proposed solution will focus on NL Queries in the requirements domain, delivering a domain model to enhance understanding of requirements-related queries, and developing and evaluating query representations which support requirements engineers in the tasks they perform.
Software projects produce a huge amount of data artifacts throughout their
development process. This data can be leveraged to analyze and improve the
software systems by supporting activities such as requirements engineering,
regression analysis, predictive analysis, coverage analysis, impact analysis and so on [
A Natural Language Interface(NLI) addresses this problem by allowing users
to express queries in their own words; however, general purpose NL interfaces
have limited information of a particular domain and as a result, they fail to
fully understand many complex domain-speci c queries [
The remainder of this paper is laid out as follows. Section 2 provides the literature review of NL queries in the software engineering domain and discusses various query mechanisms. Section 3 gives an overview of the prior work on domain model construction and query representations. Finally, Section 4 discusses the proposed work and the nal contribution of this research study. 2
This section summarizes the related work on natural language project queries and the query representation techniques.
Software Project Queries : Many researchers have identi ed the questions
asked by software practitioners, however limited studies have been done that
focus on the information needs of a requirements engineer. Fritz et al. interviewed
11 software developers and identi ed 78 questions spanning from multiple
artifacts such as source change, change sets, work items, etc., [
Query Representations : Many techniques have been developed to represent
queries in the software engineering domain. Mader et al. developed the Visual
Traceability Modeling Language(VTML) which, assumes an underlying Uni ed
Modeling Language(UML) class diagram representing the traceability strategies
of the project [
An overview of our preliminary work in creating an elementary level domain model and query representation technique is presented here. 3.1
Building a Domain Model Natural language project queries were collected from practitioners through a series of di erent user studies. The queries were then analyzed for extracting the domain terms to construct a domain vocabulary.
Query Collection : In our pilot study, we developed a web-collection tool
that displayed a set of Traceability Information Models(TIM) [
Query Analysis : From the initial parsing of the query, we determined the project speci c terms and the question terms. The project speci c terms included words or phrases present within the data artifacts while the question terms holds the query together and represents speci c query functions. For example, \List all", \Show me" or \Provide me a list of" relate to selecting a particular set of data. Similarly, the terms, \How many" or \Count the number of" are associated with aggregation functions. Further, the queries were catalogued according to their related software engineering tasks. The queries shown in Table 1 list sample query topics they were classi ed into. They were also grouped based on their semantic and syntactical charecteristics related to the query type and required functions. Example query, \Which use cases have the highest risks?" depicts a characteristic of listing some set of rows and to perform an aggregation function. 3.2
User Evaluation of Query Representation Techniques
We conducted three empirical studies to assess the e cacy of di erent query
representation techniques [
The initial study included a survey designed to comparatively evaluate the four di erent query representations, which included SQL, VTML, Reverse Snow ake Joins(SFJ) and the output of the query(QO) as shown in gure 1. SQL is one of the most common structured textual query representation. VTML and SFJ are visual query presentation techniques where VTML speci cally represents software data queries and SFJ represents more general purpose database queries. QO can help users to analyze a query by reviewing the data returned. Environmental Assumptions
Id Assumptions 1…*
Requirements
type = “Security” Reverse Snowflake Join Query Environmental Assumptions as EA
Id
Assumptions Requirements as SRS
Id type = “Security” tm_SRS_EA envId reqId
SQL Query SELECT DISTINCT environmental-assumptions.Id, environmental-assumptions.Assumptions FROM environmental-assumptions, requirements, tm_SRS_EA WHERE requirements.type= "Security" AND (tm_SRS_EA. envID = environmental-assumptions.Id AND tm_SRS_EA.reqId = requirements.Id) Sample Results Id EA1 EA6
Assumptions All temperatures shall be recorded in degrees Fahrenheit.
All changes in temperature shall be managed at increments of one degree Fahrenheit.
The multiple choice survey consisted of sixteen queries, four of each kind. For each of them, participants were asked to select the NL query which they believe best matched the given representation of the multiple options provided. The result analysis showed that the VTML representation allowed users to interpret a query more accurately in less time as compared to the other techniques.
We were then interested to know, whether users integrate knowledge from di erent query representations in order to comprehend the meaning of the query. Another study was designed that included an eye-tracker with a usability software which recorded the participants eye-gaze information from the display screen. The study included nine NL queries. Each of these queries were deliberately transformed incorrectly and presented in VTML, SQL and QO. All three representations were displayed on a single screen along with the NL query to be interpreted. The common transformation errors included mapping onto incorrect artifacts or their associated properties, logical errors etc. An example query from the study, \List any test cases which have failed in the past week" was represented as \List any test cases which have failed in the past month". The participants were asked to determine whether the NL query matched the given representation and if not then to explain how it di ered from the actual query. We asked them to use think-out-loud protocol to answer these questions.
From the result analysis, we observed that even though most users favored a speci c representation technique the majority of them leveraged information from multiple sources to analyze the query. Secondly, the users often spent too much time on super uous details thereby, increasing the total query interpretation time. To address this, we designed another user study which simpli ed the original query displays by hiding unnecessary details and JOIN paths. The results showed that the time to identify errors in reduced displays was significantly decreased in comparison to the original displays.
Results from this work have been integrated into our TiQi tool (see TiqiAnalytics.com) and will be used to support the proposed work.
Previously, we created a domain model that covers broader but more general purpose software domain queries. To customize the domain model supporting requirements related queries, we propose the following: { Identifying Requirements Engineer's Tasks and Questions: We will interview requirements engineers and business analysts to understand the kind of queries they are interested in posing. For this, we plan to use mailing lists such as the RE mailing list, and our network of contacts with professional requirements engineers to recruit at least 10-20 participants. Each participant will be asked to identify requirements-related tasks that they perform, the data they need to support those tasks, and relevant NL queries. { Constructing a Requirements-Related Domain Model: We will create a comprehensive knowledge base to support RE related queries. The knowledge base will initially be populated using our existing existing data mining techniques to mine domain-speci c terminology from the text of online requirements textbooks, white-papers, and other requirements-related documents. We will perform the same analysis on the collected queries and augment the domain model accordingly. For example, requirements related terminology that should be recognized by the NL interface includes terms such as \User Story", \Mitigating Requirement" or \Design Constraint". { Evaluating the use of NL Queries within the Requirements Engineering Context: Using the generated domain model and the display mechanisms developed in earlier phases of this dissertation work, we will evaluate the use of NL queries for supporting requirements engineering activities. We will recruit 10-12 requirements engineers using the same protocol as used in the proposed work for,\Identifying the Requirements Engineer's Tasks and Questions". Each participant will be presented with a series of identi ed RE related tasks. Half of the participants will be asked to perform tasks by issuing NL Queries using our NLI, while the others will be given only raw data in a database. For the NL queries we will evaluate the extent to which they are correctly interpreted. Furthermore, we will evaluate the extent to which the use of NL Queries supports and/or augments the completion of tasks.
Requirements Engineers can greatly bene t from increased accessibility to underlying data. This work will contribute an enriched domain model which captures RE terminology and concepts, and a NL interface which provides e ective support for requirements engineers to utilize NL Queries to perform common analytic tasks. 5
I am very thankful to my advisor Dr. Jane Cleland-Huang and co-advisor Dr. Alexander Rasin for their continuous guidance and immense support for carrying out this research.