In Model-Driven Engineering, the evolution of models is commonly defined using model transformation languages, which can be used to specify the distinct needs of a requirements or engineering change at the software modeling level. Model transformations are also a type of software abstraction that can be subject to human error. This paper presents a research proposal to investigate applying three traditional bug localization approaches (i.e., Stepwise Execution with Breakpoints, Omniscient Debugging, and Query-based Debugging) to model transformations. The research proposal outlines a plan to investigate each technique along with empirical assessments to evaluate each technique individually and in combination.
Model-Driven Engineering (MDE) has emerged as a software development paradigm that can assist in separating the issues of the problem space of a software system from the accidental complexities of implementation in the solution space. MDE approaches often use customized domain-specific modeling languages (DSMLs) that capture the intent of a particular group of end users through abstractions and notations that fit a specific domain of interest. Like all software systems, evolution also occurs in software models. In MDE, the evolution of models is commonly defined using model transformation languages (MTLs), which can be used to specify the distinct needs of a requirements or engineering change at the software modeling level. Model transformations are also a type of software abstraction that can be subject to human error. Thus, traditional approaches to bug localization may also be applied to assist in locating errors in model transformations.
Debugging is a common task that all software developers encounter across
different software artifacts. Seifert and Katscher [
One of the goals of MDE is to improve developer productivity. This goal is
obtained primarily through raising the level of abstraction away from the solution
domain by focusing on models and model transformations that focus on the problem
domain. Despite the focus on models and model transformations as opposed to
GPPLs, traditional development concerns such as debugging must still be undertaken
by developers adopting MDE practices. Bran Selic [
My research explores the application of several debugging techniques, including
stepwise execution with breakpoints, omniscient debugging, and query-based
debugging. The research is conducted within the context of AToMPM [
The majority of existing work concerning debugging has focused on the application of debugging approaches to GPPLs, such as C and Java. A wide variety of tools and techniques that aid developers in the process of debugging have been created, studied, and evolved. A number of different approaches have been introduced including the traditional combination of breakpoints and stepwise execution, as well as more advanced approaches, such as Omniscient (also referred to as Back-In-Time) Debugging and Query-Based Debugging. 2.1
Stepwise execution is the most commonly implemented feature for debugging
support. Stepwise execution allows the developer to observe hidden state information
dynamically during execution and in many implementations to alter state information,
or even the behavior of the system. Some minor differences were observed between
tools that were largely derived from differing features of the transformation language
(e.g., AToM3 [
An exploratory survey I conducted on the state of debugging support for model
transformations revealed only one MDE tool, TROPIC [
Query-based debugging has been explored more extensively in the context of
GPPLs. Both static and dynamic approaches to query-based debugging have been
investigated [
In my survey of debugging support for MDE, no tool was identified that supports
Omniscient (back-in-time) debugging. Omniscient debugging is a technique that
enables a developer to reverse the execution of a system. This technique can be
considered an extension to stepwise execution with a step-back or even step-back-to option.
Omniscient debugging has been explored in the context of GPPLs [
The existing work regarding omniscient debugging is often challenged by issues regarding efficiency. In order to step backwards through the execution of the system effectively and efficiently, a trace of the execution history must be maintained. Here, an effective technique is described as one which enables developers to reach all points of interest; and an efficient technique is one which minimizes two competing concerns, time to reach a point of interest and memory consumption. However, maintaining the needed trace information can introduce a state space explosion which opposes the goal of minimizing space consumption. Current research indicates several basic techniques to control the state space explosion. The first technique maintains all information within a rolling window of the execution history. The rolling window is a specified number of steps and represents the portion of the execution history that is able to be navigated. The second technique enables traversal through the entire execution history, but only remembers items of interest, which can be chosen either before or during execution depending on implementation. The final technique maintains all trace information initially and dynamically removes trace information related to any element that is no longer referenced in the current step of execution. This final technique functions similar to garbage collection in many modern languages (e.g., Java). Each technique focuses on limiting the amount of information maintained while maximizing the ability to review prior states of the system. 3
The two primary goals of my research are: 1) to investigate Step-wise Execution with Breakpoints, Query-based debugging, and Omniscient debugging in a manner that can support any transformation language, and 2) to evaluate Query-based debugging and Omniscient debugging empirically in comparison to the more commonly supported technique of stepwise execution with breakpoints. This will be a human-based evaluation using a control group that will not be provided any formal debugging support as a base point for comparison. 3.1
To achieve the first goal of my research, my target platform will be AToMPM (A
Tool for Multi-Paradigm Modeling) [
The investigation of the debugging facilities for AToMPM will be completed in
three phases. The first phase, currently in progress, will introduce stepwise execution
with breakpoints and tracing for model transformations. Stepwise execution will
include the three basic steps (step-over, step-in, and step-out) and a preliminary
implementation is in place that targets TCore and MoTif [
The second phase of the investigation will introduce tracing facilities into the transformation engine, which can track the complete history of a model transformation execution. The tracing facilities will be needed to implement both query-based debugging and omniscient debugging. The key difficulty will be the management of space consumption in the tracing strategy. A potential solution could be to maintain either a set of large deltas (more than a single step in the model transformation execution) or jump points (a complete copy of the model at a given point in the execution history). Large deltas could prove superior in the case where a small portion of the model is being altered frequently. Maintaining a large delta rather than incremental deltas would eliminate any changes that are only temporary. Jump points could prove superior in the case where large portions of the model are being altered frequently. The jump point would contain more information than the large delta, but would allow for state information to be reached more quickly because the entire model could be reloaded from the jump point. These techniques focus on limiting the amount of space consumed by reducing redundancies in the trace and could be further enhanced by applying any of the previously discussed techniques from the existing omniscient debugging literature. After tracing facilities have been explored, omniscient debugging will be investigated using the trace information to allow the developer to traverse back through the execution of a model transformation.
The third, and final, phase of the research will study query-based debugging in the MDE context. The development of the query language will be informed by the results of a study that will be completed in Fall 2013. This study will explore how developers form and use queries during a debugging session. The intent is to provide a query language most closely resembling that used naturally by developers, which will also enable accurate and efficient query evaluation. These competing concerns will likely involve some measure of compromise in the development of the query language. I would also like to explore the possibility of introducing suggested queries based on current contextual information during a debugging session similar to The Whyline. A query may have many results and, depending on the scope of the query, each result could be very large. Therefore, one challenge that will need to be addressed is how to display query results in an efficient and effective manner. An efficient visualization technique will minimize screen space required to display query results. An effective visualization technique will convey the query results in a manner that enables developers to select the relevant results quickly, which minimizes the impact of searching query results on the bug localization process. 3.2
The second goal of my research is to grow the body of empirical evidence regarding debugging for model transformations. Currently, there is a significant lack of empirical research in this area. Existing work largely focuses on case studies that step through a contrived scenario to illustrate the usefulness of the proposed work. This work does illustrate the usefulness of a technique, but does not provide any empirical evidence of the gains in productivity provided by using the technique or any other insight into the effect of the technique in use. I plan to empirically evaluate three techniques separately and in combination. The studies will be a series of human-based experiments. The experiments will utilize a control group that is not provided any debugging tool support. The focus of the experiments will be on evaluating several criteria, such as those addressed by the following questions: 1. What, if any, gains can be achieved by utilizing the experimental techniques as compared to the control group with regards to accuracy, recall, and efficiency? 2. How do developers use the new techniques? This will be accomplished with qualitative analysis of observations made during the experiment. 3. How do developers feel about the techniques and corresponding implementation? This will be accomplished with a post-survey that asks developers about their opinions concerning the technique used. 4. Are techniques synergistic? This will be accomplished by replicating the experiment providing an environment with all experimental debugging techniques.
The results of this series of experiments will be used to identify the positive and negative aspects of the proposed techniques, both individually and in combination, which could lead to further research questions. In addition, the experiments will be designed to be replicated by other researchers. A packet including how to implement the experiment and all relevant materials (e.g., transformations and models) will be made publicly available. The replication packets will enable future researchers to validate and directly compare their results to my experiments. The replication packets will enable future researchers to broaden the scope of these results with their own work.
I also plan to evaluate the various tracing implementations described in Section 3.1. The experiment will be an automated laboratory experiment in which several benchmark transformations will be used to identify space consumption and any slowdown produced by each implementation. These experiments will evaluate the strengths and weaknesses of each tracing technique.
My proposed research will provide two primary contributions. The first expected contribution is an example implementation of generic debugging facilities for MTLs, including generic tracing facilities, as well as three debugging techniques: stepwise execution with breakpoints, omniscient debugging, and query-based debugging. A technique will be investigated that provides support to arbitrary MTLs utilizing TCore as a basis for the underlying implementations. Several additional challenges will be explored including: efficient algorithms for collecting trace information for model transformations, development of an expressive and intuitive query language that also enables accurate and efficient queries, and techniques for displaying query results in an efficient and effective manner.
The second expected contribution is a series of replicated experiments that will investigate the three identified debugging techniques, as well as provide a basis for comparing future research in this area. The experiments will investigate the effectiveness of the three debugging techniques (stepwise execution with breakpoints, omniscient debugging, and query-based debugging) as compared to a control group that will not utilize formal debugging facilities. The experiments will also evaluate a number of qualitative measures, including: how developers utilize the debugging techniques, how developers feel concerning the debugging techniques (e.g., comfort with the technique and perception of effectiveness), and if the techniques are synergistic (i.e., if an environment that provides facilities for multiple techniques prove more or less effective than an environment that does not). Finally, a replication packet containing all of the necessary information for other researchers to replicate the experiments independently will be provided. The replication packet will enable future researchers to independently verify the results of these experiments, as well as compare future novel debugging techniques directly with the three techniques investigated in my research. The experiments have the potential to cultivate the existing body of empirical evidence regarding debugging of model transformations, as well as enable future researchers to add to the body of evidence provided by the experiments. 5
Currently, I am in the first phase of the investigation, which is focused on adding stepwise execution with breakpoints to AToMPM. I am also currently working with a small team to rework the backend of AToMPM to provide a scalable model for the cloud based multi-paradigm modeling tool. I expect to complete the first phase before the end of Fall 2013. The second phase of investigation will include generic model transformation tracing facilities and the addition of omniscient debugging for AToMPM. Additionally, in this phase I will design and begin the series of replicated experiments that will evaluate the three identified debugging techniques, as well as design and run a separate experiment to evaluate the various tracing implementations. The second phase is planned to be completed before the end of Spring 2014. The third, and final, phase will include the investigation into query-based debugging in AToMPM and completing the series of replicated experiments. The third phase is planned to be completed before the end of Fall 2014. After all phases are completed, I will begin writing my dissertation with a planned graduation date of May 2015.