Software development faces the challenge that GPLs do not provide the appropriate abstractions for domain-specific problems. Traditionally there are two approaches to overcome this issue. One is to use frameworks that provide domainspecific abstractions expressed with a GPL. This approach has very limited support for static semantics, e. g., no support for modifying constraints or type system. The second approach is to use external DSLs for expressing solutions to domain problems. This approach has some other drawbacks: these DSLs are not inherently extensible. Extensible languages solve these problems. Instead of having a single monolithic DSL, extensible languages enable modular and incremental extensions of a host language with domain specific abstractions [1].
To make debugging extensible languages useful to the language user, it is not enough to debug programs after extensions have been translated back to the host language (using an existing debugger for the base language). A debugger for an extensible language must be extensible as well, to support debugging of modular language extensions at the same abstraction level (extension-level). Minimally, this means users can step through constructs provided by the extension and see watch expressions (e. g., variables) related to the extensions.
Because language extensions can be based on other extensions and languages evolve over time, it is essential to constantly test if debugger behavior matches the expected behavior. To test debugging behavior, a GPL can be used, however this raises the same issues discussed above. We therefore propose in this paper DeTeL (Debugger Testing Language), an extensible DSL for testing debuggers.
II. MBEDDR mbeddr [2] is an extensible version of C that can be extended with modular, domain-specific extensions. It is built on top of JetBrains Meta Programming System (MPS) [3] and ships with a set of language extensions dedicated to embedded software development. mbeddr includes an extensible C99 implementation. Further, it also includes a set of predefined language extensions on top of C. These extensions include state machines, components and physical units.
In MPS, language implementations are separated into aspects. The major aspects are Structure, Type System, Constraints, Generator and Editor. However, for building debugging support, the Editor aspect is irrelevant.
To give an idea of building language and debugger extensions, we first build MUnit, a language for writing unit tests, and a corresponding debugger extension. Later, we will describe how to test this debugger extension with a DSL. ExecuteTestExpression returns the number of failed unit tests, hence we specify Int32tType as its type (see rule below). Later, the same type is used in the generator.
The MUnit generator consists of many different transformation rules, which translate code written with the language directly to mbeddr C. Listing 1 shows on the left hand side an example program, written with mbeddr C and MUnit. The right hand side shows the C program generated from it. While regular mbeddr C code is not colored, the boxes indicate how Abstract Syntax Tree (AST) nodes from the left are translated to C code on the right. mbeddr comes with a debugger, which allows users to debug their mbeddr code on the abstraction levels of the used languages. For that, each language contributes a debugger extension, which is built with a framework also provided by mbeddr [4]. Those extensions are always language-specific in contrast to domain-specific debuggers (e. g., the moldable debugger [5]), which provide application-specific debug actions and views on the program state. Hence, debugging support is implemented specifically for the language by lifting the call stack/program state from the base-level to the extension-level (see Fig. 2) and stepping/breakpoints vice versa. Fig. 2. Flow of debug information between base and extension level [4] The debugger framework can be separated into two different parts: First, a DSL and a set of interfaces (shown in Fig. 3) for specifying the debugging semantics of language concepts. Second, a runtime for executing those specifications and thus achieving the mapping described in Fig. 2.
In this section, we provide an overview of the specification part (see Fig. 3) that is required for understanding how the debugger extension for MUnit is built. While this paper concentrates on testing debuggers for extensible languages, we have published another paper [4] describing details about the debugger framework and its implementation with MPS.
Breakables are concepts (e. g., Statements) on which we can set breakpoints to suspend the program execution.
WatchProviders are translated to low-level watches (e. g., Argument) or represent watches on the extensionlevel. They are declared inside WatchProviderScopes (e. g., StatementList), which is a nestable context.
Steppables define where program execution must suspend next, after the user steps over an instance of Steppable (e. g., Statement). If a Steppable contains a StepIntoable (e. g., FunctionCall), then the Steppable also supports step into.
StepIntoables are concepts that branch execution into a SteppableComposite (e. g., Function).
All stepping is implemented by setting low-level breakpoints and then resuming execution until one of these breakpoints is hit (approach is based on [6]). The particular stepping behavior is realized through stepping-related concepts by utilizing DebugStrategies.
StackFrameContributors are concepts that have callable semantics on the extension-level or are translated to low-level callables (e. g., Functions). While the latter do not contribute any StackFrames to the high level call stack, the former contribute at least one StackFrame.
This section describes the implementation of a debugger extension for the MUnit language. This extension is defined with the mbeddr debugger specification DSL and the abstractions of the debugging meta-model shown in Fig. 3.
To enable breakpoints on AssertStatements, an implementation of the Breakable interface is required. Assert-Statement is derived from Statement that already implements this interface, thus breakpoints are already supported.
Since ExecuteTestExpression's stack frame is not shown in the high-level call stack, none of its watches are mapped. In contrast, stack frames for Testcases are visible thus we need to consider its watches. In case of Testcase, the LocalVariableDeclaration _f has no corresponding representation on the extension-level, and is therefore not shown (specified in listing below).
The mbeddr debugger framework uses a pessimistic approach for lifting watches: those that should not be shown in the UI are marked as hidden. Otherwise, the debugger shows the low-level watch (in this case the C local variable _f) with its respective value.
Testcase and ExecuteTestExpression are translated to base-level callables and therefore implement StackFrame-Contributor. They contribute StackFrames, each is linked to a base-level stack frame and states whether it is visible in the extension-level call stack or not.
The implementation of ExecuteTestExpression links the low-level stack frame to the respective instance (see listing below). Further, it hides the frame from the high-level call stack, since ExecuteTestExpression has no callable semantics. contribute frame mapping for frames.select(name=getName());
Similarly the mapping for Testcase also requires linking the low-level stack frame to the respective instance. However, it declares to show the stack frame in the high-level call stack:
String frameName = "test _ " + this.name; contribute frame mapping for frames.select(name=frameName); Further, we provide the name of the actual Testcase, which is represented in the call stack view: Consider Listing 1, where we would show the name forTest instead of test_forTest.
The debugger testing DSL must allow us to verify at least four aspects: call stack, program state, breakpoints and stepping. To cover these requirements in DeTeL we delineate in this section requirements. While we consider some of those requirements as required (R), others are either context (CS) or mbeddr specific (MS).
R1 Debug state validation: Changes in generators can modify names of generated procedures or variables and this way, e. g., invalidate program state lifting in the debugger. For being able to identify those problems, we need a mechanism to validate the call stack, and for each of its frames the program state and the location where execution is suspended. For the call stack, a specification of expected stack frames with their respective names is required. In terms of program state, we need to verify the names of watches and their respective values, which can either be simple or complex. Further, a location specifies where program execution is expected to suspend and tests can be written for a specific platform.
R2 Debug control: Similarly as in R1, generator changes also affect the stepping behavior. Consider changing the FunctionCall generator to inline the body of called functions instead of calling them. This change would require modifications in the implementation of step into as well. For being able to identify those problems, we need the ability to execute stepping commands (in, over and out) and specify locations where to break.
R3 Language integration: The DSL must integrate with language extensions. This integration is required for specifying in programs under test locations where to break (see R2) and for validating where program execution is suspended (see R1). CS2 Extensibility: Languages should provide support for contributing new validation rules thus achieving extensibility. Those new rules can be used for testing further debugger functionality not covered by DeTeL (e. g., mbeddr's upcoming support for multi-level debugging [7]) or for writing tests more efficiently.
CS3 Automated test execution: For fast feedback about newly introduced debugger bugs, we require the ability to integrate our tests into an automatic execution environment (e. g., an IDE or a build server).
MS1 Exchangeable debugger backends: mbeddr targets the embedded domain where platform vendors require different compilers and debuggers. Hence, we require the ability to run our tests against different debugger backends and on different platforms.
DeTeL is open-source and is shipped as part of mbeddr [8]. It is integrated in MPS and interacts with the mbeddr debugger API. DeTeL is currently tightly coupled to mbeddr, however it could interact with a generic debugger API and could be implemented independent of MPS. This section describes the structure of DeTeL and the implementation of requirements discussed in Section VI. The language syntax is not documented, but can easily be derived by looking at its editor definitions in MPS.
Fig. 4 shows the structure of DebuggerTest, which is a module that contains IDebuggerTestContents, currently implemented by DebuggerTestcase and CallStack (described later). This interface facilitates extensibility inside DebuggerTest (CS2). Further, DebuggerTest refers to a Binary, which is a concept from mbeddr representing the compiled mbeddr program under test (R3), the imports of IDebuggerTestContents from other DebuggerTests (CS1) and an IDebuggerBackend that specifies the debugger backend (CS2, MS1). The later is implemented by GdbBackend and allows this way to run debugger tests with the GNU Debugger (GDB) [9]. MPS already contains the language mps.lang.test for writing type system and editor tests. This allows (1) automatic execution of tests on the command-line and (2) visualization of test results in a table view inside MPS. All of that functionality is built for future implementations of ITestcase -an interface from mps.lang.test. By implementing this interface in DebuggerTest (our container for DebuggerTestcases), we benefit from available features (CS3).
CallStack implements IDebuggerTestContent (see Fig. 5) and contains IStackFrames (CS2, R1), which has two implementations: StackFrame and StackFrameExtension.
An extending CallStack inherits all StackFrames from the extended CallStack in the form of StackFrameExtensions, with the possibility of specializing inherited properties (CS1), and can declare additional StackFrames. SteppingConfig is optional and contains a list of IStep-pingCommands (CS2) that are executed after suspending on location (R2). This interface is implemented by StepInto, StepOver, and StepOut (each performs the respective command n times).
ValidationConfig contains a list of IValidations (CS2, R1), implemented by CallStack, CallStackRef and OnPlatform. CallStackRef refers to a CallStack that cannot be modified. Finally, OnPlatform specifies a Platform (Mac, Unix or Windows) and contains validations, which are only executed on the specific platform (R1).
In this section, we describe an application scenario where we apply DeTeL to test the debugger extension of MUnit.
Before
Our test cases from the previous section are generated to plain Java code and can be executed in MPS with an action from the context menu. This functionality is obtained by implementing ITestcase in DebuggerTest (see Section VII-A). By executing this action, test results are visualized in a table view, provided by MPS: for each DebuggerTestcase, the result (success or fail) is indicated with a colored bubble and a text field shows the process output.
As indicated by a green bubble on the left side of Fig. 7, all of our previously written DebuggerTestcases pass. We show in the next section how language evolution will invalidate the debugger definition and this way cause all of our tests to fail.
In this section we modify the MUnit generator to demonstrate how this affects the debugger. Currently, the generator reduces a Testcase to a Function: its name is prefixed with test_, followed by the Testcase name (see Listing 1). We now change this generator, so the Function name is prefixed with testcase_, instead of test_. The listing below shows how our example program from Listing 1 is now generated to C. Because of our generator modification, Testcases are now generated to Functions with a different identifier as before. However, we have not updated the debugger extension, therefore, the call stack construction for all Testcases fails and this way all of our DebuggerTests fail as well (see Fig. 8). Although those debugger tests fail, they are still valid, since they are written on the abstraction level of the languages, not the generator. The next section shows how we update the debugger extension to solve the call stack construction.
The MUnit debugger extension tries to lift for each Testcase a stack frame whose name is prefixed with test_, followed by the name of the respective Testcase (see Section V-D). However, due to our generator modification, this frame is not present and therefore the whole call stack construction fails with an error. To solve this problem, we update the name used for matching the stack frame name:
String frameName = "testcase _ " + this.name; contribute frame mapping for frames.select(name=frameName);
Other aspects, such as stepping, breakpoints or watches are not affected by the generator modification and hence do not need to be changed. Therefore, after fixing the call stack lifting for Testcase our debugger tests pass again.
Wu et al. describe a unit testing framework for DSLs [10] with focus on testing the semantics of the language. However, from our perspective, it is necessary to have testing DSLs for all aspects of the language definition, e. g., editor (concrete syntax), type system, scoping, transformation rules, and finally the debugger. 1 mbeddr contains tests for the editor, type system, scoping and transformation rules, our work contributes the language for testing the debugger aspect.
The Low Level Virtual Machine (LLVM) project [11] comes with a C debugger named Low Level Debugger (LLDB). Test cases for this debugger are written in Python and the unit test framework of Python. While those tests verify the command line interface and the scripting Application Programming Interface (API) of the debugger, they also test other functionality, such as using the help menu or changing the debugger settings. Further, some of the LLDB tests verify the debugging behavior on different platforms, such as Darwin or Linux. In contrast, we only concentrate on testing the debugging behavior, but also support writing tests for specific platforms. Our approach for testing the debugging behavior is derived from the LLDB project: write a program in the source-language (mbeddr), compile it to an executable and debug it through test cases, which verify the debugging behavior.
The GDB debugger takes a similar approach as the LLDB: tests cover different aspects of the debugger functionality and are written in a scripting language [9]. Contrarily, to our approach of testing debugging for one extensible language, the GDB project tests debugging behavior for all of its supported languages, such as C, C++, Java, Ada etc. Further, those tests run on different platforms and target configurations. Our work supports writing tests against different platforms, but does not allow users to change the target configuration via the DSL.
The mbeddr extensible language comes with an extensible debugger. To test this debugger, we have introduced in this paper a generic and extensible testing DSL. The language is implemented in MPS with focus on mbeddr, but the underlying 1 Specific language workbenches might require testing of additional aspects approach is applicable for testing any imperative language debugger. Further, we have shown in this paper (1) the implementation of a language extension, (2) how debugging support is build for it and (3) how the debugger is tested with use of our DSL. The language is designed for extensibility, so others can contribute their own context-specific validation rules. In addition, we concentrated on reuse, so test data, test structures and validation rules can be shared among tests.
In the future, we plan to investigate ways for integrating the debugger specification DSL with the DSL for testing the debugger extension. From this integration we expect to (1) gain advances in validating debugger test cases and (2) the possibility to automatically generate test cases from formal debugger specifications (based on work from [12], [13] and [14]). In addition, we will continue researching on languages for testing non-functional aspects, such as testing the performance of stepping commands and lifting of program state.