- It is presented a promising approach to test interactive components, supporting the automatic generation of test cases from a specification. The relevance and difficulties (issues and challenges) associated with the testing of interactive components are first presented. It is shown that a formal specification with certain characteristics allows the automatic generation of test cases while solving some of the issues presented. The approach is illustrated with an example of automatic testing of the conformity between the implementation of a button, in the .Net framework, and a specification, written in the AsmL language, using the AsmL Tester tool. The conclusion discusses the characteristics of the tool and gives directions for future work.
Tapplications is a difficult and time-consuming task, he development of high-quality interactive systems and requiring expertise from diverse areas (software engineering, psychology). Current IDE's are not powerful enough for specifying/modeling, building and testing those systems in an effective way. The development of interactive systems and applications based on reusable interactive components is the key to achieve higher quality and productivity levels. Improving the quality of interactive components should have a major impact in the quality of interactive systems and applications built from them, and should contribute to their increased reuse.
In this paper, it is presented a promising approach to the testing of interactive components. By interactive components we mean reusable controls or widgets or interactors, capable of both input from the user and output to the user, written in a general-purpose object-oriented language, such as Java or C#. Interactive components range from the more basic ones (such as buttons, text boxes, combo boxes, list boxes, etc.) to the more sophisticated ones (calendars, data grids, interactive charts, etc.) built from simpler ones. The overhead incurred in testing reusable interactive components pays-off, because of their wider usage and longevity, when compared to special purpose and short lived "final" user interfaces.
The paper is organized as follows: next section (section 2) presents some important issues and challenges of testing interactive components. Section 3 explains the type of test automation that is envisioned (automated specificationbased testing), discusses the type of formal specification required, and discusses its costs and benefits. Section 4 presents an example of performing automated specificationbased tests using the AsmL language and the AsmL Tester tool. Some conclusions and future work can be found in the final section.
Testing interactive components is particularly difficult
because it shares and combines the issues and challenges of
testing object-oriented systems [
Interactive components (and interactive applications and systems in general) have complex event-driven behaviour, difficult to analyze and predict, and, consequently, also difficult to test and debug. Even basic interactive components, such as buttons and text boxes, may react to and generate dozens of events. Most of us have already experienced "strange" behaviours (blocked interfaces, dirty displays, etc.) apparently at random when using wide-spread interactive applications and systems. This should not be a surprise given their complex event-driven behaviour.
Reusable interactive components usually have a highlyconfigurable (or customizable) behaviour. This can be done statically or dynamically by setting configuration properties or attributes, by adding event-handlers or by defining subclasses and method overriding. Testing an interactive component in all the configurations allowed is almost impossible because of the huge set of possible configurations and the difficulty to predict the customized behaviour.
Interactive components have both a user interface (GUI) and an application interface (API). The application interface is used for customizing and composing them, and for linking them with the underlying application logic. Different kinds of inputs and outputs occur via these different interfaces. Adequate testing of an interactive component cannot look at just one of these interfaces in isolation, and has to take into account all these kinds of inputs and outputs in the definition of test cases and in test execution.
Automating the testing of graphical user interfaces poses well-known challenges: 1. How to properly simulate inputs from the user (mouse, keyboard and other higher-level events that are generated by the user)? 2. How to check the outputs to the user without excessive sensitivity to formatting and rendering details?
The designer of a reusable interactive component defines its methods but does not know in advance which kind of applications will make use of them. Method calls between an interactive component and an application occur in both directions: 1. The application (or test driver) may call methods of the interactive component. From the testing perspective, inputs are methods invoked with parameters while outputs are the values returned by those methods. This is the traditional situation in unit testing. 2. The interactive component may generate events (originated from the user or internally generated) that cause the invocation of methods in the application (or test stub), by some kind of callback mechanism (event handlers, or subclassing and method overriding). Again, from the testing perspective, the outputs are the events and parameters passed to the application, while inputs are returned parameters.
Testing the second kind of interaction (callbacks) poses
specific issues and challenges, as already noted in [
Interaction with the user is mediated by the operating system in non trivial ways (often, several layers are involved), introducing more dimensions of configurability, and complicating the analysis and prediction of its behaviour, as well as the testing and debugging tasks.
The documentation supplied with interactive components is usually scarce and not rigorous enough for more advanced uses, such as advanced customization and thorough testing. For example, from the documentation, it is difficult to know precisely: 1. when are events signalled and by what order; 2. what is the internal state of a component when it signals an event; 3. what is safe for an event handler to do; 4. what interactions exist between events.
This usually leads to a trial-and-error style of application programming and poor application quality, and also complicates the design of test cases.
Testing of interactive components is usually difficult and time-consuming due to: 1. the lack of rigorous, unambiguous and comprehensive documentation; 2. the reduced observability (capability to observe the internal state, display produced, and events raised); 3. the deficient controllability (capability to simulate user input).
Some of the issues and challenges described in this section will be addressed by our testing approach and discussed in the next sections. 3
Manual testing of GUIs and interactive components is labour-intensive, frequently monotonous, time-consuming and costly. Some of the reasons are the existence of varied possibilities for user interaction and a large number of possible configurations for each component, and other issues described in section 2, making it impractical the satisfaction of adequate coverage criteria by manual testing. It is necessary to use some type of automation to perform those tests.
The degree of automation we envision is the automatic generation of test cases (inputs and expected outputs) from specification, and not just the type of automation that is provided by unit testing frameworks and tools, such as JUnit (www.junit.org) or NUnit (www.nunit.org), or the type of automation provided by capture and replay tools, such as WinRunner (www.mercure.com) and other tools (www.stlabs.com/marick/faqs/t-gui.htm).
Unit testing frameworks and tools are of great help in organizing and executing test cases, particularly for API testing, but not in generating test cases from a specification.
Capture and replay tools are probably the most popular
tools for GUI testing, but don’t support the automatic
generation of test cases. With these tools, it is possible to record
the user interactions with a graphical user interface (mouse
input, keyboard input, etc.) and replay them later. Capture
and replay tools are of great help in several scenarios, but
also have widely recognized limitations (see e.g. the lesson
"capture replay fails" in [
From a higher perspective, these different approaches and types of automation are complementary and not opponents.
The automatic generation of test cases from specification requires some sort of formal specification of the software to be tested, that can be used to generate concrete input values and sequences, as well as the expected outputs (as a test oracle).
It is possible to design test cases (for black-box testing) from informal specifications, but not in an automated way. At most, inputs can be generated automatically based on the signatures of methods and events (their calling syntax), but expected outputs can only be generated based on a formal specification of their semantics.
A popular type of specification of object-oriented systems is
based on the principles of design by contract [
Although with limitations, some test tools, such as JTest
(www.parasoft.com), have the capability of generating unit
tests based on the specification of pre and post-conditions.
While pre and post-conditions are a good mean to restrict
the allowed behaviours of an object, they are not adequate,
in general, to fully specify their intended behaviour,
particularly when callbacks are involved. As already noted by
Szyperski in his book [
The Object Constraint Language (OCL) (see www.uml.org) goes a step further, by allowing the specification, in the post-condition of a method, of messages that must have been sent (method calls and sending of signals) during its execution. However, in general, it is not possible to specify the order by which messages are sent and the state of the object when each message is sent (important because of re-entrance, as explained in section 2). The definition of post-conditions in OCL has another advantage over its definition in Java or Eiffel, because OCL is a higherlevel formal language supporting formal reasoning and automation.
AsmL [
With appropriate tool support (as is the case of the AsmL
Tester tool), model programs can be used as executable
specification oracles [
For the generation of test cases, the AsmL Tester tool first generates a FSM (Final State Machine) from the AsmL specification, and then generates a test suite (with one or more test cases) from the FSM, according to criteria provided by the user. Since the number of possible object states (possible combinations of values of instance variables) is usually huge, the states in the FSM are an abstraction of the possible object states, according to some criteria provided by the user.
It is well known that state machine models are appropriate for describing the behaviour of interactive systems (and reactive systems in general), and a good basis for the generation of test cases, but usually there is not a good integration between the object model and the state machine model. AsmL and the AsmL Tester tool solve this problem with the generation of the FSM from the specification (formal object model).
When compared to other testing techniques, automated specification-based testing has the disadvantage of requiring a formal specification (to achieve a higher degree of automation). But the investment in the formal specification of reusable interactive components may be largely compensated by the multiple benefits it can bring:
In this example, the AsmL Tester tool is used to test the conformity between the implementation of the button control in the .Net framework ( class) and a specification of a small part of its behaviour (related to mouse and keyboard events only) in the AsmL language. The example is small but was selected mainly to illustrate the testing process and turnarounds to some difficulties, and not the power of the AsmL language. The approach presented can easily scale to be used in larger interactive controls.
The button specification has instance variables ( Specification 1), events ( Specification 2), and methods ( Specification 3).
Two types of events should be distinguished: events
received by the button ( , , and
), and events generated by the button in response
to the previous ones ( ). Both these types of events
may be sent by the button to the application via event
handlers. There are more button events in the .Net platform but
only these events are considered in this example.
Instance variables , and (
Specifi1. Formal specifications and models are an excellent
complement to informal specifications and
documentation, because ambiguities are removed and
inconsistencies are avoided.
2. Formal specifications allow the automation of
specification-based testing, as described in this paper.
3. Besides being useful as the basis for the generation
of test cases, FSM's can also be used to automatically
prove required properties of a system, with
modelchecking tools that exhaustively search the state
space. The properties are written in temporal logic.
For example, Campos, in [
Overall, higher rigor in the description and verification
of interactive components is important to gain confidence
on their correctness and encourage their reuse [
Instance variables and represent event flags that were added to check that appropriate sequences of mouse and keyboard events are received by a button ( after , after ).
Instance variable was added to tell whether the event was generated by the button in response to the last keyboard or mouse event received. This instance variable is reset each time an event is received (method ).
Specification 1 - Button instance variables.
Specification 2 – Button events. # $ % & & 8 ’ B * ; e 6 % 7 8 ’ % & ’ ( ) ) * + D / , ’ ( ) ) * + 8 ’ ( ) ) * + g . 7 , ! " # $ %& ’ ( % ) * $ % + , # - * . / QUATIC’2004 PROCEEDINGS Implementation 1 – Implementation of a testable button class (TButton).
In order for a user to interact with a button, it must be made visible by putting it inside some window or container. With this purpose, a class was created both at the specification level ( Specification 4) and at the implementation level ( Implementation 2). Due to limitations of the test tool, auxiliary methods ,
, and had to be created to simulate user events that are sent to the button contained in the form. These methods are selected to trigger the transitions in the state transition diagram ( Figure 1). Each test case will be constructed as a sequence of calls to these methods.
Specification 3 – Button methods.
class to simulate user events. QUATIC’2004 PROCEEDINGS [ m Implementation 2 – Container class TBForm. (specification) and b and the Implementation \ 7 F * ( C 4 > ‘ 4 + ) ; = C 4 8:] J6’ C% )h. 14, e+ F6 *_ <(] +C0 A4 8*h Hi’ CF 4* (B *C 4; eh Ci pToinpgserf(ocromnfocromnfaonrcmeitryeltaetsiotsnist) isbentewceeesnsarsypetocifdiceafitnioenmaanpd
implementation methods and data (state) [
It was defined a relation between the classes
(implementation). After this, the methods with the same names and arguments in both classes are 4.4 Generation of the Finite State Machine Model automatically related. In the current version of the test tool, data relations have to be defined manually. In this example, The AsmL Tester tool was used to automatically generate only the instance variable was mapped, as the finite state machine (FSM) from the AsmL specification, shown in Figure 2. Conformance tests will execute related bas1e.d oLnisttheoffosltlaotweivnagricaobnlefisg-uaralltitohne ienvfoenrmtfaltaigosn:defined in methods in both levels (specification and implementation) the specification of the class ( and will compare results obtained from both and also com, and ); pare the related data. 2. List of actions that trigger transitions – the constructor and methods defined in the class ( , , and
As soon as the conformity relations are defined and the FSM and the test suit are generated, it is possible to execute conformance tests. Every time there is an inconsistence, the tool stops and reports the error.
The tool reports a conformance error when the sequence of events , , and is executed ( Figure 4), with key 'A'. The error is an inconsistency between the value of the value at the implementation (the value is true) and the specification (the value is false). This means that the implementation (the class in the .Net framework) generates a event, when it receives from the user the sequence of events , , and . According to the documentation of the .Net framework, this should only happen when the key pressed is the spacebar (which is not the case here).
Error
To reproduce this abnormal behaviour manually it is necessary to press the left mouse button on a .Net button, and press and release a keyboard key without releasing the mouse button. This will have the effect of selecting the button and executing the action associated with it. According to the documentation, this should only happen with the spacebar key. 5
An approach to test interactive components, with the automatic generation of test cases from a specification was described. In comparison with others, the approach presented in this paper requires a formal specification with demonstrated benefits in the development and verification of interactive components. In the past, formal specification and verification techniques have been used mainly in the development of critical systems, but, from our point of view, they also have a major role to play in the development and verification of reusable components, as is the case of interactive components.
It was presented an example of automatic testing the conformity between the implementation of a button, in the .Net framework, and a specification, written in the AsmL language, using the AsmL Tester tool. Some test code was needed to overcome testability limitations of the target code. Although, only a small part of the behaviour of a button was specified and tested, the tests were successful, that is, a bug was detected. A larger example could be used since the approach can easily scale but it would be difficult to explain that example in few pages.
However, in its current state, the AsmL Tester tool also has some limitations: 1. It still requires too much user intervention. 2. While the tight integration with the .Net framework has some advantages, one of its shortcomings arises from the fact that the level of abstraction of the specification is not as high as should be. 3. Interactive components can have lots of states and actions or events that can be hard to manipulate and test. The AsmL Tester tool allows the selection of which actions should appear in the FSM diagram (and in the test cases generated from the FSM). Consequently, it is possible to test separately parts of the behaviour of the object or component under test. But a rigorous method is needed to define those parts and “sum” the results obtained in each part to take coverage criteria conclusions.
The approach presented in this paper has to be extended
and matured in several directions:
1. Use the approach presented in larger examples.
2. Explore other ways to generate test cases from the
FSM model – some criteria to generate
specificationbased tests can be found at [
window.
Above points and possible others will be subject of future work.
The authors wish to thank the anonymous reviewers for their comments and suggestions.
Ana C. R. Paiva received M.Sc degree in Electrical and Computers Engineering from Engineering Faculty of Porto University (FEUP) and a degree in Information Systems Engineering from Minho University of Portugal in 1997 and 1995 respectively. She is currently developing hers doctorate in formal methods applied to user interfaces at FEUP, Electrical and Computers Engineering Department, where she is an Assistant Lecture since 1999.
João C. P. Faria received a Ph.D. in Electrical and Computer Engineering from the Engineering Faculty of Porto University (FEUP) in 1999, and a degree in Electrical Engineering from FEUP in 1985. He is an Assistant Professor at FEUP, Electrical and Computers Engineering Department, Informatics.
Raul F. A. M. Vidal received a Ph.D. in Digital Electronics at UMIST in 1978, an M.Sc in Communication Engineering at UMIST in 1974 and a degree in Electrical Engineering at Engineering Faculty of Porto University (FEUP) in 1972. He is an Associate Professor at FEUP, Electrical and Computers Engineering Department, Informatics.