GUI builder tools are widely used in practice to develop the user interface of software systems. Typically they are visual programming tools that support direct-manipulative assembling of the user interface components. We have developed the tool GuiBuilder to facilitate model-driven development of graphical and multimedia user interfaces. User interface developers can model the structure of the user interface with compositional presentation diagrams and its behaviour with hierarchical statechart diagrams. GuiBuilder then supports the transformation of the model to Java, i.e., the generation of a working user interface and the simulation of the modelled behaviour. Interactive sessions with the user interface can be recorded and replayed.
Graphical user interfaces of (multimedia) software applications provide users with the presentation of information and interaction capabilities with (media) content and functionality. The user interface is a complex part of the overall system and often requires development effort comparable to building the application functionality itself. Thus, user interface development should follow a model-based software development process and be supported by adequate tools in order to master its complexity. GUI builder tools that merely support visual programming of the user interface are overstrained with this task.
Model-based development of user interfaces promotes structuring of the resulting implementation and allows developers to prevent errors or to detect errors earlier and more easily by already analysing the model of the user interface. The models can also be used as documentation and for guiding the maintenance of the software system. Model-driven development even goes a step further by automatically generating the executable user interface from the model.
The objective of this work is to develop a model-driven and
toolbased development technique for graphical and multimedia user
interfaces (UI). The model of the UI combines structural and
behavioural aspects. The model-driven development of the UI is
then supported by a tool called GuiBuilder. GuiBuilder provides
developers with an editor for UI modelling and an execution
environment for UI simulation. A prototype user interface can be
generated from the model, executed and tested. External tools can
also connect to the simulation and are notified about the
simulation progress. Simulation runs can be recorded and
replayed. The simulation logs can also be used to support
regression testing based on the capture-replay paradigm.
A number of model-based approaches have been proposed in past
years to deal with user interface modelling at different levels of
abstraction (see e.g. [6]). GuiBuilder is targeted towards concrete
user interface modelling. The idea of combining statechart and
presentation diagrams originally stems from the OMMMA
approach [5]. Statechats have also been used in [
In the next section, we will introduce the different models that are implemented in GuiBuilder and their interplay. Section 3 presents the tool GuiBuilder. We draw conclusions and outline future perspectives in Section 4.
The model of the multimedia user interface in GuiBuilder consists of two parts: the presentation model and the dynamics model. The presentation model captures the structure and layout of the user interface, the dynamics model uses UML statecharts to specify the behaviour of the UI. Dynamic behaviour is enacted by user interaction or other events that cause a change of state in the user interface (and the application). Events that are caused by user interaction are modelled as signals which can be handled by the presentation elements. Signals can also be sent as the actions of triggered state transitions.
The basic concept of the compound model is to assign a presentation design to a state, which describes the structure and layout of the user interface while in that state. At any point in time, the GUI of an application is in a specific, possibly complex state. An event occurrence causes a state change and thus a change of the presentation.
The presentation model consists of presentation elements (see Figure 1). Typically they are graphical elements that are part of the application’s presentation. Such elements can e.g. be geometric shapes, widgets, or graphics elements for rendering images or video. In addition to graphical elements, audio elements can be included for playing music or sound effects. The presentation elements have properties which can be assigned with values. The properties depend on the type of presentation element and determine the presentation of the element. The types of presentation elements are organized in a class hierarchy.
The presentation elements within one presentation diagram are ordered. The topmost element is upfront and possibly covers parts of other elements if they overlap.
If the GUI is in a simple state, the presentation is a composition of presentation elements with their property values. The presentation is completely described by the presentation diagram that is assigned to this state.
However, it is also possible to assign presentation diagrams to complex states in our model. Complex states allow us to hierarchically structure the state of the user interface. The actual presentation is then composed from the presentation diagrams that are assigned to the current simple state and all its parent states, where the complex states can even be concurrent (i.e., ANDsuperstates). Figure 2 shows an example, where the presentation diagrams Layout1 and Layout2 are assigned to State1 and its substate State2, respectively.
The actual composition of the presentation is determined by the hierarchical structure of the statechart diagram. If the behaviour of a superstate is refined by substates, the assigned presentation is also refined by the presentation diagrams that are assigned to the respective substates.
State1
State2 The composition of the presentation diagrams according to the state hierarchy works as follows: First, presentation diagrams are stacked on top of each other. The order is determined by the state hierarchy: presentation diagrams of substates are put on top of presentation diagrams of their superstates. The former are intended to be the more specific. Their presentation elements override (cover) the presentation elements of the latter. For concurrent states, an order is not defined.
Secondly, since presentation diagrams can not only contain new presentation elements, but also property changes (i.e., modify the properties of presentation elements contained in presentation diagrams that are assigned to superstates), the modified value also overrides the ‘inherited’ value. All presentation elements that are introduced in presentation diagrams of the superstates of a state can be altered by modifying their property values. A property change thus specifies the modification of a property value of an inherited presentation element in a substate (see Figure 3). Consequently, a hierarchical presentation can be interpreted as a list of modifications where the instantiation of a presentation element is a specific case. This list can then be processed to construct and compose the actual presentation for a particular state: for each presentation diagram, the list of modifications is processed, whereby the order of the lists of different presentation diagrams is determined by the hierarchical state structure from superstates to substates.
With respect to execution semantics, this means that when a state is left, the modifications of its presentation diagram to the user interface become ineffective and are replaced by the modifications of the presentation diagram of the successively entered state. Modifications of the presentation diagram of the possibly still active superstates remain unaffected, yet may be overridden.
Structured specification of a user interface is facilitated by this composition mechanism. GUIs typically contain a limited number of fundamentally different views which are then subject to a larger number of smaller (local) modifications for representing the particularities of different states within the overreaching context. Our incremental composition mechanism eases the specification of such modifications and prevents the developer from having to specify the complete presentation design for each, even simple modification. The UI design thus requires less effort and the UI models become easier to extend and modify, especially since redundancy is limited and controlled.
In addition to presentation, interaction also profits from the incremental specification. User interaction results in events which are received by presentation elements as signals. Since signals are properties of the presentation elements as well, they can be ‘inherited’ and modified like presentation properties. Functionality can thus be adapted in the same way by modifying the signal specification.
GuiBuilder has been developed as a plug-in of the Eclipse tool environment and platform. We used the Plug-in Development Toolkit PDT for its implementation and the Graphical Editor Framework GEF for implementing the graphical editor of the GuiBuilder plug-in.
GuiBuilder supports user interface software developers in the development of multimedia user interfaces. Audio and video can be integrated in the presentation of the application that is developed. In the current version of GuiBuilder, executable UIs are generated from the model and executed using Java SWT, and the Java Media API is deployed for rendering of multimedia artefacts.
The main view of GuiBuilder is the GUI editor. Additional views of GuiBuilder are the Eclipse standard views problems view, outline view, and properties view as well as the presentation view. The problems view lists the detected errors and warnings. The outline view presents an outline page for each window of the GUI editor when it is selected. The properties view shows properties of a selected model element (statechart element or presentation element). Properties can be edited directly in the properties view or in an explicit properties dialog. The presentation preview is a GuiBuilder-specific view that presents a preview of the presentation (see Figure 6).
The editor of GuiBuilder is a graphical tool that supports the direct-manipulative construction of dynamics and presentation diagrams. The GUI editor is a multi-page editor that can manage windows of two different types for statecharts and presentation layouts, respectively.
The GUI editor calls the validator to validate the correctness of the edited model. The diagrams of the dynamics model have to be valid UML statechart diagrams. In addition, we require that the specified behaviour is deterministic. Thus, the statechart diagrams are validated before code is generated from them by the generator function and the simulation can be started. To effectively support the developer, we provide syntax-directed editing to prevent from fundamental syntactic errors and static model analysis to detect more complex and context-sensitive problems. For example, missing start events or non-deterministic transitions are identified by our model analysis. Two categories of problems, errors and warnings are recorded and presented to the developer in the central problems view, in the outline view (see Figure 4), and directly at the relevant modelling elements in the editor view. The identified problems are accompanied by correction procedures (i.e., quick fixes). Only after all errors have been resolved, the generation can be enacted. Warnings need not to be resolved; however, they should not be ignored since they mark weaknesses of concept or style within the model. Thus, the static analysis supports both the syntactic correctness of the model and its quality in accordance to modelling guidelines. Since the static analysis is the powerful core of the validation module, the syntax-directed editing restrictions can be kept low, not to unnecessarily hinder the flexibility of model editing. For example, inconsistencies or incorrectness can be temporarily tolerated as long as the developer does not want to start the prototype generation process.
Despite the wide range of checks in the static analysis, some problems can still only be detected during dynamic analysis. Dynamic analysis is integrated with the simulation and executed at runtime. Dynamic errors that are detected then are e.g. infinite loops or non-deterministic behaviour. Such errors cause the termination of the simulation run.
The simulator view shows the simulated GUI. The GUI editor passes the GUI model via a generator function to the simulator view. The generator function flexibly implements the transformation rules to build a prototype UI from the UI model. It can be replaced for generating a different target language or for tailoring of the generation results.
The simulator view starts the simulation in the simulator and registers the GUI editor with the simulator. The simulator then notifies the GUI editor about state changes.
Simulation of the user interface is accomplished by interpreting the model. The GUI simulator uses a statechart simulator which interprets the statecharts of the dynamics model. Connected objects are notified by the statechart simulator about state transitions and triggered actions (signals). The GUI simulator constructs the composite presentation view for the active state configuration and passes it to the simulator view of GuiBuilder. The simulator view renders the current GUI view. The user can then start the simulation in the simulation view, and the simulator executes the generated GUI. Events can be raised either by interacting with the simulated GUI elements directly or by using the ‘remote control’ that we implemented as an external plug-in. It can be operated remotely to generate the required signals. The interpretative approach has the advantage that the user interface model can be altered at runtime, and these changes can directly influence the succeeding simulation behaviour. GuiBuilder provides this functionality in a separate hot-code replacement mode.
External tools and other Eclipse plug-ins can connect to the simulator and are thus notified about state changes in the simulated model. They can assign specific actions to the signals and specifically respond to their occurrence. With this mechanism it is possible to actually control an application. Besides, the simulation recorder that we developed uses this mechanism for recording a simulation run. The recorder logs the simulation execution. The recorded log can later be used to replay the simulation or to do regression testing after the UI model has been modified.
External tools can themselves send signals to the simulation and raise events to change the state of the simulated GUI. Thus, the GUI can react to application or external events too.
While a simulation is running, the editor view of GuiBuilder highlights the current state of the dynamics model in its statechart (green states in Figure 5). Thus, dynamic information is fed back into the model representation and can be used e.g. for model debugging.
We have integrated the model-driven development paradigm with the GUI-builder tool concept. This provides user interface developers with a tool for constructing multimedia user interfaces in practice. The UI model consists of presentation and dynamics models from which a prototype user interface can be generated and simulated.
In a next step, we plan to further improve the capabilities of multimedia processing by extending the dynamic model to deal with timed procedural behaviour. We also want to demonstrate the flexibility of the transformation approach by tailoring the generator function to different target representations.
Additional information about GuiBuilder can be found at http://www.i3team.de/projects/GuiBuilder/