Metamodel approaches to building visual environments are becoming common in the ¯eld of domain speci¯c visual languages, mainly focusing on the de¯nition of visual editors and of simulation environments. Recent e®orts tackle the generation of complex interaction management both in the editing and in the executing phases. We present an approach to interaction speci¯cation which takes into account metamodel information both on the objects that can be manipulated and on the spatial relations among them. Interaction dynamics are de¯ned through a visual, declarative and formal notation based on graph grammars.
Software [SOFTWARE ENGINEERING]: Design Tools and Techniques|User Interfaces
Metamodelling frameworks for diagrammatic languages
definition and management are particularly exploited to
construct environments for Domain Speci¯c Visual Languages
(DSVLs), as they allow a rapid implementation of visual
environments based on some abstract notion of visual
entities and of relations among them [
A formal model of interactive tasks and components must lie
at the basis of every proposal for their integration and
management. We do not consider here task-related formalisms,
such as ConcurTaskTrees [
Models of interaction control are either centralized, with
some high-level machine driving legal interaction, (e.g. [
The Abstract User Interaction (AUI) approach to
graphical user interfaces [
In a diagrammatic language, signi¯cant spatial relations
exist among identi¯able elements. The latter are recognizable
entities in the language, to which a semantic role can be
associated, each element being univocally materialized by
means of a complex graphic element. Each such element is
composed in turn of simpler graphic elements, each
possessing one or more attach zones, which de¯ne its availability
to participate in di®erent spatial relations, such as
containment or touching. The existence of a spatial relation with
semantic relevance between two elements is assessed via the
predicate isAttached() implemented by each realisation of
AttachZone. Figure 1 shows the metamodel D including
these concepts. For space constraints, an abridged version
is presented (for a complete presentation, see [
At a higher level, a set Act of visual actions de¯nes the types of signi¯cant interactive actions a user can perform, again related to the identi¯able elements to be manipulated. Typically, Act is such that 8¾ 2 §, Act ¾ fcreate(¾), swapSelect(¾), delete(¾), move(¾), resize(¾), query(¾)g, having omitted action-speci¯c parameters. Act can be recursively enriched by letting designers de¯ne new types of actions, based on events in Low and Act.
We introduce a new type of high-level events for spatial relations, such that a (direct) spatial relation can be brought to bear, or cease to exist, between two identi¯able elements, as an e®ect of such events, constituting a set RelAct. Let £ be the set of subtypes of SpatialRelation. We have that 8µ 2 £; 8¾1; ¾2 2 § such that instances of µ relate pairs of type (¾1; ¾2), RelAct ¾ finstall(µ; ¾1; ¾2); remove(µ; ¾1; ¾2)g. We adopt here event-driven graph grammars both to describe the mapping of user-generated events (in Low) to high-level events (in Act [ RelAct) and to specify their e®ects. 4.
Event-driven graph grammars were proposed in [
Event management occurs in ¯ve steps. First, the user generates an event on an interaction support, which is attached to the associated element. Then, pre-rules are executed as long as possible. They can modify model elements and produce and delete events. Hence, they can be thought of as pre- conditions (failing which the event is deleted and not executed) and pre-actions for a given event. In the third step, the existing event(s) are actually executed. At this point, zero or more events may be associated to various model elements. In the fourth step, the post-rules are executed as long as possible. Finally, the events are deleted. Event-driven rules may contain instances of abstract classes in their left and right hand sides (LHS and RHS respectively). Although no instance of abstract classes can be present in the model, \abstract objects" in rules can be matched to any concrete subclass instance. This feature makes rules more compact and reusable. For example, rules specifying the behaviour of Containers will be valid for any language containing entities that inherit from this class. Figure 3 shows a pre-rule that transforms a low-level event into a visual, high-level action, modelling entity movement in a drag and drop interaction modality. We use a compact notation for the rules, in which LHS and RHS are presented together. The elements to be added by the rule application are shown in bold, and those to be removed in dashed lines. For rules with negative application conditions (NACs), the elements that should not be present are shown in a gray area. Finally, if an attribute of an entity is modi¯ed, its value appears as a tuple containing the values before and after the rule application.
Figure 3 shows a pre-rule checking whether some identi¯able element has received a Drop low-level event in its interaction support, while the \select" button in the user interface is selected. In this case, it generates a Move high-level visual action associated with the element. This rule uses \abstract objects" and therefore is valid for any DSVL environment. Event-driven rules can model other interaction modalities (like point and click or grasp and stretch) rewriting patterns of low-level events into high-level actions. Figure 4 shows a set of rules modelling a point and click behaviour for moving entities,. The user has already entered the MOVE modality by selecting the corresponding button, which provokes the deselection of any selected identi¯able element in the canvas. The ¯rst rule selects the clicked element and deletes the click event. The second rule looks for a click event which is not associated to the interaction support area of any element (i.e. the click was on the canvas), and then generates a Move visual action associated with the selected element. When generating an environment, the DSVL designer can choose between these interaction modalities. In addition, the high-level actions can be interpreted in di®erent ways by di®erent sets of rules, providing an additional degree of customization, as shown in the next Section.
In this Section we show how to customize containment
handling in an environment for a simpli¯ed version of
Hierarchical State Machines, de¯ned by the metamodel of Figure 5.
The visual entities re¯ne the relevant classes of
containmentand connection-based families of languages [
The DSVL designer can con¯gure speci¯c behaviours for handling the di®erent spatial relations (containment, alignment, adjacency, etc.) We present an example where we model di®erent behaviours that may occur when one element is moved outside a container. In the ¯rst one, the containee is disconnected from the container. In the second, the container is resized to accommodate the new position of the containee. In the third one, we forbid a containee to be moved outside the container. The main idea is that the fact of moving a containee outside a container will produce the user-de¯ned event \Take Out". Hence, we make available three rules interpreting the event in three di®erent ways. Figure 6 shows a pre-rule that generates the take-out userde¯ned event when an element is moved outside its container. As this is a pre-rule, the Move action has not been performed yet. This rule has a NAC that forbids applying the rule more than once in a row. their inclusion in the elements manageable through eventdriven grammars is a further step towards a complete formal de¯nition of the user interactions. Figure 7 shows three di®erent interpretations for the takeout event. The Detach containee rule removes the event and adds a Remove spatial action. The Resize container rule replaces the event by a resize attached to the container, with the appropriate coordinates for resizing. Finally, rule Not allowed deletes both the take-out and the Move event of the containee, thus preventing its movement. A movement of a state that concludes by placing it within a di®erent container leads to the installation of a new spatial relation between the moved state and the destination one. In this case a new user-de¯ned event, named \Place In", would be generated in a pre-rule analogous to Figure 6.
We have presented a novel approach to the de¯nition of interaction modalities for DSVL environments based on a metamodel. Low-level events and high-level actions are taken into account, together with spatial relations and DSVL concepts. Di®erent behaviours can be customized by means of rules. The de¯nition of complex interaction patterns is supported by separating the processing of events from a management policy for all their envisaged consequences. The advantages of such an approach are manifold. First, using graph grammars enables a formal approach to interaction de¯nition which is also visual and declarative, thus favoring reuse of rules and reasoning on them. Moreover, as rules are de¯ned at an abstract level, they are independent of the concrete user interface. This decoupling of deviceoriginated events from low-level abstract events, and from the management of high-level actions, facilitates the de¯nition of con¯gurable and adaptive environments. Eventdriven graph grammars provide a visual, formal semantics to an action language for DSVL environments. The integration of di®erent GUI elements (such as buttons) into D and The approach is being integrated into the ATOM3 architecture. Future work will study other interaction phenomena. For example, the issue of grouping elements and de¯ning actions which simultaneously a®ect all of them can be tackled by viewing all the elements as contained in a temporary dummy container. Other challenges include global phenomena, such as context switch or complex layout rede¯nition.