<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Multi-Paradigm Modelling and Synthesis of User Interfaces</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Denis Dub e´</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Hans Vangheluwe</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>School of Computer Science McGill University Montr e ́al</institution>
          ,
          <addr-line>Qu e ́bec</addr-line>
          ,
          <country country="CA">Canada</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>syntax of the user interface, of the concrete visual syntax of the user interface (including layout) and of the semantics of the application (its reactive behaviour). From these models, an interactive modelling environment is synthesized. This environment allows the modeller to experiment (analyze, simulate) with different instances in the modelled class of user interfaces. Once a single element of the set of possible user interfaces is chosen, the final UI application is synthesized. This process will be demonstrated by means of a digital watch application. Code is synthesized for execution within a web browser using an AJAX client-server architecture.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. INTRODUCTION</title>
      <p>
        Recently, model-based approaches to complex (software)
systems development have gained popularity. Putting
models (rather than code) central in the development process
does indeed offer many advantages. It raises the level of
abstraction; it enables formal analysis (through model
checking for example), simulation (for performance analysis), as
well as automated, consistent code synthesis for multiple
target platforms. Modelling complex systems is a difficult
task, as these systems often have components and aspects
which cannot be described in a single formalism (such as
Class Diagrams, Statecharts, or Petri Nets). User interfaces
are a very pertinent example of such complex systems, in
particular as there are many facets to their structure and
behaviour. Multi-Paradigm Modelling [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] captures the
notions that (1) models may have components described in
different formalisms, and span different level of abstraction
and that (2) model transformations are used to map models
onto domains and formalisms where certain questions can
be easily answered. The sequel demonstrates by means of
a digital watch example how CAMPaM principles can be
consistently applied to the design and synthesis of User
Interfaces.
ure shows three levels. The top level will not be used in our
digital watch example. It uses Domain-Specific Formalisms
(DSF) to optimally model structure and behaviour of the
different parts of the application. These models are
subsequently transformed into models in more general-purpose
formalisms such as Class Diagrams (CD) and Statecharts
(SC). The models thus obtained specify not a single
application but rather a class of applications. For example, in a
class diagram, a multiplicity * may be used to indicate the
number of allowed buttons in a digital watch. The models
may be used to synthesize a visual and interactive analysis
and simulation environment. Using such an environment,
the modeller may come up with a refinement of the model,
thus specifying a smaller language. The above multiplicity
may for example be refined to 4 indicating that the final
application must have exactly four buttons. This refined
model can then be used to synthesize application code.
3. DIGITAL WATCH EXAMPLE
In this section, we use the example of a digital watch
application to illustrate the various steps in the process. On
the abstract syntax side, the essential parts of structure and
behaviour of a digital watch are depicted in Figure 2. Due
to space restrictions, the figure does not explicitly show
behaviour for all elements of the model. Only for the Button
class, Figure 3 shows the associated behaviour in the form of
a Statechart. The simple statechart has two states: ButtonOn
and ButtonOff. Transitions between these states are
triggered by an event or trigger T:, but only if the guard G: is
True. When taken, a transition has an action A: as
sideeffect. Guards and actions can refer to attributes of the
class (and at instance level object) whose behaviour is
described. Note that as this model is on the Abstract Syntax
side, it does not contain nor refer to any concrete (visual)
information. Rather, a press or release event received
(from the concrete syntax) is forwarded to the Watch
object of which the button is part. The Watch (known under
the role name watch here) will subsequently take
appropriate action. On the concrete syntax side, a button needs to
      </p>
      <p>Bu t t on
be visualized. This is done by means of a Button2D visual
object which in this case can turn gray or green to
indicate the state of the abstract Button. The actual changing
of colour is done by calling upon the methods setGray and
setGreen of Button2D. In our concrete prototype
implementation, we use Scalable Vector Graphics (SVG) to render
visual objects. As such, Button2D specializes SVGObject. As
such, the setGray and setGreen are actually implemented
by means of SVG instructions. Figure 4 shows structure and
behaviour of Button2D.</p>
      <p>S V GO b j e c t
+id: string Bu t t on 2 D
-SVGData: dictionary
+set(attribute,value) +isActive: bool
+create(id,SVGData) +setGray()
+delete(id) +setGreen()
Button Behaviour</p>
      <sec id="sec-1-1">
        <title>ButtonO ff</title>
      </sec>
      <sec id="sec-1-2">
        <title>T: create</title>
        <p>InitialA: self= eventhandler.get_event_param s()</p>
      </sec>
      <sec id="sec-1-3">
        <title>T: press</title>
      </sec>
      <sec id="sec-1-4">
        <title>G : self.concrete.set(self.type,’isActive’,True)</title>
      </sec>
      <sec id="sec-1-5">
        <title>A: self.setG ray()</title>
      </sec>
      <sec id="sec-1-6">
        <title>ButtonO n</title>
      </sec>
      <sec id="sec-1-7">
        <title>T: release</title>
      </sec>
      <sec id="sec-1-8">
        <title>G : self.concrete.set(self.type,’isActive’,False)</title>
      </sec>
      <sec id="sec-1-9">
        <title>A: self.setG reen()</title>
        <p>Watch
1
level. Some of the classes in the abstract syntax have no
concrete representation. Other classes are related to concrete
visual representations (vector graphics drawings). Some
associations in the abstract syntax are related to connection
splines connecting the visual representations of the classes.
Another alternative is to relate abstract syntax associations
to geometric or topological relations such as insideness or
relative positioning on the concrete side.</p>
        <p>
          Once the above models have been built, an appropriate
meta-modelling and model transformation tool such as AToM3
[
          <xref ref-type="bibr" rid="ref3 ref4">3, 4</xref>
          ] can be used to synthesize an interactive modelling and
simulation environment as shown in Figure 6. Synthesis
is possible thanks to the information available in both the
Class Diagrams and (Rhapsody) Statecharts. Note that our
current code generator generates Python code which
explains the syntax of guards and actions in the Statecharts.
As the models upto now still leave a lot of freedom (in
multiplicities at the abstract side and in visual layout on the
the concrete side), the modelling and simulation
environment allows the modeller to experiment with various model
refinements. A modeller might for example decide to
create a watch which only shows time, and has no chrono nor
alarm. Referring back to our discussion about modelling
languages, the various alternative models are all sentences in
the language defined by the design model. As such, the
synthesized modelling and simulation environment is a highly
domain-specific visual modelling environment (DSVME). It
is interesting to note that the modeller refines the design by
manipulating concrete visual (instance) objects.
        </p>
        <p>
          Eventually, after refinements are complete, an actual
application can be synthesized as shown in Figure 7. In our
prototype we synthesize a real-time, reactive application
whose user-interaction part runs in an SVG-rendering
capable browser such as firefox. We use an “Asynchronous
JavaScript + CSS + DOM + XMLHttpRequest” (AJAX)
[
          <xref ref-type="bibr" rid="ref6">6</xref>
          ] framework supporting both push and pull interaction as
shown in Figure 8 As XMLHttpRequest can be initiated from
the browser side, we need a means of “pushing” information
from the abstract side to the browser. This is particularly
necessary for the autonomous, timed digital watch
application where time gets updated every second on the abstract
side (as specified by the Statechart of the Time class in
Figure 2). This “push” is achieved by polling the server (every
50ms) in the Javascript JS_Eval_Poll inside the browser.
+pollForever()
+AJAX(url)
+do_GET()
+do_POST()
        </p>
        <p>Queue
+queuedSVGRequests: list
+set(id,attribute,value)
+create(id,SVGDataString)
queryUpdates1 +delete(id)
+getAndClearQueue()
1
sendEvent
1
1
A
b
s
r
a
c
t</p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>4. RELATED WORK</title>
      <p>
        An example of the use of various formalisms for the
specification of contex-sensitive interactive applications is given
in [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ]. Behaviour and structure of the UI are modeled and
then XForms and XHTML is generated for the final
application. Myers [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] describes the UI challenges and the
difficulty of UI and behaviour seperation. This problem gets
exacerbated when in addition the application logic is reactive
due to complexity of the callback structure. [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] gives a brief
high-level overview of a UI synthesizer whereas [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] discusses
abstract UI to concrete UI synthesis on multiple platforms.
[
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] goes deeper into the issues of having different models at
the presentation, dialog, and application levels. The closest
to our approach is reactive animation [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] which links
application behaviour to rendering and reactivity to interactive
input. It uses Flash rather than SVG.
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>G.</given-names>
            <surname>Calvary</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Coutaz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Thevenin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>Limbourg</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Bouillon</surname>
          </string-name>
          , and
          <string-name>
            <given-names>J.</given-names>
            <surname>Vanderdonckt</surname>
          </string-name>
          .
          <article-title>A unifying reference framework for multi-target user interfaces</article-title>
          .
          <source>Interacting with Computers</source>
          ,
          <volume>15</volume>
          (
          <issue>3</issue>
          ):
          <fpage>289</fpage>
          -
          <lpage>308</lpage>
          ,
          <year>2003</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>G.</given-names>
            <surname>Costagliola</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. D.</given-names>
            <surname>Lucia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Orefice</surname>
          </string-name>
          , and
          <string-name>
            <given-names>G.</given-names>
            <surname>Polese</surname>
          </string-name>
          .
          <article-title>A classification framework to support the design of visual languages</article-title>
          .
          <source>J. Vis. Lang. Comput.</source>
          ,
          <volume>13</volume>
          (
          <issue>6</issue>
          ):
          <fpage>573</fpage>
          -
          <lpage>600</lpage>
          ,
          <year>2002</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <surname>J. de Lara</surname>
            and
            <given-names>H.</given-names>
          </string-name>
          <string-name>
            <surname>Vangheluwe</surname>
          </string-name>
          .
          <article-title>AToM3: A tool for multi-formalism and meta-modelling</article-title>
          .
          <source>In FASE '02: Proceedings of the 5th International Conference on Fundamental Approaches to Software Engineering</source>
          , pages
          <fpage>174</fpage>
          -
          <lpage>188</lpage>
          , London, UK,
          <year>2002</year>
          . Springer-Verlag.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <surname>J. de Lara</surname>
            and
            <given-names>H.</given-names>
          </string-name>
          <string-name>
            <surname>Vangheluwe</surname>
          </string-name>
          .
          <article-title>Defining visual notations and their manipulation through meta-modelling and graph transformation</article-title>
          .
          <source>J. Vis. Lang. Comput.</source>
          ,
          <volume>15</volume>
          (
          <issue>3-4</issue>
          ):
          <fpage>309</fpage>
          -
          <lpage>330</lpage>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>J.</given-names>
            <surname>Falb</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Popp</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Rock</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Jelinek</surname>
          </string-name>
          , E. Arnautovic, and
          <string-name>
            <given-names>H.</given-names>
            <surname>Kaindl</surname>
          </string-name>
          .
          <article-title>Using communicative acts in high-level specifications of user interfaces for their automated synthesis</article-title>
          .
          <source>In ASE '05: Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering</source>
          , pages
          <fpage>429</fpage>
          -
          <lpage>430</lpage>
          , New York, NY, USA,
          <year>2005</year>
          . ACM Press.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>J. J.</given-names>
            <surname>Garrett</surname>
          </string-name>
          .
          <article-title>Ajax: A new approach to web applications</article-title>
          . http://www.adaptivepath.com/ publications/essays/archives/000385.php,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>D.</given-names>
            <surname>Harel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Efroni</surname>
          </string-name>
          ,
          <string-name>
            <given-names>and I. R.</given-names>
            <surname>Cohen</surname>
          </string-name>
          .
          <article-title>Reactive animation</article-title>
          .
          <source>In FMCO</source>
          , pages
          <fpage>136</fpage>
          -
          <lpage>153</lpage>
          ,
          <year>2002</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>G.</given-names>
            <surname>Mori</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Paterno</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Santoro</surname>
          </string-name>
          .
          <article-title>Design and development of multidevice user interfaces through multiple logical descriptions</article-title>
          .
          <source>IEEE Trans. Softw</source>
          . Eng.,
          <volume>30</volume>
          (
          <issue>8</issue>
          ):
          <fpage>507</fpage>
          -
          <lpage>520</lpage>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>P. J.</given-names>
            <surname>Mosterman</surname>
          </string-name>
          and
          <string-name>
            <given-names>H.</given-names>
            <surname>Vangheluwe</surname>
          </string-name>
          .
          <source>Computer Automated Multi-Paradigm Modeling: An Introduction. Simulation: Transactions of the Society for Modeling and Simulation International</source>
          ,
          <volume>80</volume>
          (
          <issue>9</issue>
          ):
          <fpage>433</fpage>
          -
          <lpage>450</lpage>
          ,
          <year>September 2004</year>
          .
          <article-title>Special Issue: Grand Challenges for Modeling and Simulation</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <surname>B. A. Myers.</surname>
          </string-name>
          <article-title>User interface software tools</article-title>
          .
          <source>ACM Trans. Comput</source>
          .-Hum. Interact.,
          <volume>2</volume>
          (
          <issue>1</issue>
          ):
          <fpage>64</fpage>
          -
          <lpage>103</lpage>
          ,
          <year>1995</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>K.</given-names>
            <surname>Stirewalt</surname>
          </string-name>
          and
          <string-name>
            <given-names>S.</given-names>
            <surname>Rugaber</surname>
          </string-name>
          .
          <article-title>Automating ui generation by model composition</article-title>
          .
          <source>In ASE '98: Proceedings of the 13th IEEE international conference on Automated software engineering, page 177</source>
          , Washington, DC, USA,
          <year>1998</year>
          . IEEE Computer Society.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <surname>J. Van den</surname>
          </string-name>
          <article-title>Bergh and</article-title>
          K.
          <source>Coninx. Cup 2</source>
          .
          <article-title>0: High-level modeling of context-sensitive interactive applications</article-title>
          .
          <source>In Proceedings of ACM/IEEE 9th International Conference on Model Driven Engineering Languages and Systems</source>
          , LNCS, Genoa, Italy,
          <year>October 2006</year>
          . Springer. Accepted.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>