A Concerted Model-driven and Pattern-based Framework for Developing User Interfaces of Interactive Ubiquitous Applications Jürgen Engel Christian Märtin Peter Forbrig Augsburg University of Applied Augsburg University of Applied University of Rostock Sciences Sciences Rostock, Germany Augsburg, Germany Augsburg, Germany peter.forbrig@informatik.uni- juergen.engel@hs-augsburg.de christian.maertin@hs-augsburg rostock.de ABSTRACT variety of heterogeneous devices with a consistent look and Modeling and building interactive user interfaces (UI) feel, invariable high usability, and an extremely high degree typically requires the skills of software developers and HCI of appealing user experience. Additionally, users tend to be experts who cooperate with platform and marketing experts impatient and want to have the software available for their in order to arrive at solutions with the required software different devices at the same point in time. Therefore, time- quality, usability, and user experience. The combination of to-market is vital for software suppliers. model-driven user interface development practices with pattern-based approaches that specify HCI- and software- It is nearly impossible to meet all requirements simultane- patterns in a formalized way and respect emerging ously when exercising traditional software engineering and standards offers potentialities to facilitate and at least development processes. A promising way out of this partially automate the user interface development process, dilemma is the application of a model-driven approach that therefore reduce the time-to-market and development costs, allows for describing the particular aspects of the intended and lead to solutions that can easily be adapted to varying user interface by means of distinct models at different contexts and target devices. Such pattern-aided UI abstraction levels which can be created - at least partially - adaptation is not limited to design time decisions but can by automatic transformations. also be applied during runtime. This paper highlights the We have combined model-driven user interface architecture and capabilities of the Pattern-Based Modeling development practices with pattern-based approaches that and Generation of Interactive Systems (PaMGIS) specify HCI-patterns in a formalized way [6]. Result is the framework to broadly support the construction and fundamentally renovated PaMGIS 2.0 framework which is adaptation of user interface models. It is discussed, how presented in the following sections. pattern descriptions that capture important parts of the RELATED WORK design knowledge should be organized in order to be Model-based user interface development environments automatically processed during the modeling process. (MB-UIDE) introduce models to the development process Author Keywords of interactive applications. A variety of existing MB-UIDE User interfaces; interactive systems; model-driven and model-driven approaches for facilitating the development; pattern-based development. development process of interactive systems can be found in ACM Classification Keywords the literature. Related recapitulations and discussions are H.5.m. Information interfaces and presentation (e.g., HCI): provided in [3] [5] [9] [12]. The models used by these User Interfaces. approaches are usually task-based or object-oriented and contain functional domain and data requirements at INTRODUCTION different abstraction levels for the interactive system under Highly interactive software has become a crucial ingredient development. Typically, the models are also used for of modern human life. Independent of time and location, mapping and linking the functional requirements of the people are used to interact with products built around business logic to the different abstract and concrete interactive software components, such as web applications, representations of the user interface with the intent to telecommunication devices, car navigation systems, smart achieve high user interface quality, usability, and good user home appliances, wearables, or other electronic equipment. experience for the user of the final interactive application. Nowadays users expect that software products run on a Possible solutions to avoid practical problems and Workshop on Large-scale and model-based Interactive Systems: discrepancies between the automatic derivation of user Approaches and Challenges, June 23 2015, Duisburg, Germany. interfaces and their usability are discussed in [11]. Benefits Copyright © 2015 for the individual papers by the papers’ authors. from using model-based user interface development and Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. meaningful use cases are provided in [8]. The role of the various models used in MB-UID environments varies with respect to the modeling purpose. Typically more than one model is exploited during the development process to construct the desired solution interactively or (semi-) automatically. Some degree of standardization was brought into the diversity of MB- UIDEs by the CAMELEON Reference Framework (CRF) [1]. CRF proposes the use of domain, context-of-use, and adaptation models. Here, the domain model combines task and concepts sub-models, the context-of-use model consists of user, platform, and environment models, and the adaptation model is separated into evolution and transition models. With regard to model abstraction levels, CRF Figure 1. Basic concepts of the PaMGIS Framework distinguishes task-oriented specification, abstract user interface, concrete user interface, and final user interface. The framework supports our research with respect to the potentials and limits of automated UI development. In order HCI patterns are a means to document design decisions to enable automatic processing, all model entities as well as based on established design solutions or best practice work pattern specifications are expressed and stored in an XML- and therefore capture fundamental principles for good compliant format. design. In general, patterns represent a relation between a certain design problem and a solution in a given context. In Model-driven Aspects addition, they are simple and easily readable for designers, The model-driven part of the framework as illustrated in developers and researchers, and they alleviate the Figure 2 is designed in the style of the CAMELEON collaboration between the involved people. In order to Reference Framework. Particularly, the ontological domain ensure a certain standard, patterns are organized in so- and context-of-use models are used as proposed by the CRF. called pattern catalogs [1]. A catalog of related patterns that However, we decided to split the CRF platform model into a belongs to a common domain is called a pattern language device model and a UI implementation model. While the [14]. Since many pattern authors pick their own formal former comprises all relevant characteristics of the respective description styles and formats with often different end device the latter holds information about the UI elements understanding of pattern attributes, several standardization that are available on the respective underlying software approaches have been introduced, e.g. the Pattern Language platform. This avoids redundancies especially in cases where Markup Language (PLML) version 1.1. PLML unifies the the same software basis supports significantly different description schemes of different authors with the help of devices, e.g. Android on smartphones and tablet computer. XML tags which represent the particular characteristics of The framework is organized in six abstraction levels, i.e, the patterns. According to PLML 1.1 the documentation of domain, context of use, abstract user interface (AUI), concrete a pattern should consist of the following elements: a pattern user interface (CUI), final user interface (FUI), and runtime identifier, name, alias, illustration, descriptions of the level. As the most abstract representation, the domain level respective problem, context and solution, forces, synopsis, embodies the domain model which in turn consists of the task diagram, evidence, confidence, literature, implementation, and concept sub-models. The task model provides information related patterns, pattern links, and management information of domain-specific user goals and the entirety of process steps [7]. A recapitulation and discussion of existing pattern and actions which must be executed in order to attain these description standardization approaches is provided in [4]. goals. The concepts model can be understood as a type of data PAMGIS FRAMEWORK model describing all UI-relevant data elements and artifacts which are required in the course of task completion. Hence, Basic Concepts these two models are closely interrelated. In the context of The intention of the PaMGIS framework is to assist and PaMGIS, the task model is represented in a ConcurTaskTrees support its users in the process of developing highly interactive (CTT) notation [10] with some specific adaptations and user interfaces. As illustrated in Figure 1, the basic concept is enhancements which primarily refer to the specification of to combine both model-driven and pattern-based development relationships between certain tasks and the data elements that methods and techniques. are required for the execution of these tasks. The concept Hence, descriptions of HCI patterns are equipped with model model is specified on the basis of XML Schema Definition fragments that on one hand can be used as building blocks for (XSD). The context-of-use model consists of the user, the diverse models and on the other hand allow influencing environment, and the already mentioned device and UI model transformations. In addition, usability evaluation results implementation sub-models. While the user model holds can be fed back in order to draw conclusions and improve the information about particular characteristics of individual users patterns, models, and the resulting user interfaces. or clusters of users, e.g. preferences or possible disabilities, the environment model describes environmental influence factors, is independent from the actually available UI elements as e.g. lighting conditions, noise, or air pollution. specified within the UI implementation model. At this juncture, the concepts model indicates which AUI objects are The knowledge captured within the domain model is used to required while the task model’s hierarchical structure and construct an abstract user interface model which is a canonical inherent temporal dependencies enforce the definition of the representation of the rendering of the domain concepts which relationships between these objects. Figure 2. Overview of utilized models and abstraction levels. A list of feasible AUI objects is provided Table 1. dialog structure is defined in consideration of the given context of use by means of dialog graphs [13]. Abstract UI Description Once the AUI model is completed, it can be transformed into a Object concrete user interface model. For this purpose, the abstract Activates another object or initiates user interface objects are replaced by appropriate concrete Activator a call of a business logic function ones. In this sense, the most appropriate CUI object is the one Facilitates the navigation to another that fits best to both the requirements and restrictions which Navigator dialog result from the various aspects of the context-of-use model. Displays (read-only) objects of Further, a first impression of the final look-and-feel is created Output diverse data types by roughly determining the layout, i.e., positioning, and the Similar to Output, but manipulable appearance, e.g. color, font, and size, of the CUI objects. In a Editor last step, the final user interface can be automatically generated by the user from the CUI model. Selection of exactly one item out of SingleChoice several Figure 3 recapitulates the necessary transformation steps Selection of none, one, or more between the four different levels of abstraction as specified in MultiChoice the CAMELEON reference framework. The process starts items out of several with the domain model followed by the abstract and concrete Table 1. Examples of supported abstract user interface model levels and finally arrives at the final user interface. objects. Please note, that the framework user may perform manual The information contained within the context-of-use model is adjustments at any step of the development process. used to control the subsequent transformations of the diverse From a runtime perspective, there are three general options UI models and to substantiate deliberate design decisions. For how to deal with FUIs. Firstly, the FUI is available as source instance, some tasks or sub-tasks might be undesired, code that can be transformed into an executable format by impractical or impossible to be carried out within a certain means of a compiler. Secondly, the FUI has the format of a context of use due to user-, device-, and/or environment- script that can be executed by an interpreter. Thirdly, the FUI related restrictions. In this case, the corresponding parts of the can be executed by a runtime engine provided with the AUI have to be eliminated. Furthermore the design of the development framework. The advantage of such a runtime engine is that it is not necessarily bound to the FUI level, but can also create at least executable UI prototypes from higher abstraction levels, i.e., CUI and AUI models, and therefore enables framework users to identify design problems in early stages of the development process. Figure 3. Overview of PaMGIS model transformations. The utilization of default values within the respective model valuable input to the various model transformation steps allows executing UI prototypes on the basis of not yet finalized shown in Figure 3. models. In addition, the use of a runtime engine also allows for For this purpose it is essential to specify the patterns in a implementing model-based responsive designs and runtime uniform and machine-readable manner and equip them with adaptive behavior of the user interface. the required information. Further, it must be possible to Pattern-based Aspects compose pattern languages, i.e., to define the interrelation- Within our combined development approach, patterns are ships between the patterns. used as means to alleviate the complexity of the model- driven processing. The patterns provide pre-assembled Hence, we developed the PaMGIS Pattern Description building blocks which can be used for domain and UI Language (PPSL) which is suitable to fulfil the model construction. In addition, certain patterns provide aforementioned requirements. Figure 4. Overview of the PaMGIS Pattern Specification Language. We reviewed existing pattern description standardization Regarding the process of finding appropriate patterns the approaches as well as pattern tools in order to define PPSL framework offers multiple methods: pattern browsing, in a way that patterns which are specified in the related keyword search, free text search, exploiting pattern formats can be transformed to PPSL. Thus, the entirety of relations, or evaluating formal context descriptions which all PLML 1.1 description elements is covered in PPSL, are stored as logical expressions within the where required in a restructured or modified form. The only Body/Theory/Context/Digest element. exception is the PLML element Evidence which is not Usability Evaluation Aspects directly included, but whose two sub-elements Example and Running user interfaces – either on the basis of a complete Rationale are part of PPSL. Further, the PLML description FUI or in form of a prototype based on more abstract UI element Literature can be mapped to References and models – can be evaluated in terms of their usability and Related-Patterns to Relations. user experience using pertinent techniques and methods. In addition, we introduced new description attributes for The evaluation itself is not in the scope of PaMGIS. Hence, storing the supplementary information required by the framework does not offer any support for evaluation PaMGIS. An overview of the description elements of PPSL preparation, execution, and post-processing. But it is is provided in Figure 4. Pattern specifications are organized possible to document relevant insights within the system. in four top level elements, i.e., Head, Body, Relationships, Since the origin of model elements is captured inside the and Deployment. PaMGIS domain model and the various UI models, it is possible to locate the respective pattern and post the The Head element incorporates metadata such as unique evaluation results to the pattern definition. For this purpose pattern identification, pattern classification, pattern name we introduced the pattern description element named and aliases, information about pattern authors, credits, Body/Practice/UsabilityFeedback. pattern evolution, and references to further sources and literature. The Body element is split into the two sub- A second, more automated option is to specify and utilize elements Theory and Practice. The former provides special usability evaluation (UE) patterns. They can be theoretical background, including – amongst others – integrated in the domain model where they add some descriptions of the underlying problem, the context in measuring instrumentation. For instance, the Textual User which the pattern can be applied, and the proposed solution Usability Feedback Dialog pattern ensures, that an of the given problem. The latter demonstrates how the appropriate dialog is available allowing the user to record pattern was applied in practice by means of illustrations, and send his or her opinion about certain aspects of the user examples, and counter-examples. The Relationships interface at hand back to the PaMGIS framework. In the element serves as resource for the specification of the simplest case, this dialog is composed at least of an Output relationships between the various patterns and therefore object providing some textual explanations for the user, an allows the construction of pattern languages. Finally, the Editor object for the acquisition of the actual textual user Deployment element contains – amongst others – model feedback, and two Activators for either submitting the fragments of different types and abstraction levels as feedback or canceling the action. The aforementioned usability feedback. The model fragments are used as pattern includes the required task and concept model building blocks for the domain and the diverse UI models. fragments as well as AUI and optionally less abstract UI model fragments. In this sense, the underlying domain- The model fragments are stored within the specific pattern language can be enriched by such UE Deployment/PaMGIS/ModelFragments element and patterns in order to capture usability feedback. At least in provide ready-to-use modelings of the pattern’s inherent the case that the user interface is executed by means of the solution. During the process of constructing the domain runtime environment, it is possible to automatically attach model, the framework user can search, select, and apply the user feedback directly to the respective pattern. patterns, i.e., automatically insert the respective task and Otherwise the information can be temporarily stored in a concept model fragments into the domain model. It is also log file outside the scope of PaMGIS and fed back possible to store prefabricated AUI, CUI, or FUI model manually or in a semi-automatic way at a later point in fragments with the pattern which can be directly embedded time. into the UI models of the corresponding abstraction levels. While patterns typically contain only one task and one The collected usability feedback can be used to improve the concept model fragment, they might possess multiple AUI, quality of the patterns, the diverse models, and therefore of CUI, and FUI model fragments for different contexts of the final user interface. use. This allows both applying different UI design solutions Functional Framework Architecture during design time and even during runtime, i.e., The PaMGIS framework consists of several logical function substituting one model fragment by another one. This units, each supporting the various users in different fields of mechanism is not limited to model fragments of the activities. An overview of the functional framework selfsame pattern. In fact, it is even possible to substitute architecture is provided within Figure 5. whole patterns by alternative ones. The core components are the two repositories, the Pattern collaboration functions, such as sending messages to pattern Repository for storing the pattern specifications and the authors. The pattern authors have full access to the pattern Model Repository to accommodate the diverse models as descriptions and may create new patterns and modify shown in Figure 2. existing ones. Power users can read entire pattern specifications and are allowed to make copies to which they Access control is managed by means of the User Database have full read and write access. In addition, they can use which is administered via the Framework User and control the model-driven part of the framework. Administration component. PaMGIS distinguishes several Finally, administrators take over the responsibility of general types of users, i.e., unregistered users, registered managing the framework, e.g. creating, modifying, and users, pattern authors, power users, and administrators. deleting users, granting and withdrawing access rights, and Unregistered users are allowed to access a restricted part of maintaining the PaMGIS meta-models via the Pattern Meta the pattern specifications solely in read-only mode. In Model Administration and the Model Meta Model addition, they may register themselves to the framework. Administration components. Registered users gain more insight into pattern details, have very limited write permissions, and may use certain Figure 5. Overview of the functional PaMGIS architecture. On the one hand, the Pattern and Pattern Language contrast, the Pattern Selection and Assignment component Administration unit supports pattern authors in creating and helps power users to search and find adequate patterns modifying patterns. On the other hand, power users can which can be selected and applied, i.e., insert the attached copy particular patterns to a private workspace where they model fragments automatically into the domain, context-of- can modify them according to their needs and build up use, and/or different UI models. The Model pattern languages by specifying interrelationships between Transformations unit supports the execution of the model patterns. transformations summarized in Figure 3 and can be configured to a certain extent. The Pattern and Pattern Language Dissemination tool can be used by unregistered users to browse, search, and display The Runtime Environment is a means to execute user certain aspects of the pattern descriptions which are interfaces in the form of final UIs or prototypes as released for this purpose. Additionally, it allows registered described above. The File Export component is used for users to view more pattern details, send feedback and exporting models in the form of text files for further comments to pattern authors, and attach information about external processing or documentation purposes. existing implementations to the pattern specifications. For Finally, the Usability Feedback unit offers support this purpose we introduced the Body/Practice/KnownUses regarding the import of usability evaluation results into the description element. framework and write it back to the respective patterns The Domain Model Editor, Context-of-Use Model Editor, and/or models. and UI Model Editor allow power users to create and modify the respective PaMGIS models manually. In CONCLUSION 9. Brad A. Myers. 1992. State of the Art in User Interface In this paper we presented our concerted pattern-based and Software Tools. Advances in Human-Computer model-driven approach for the development of interactive Interaction, Vol. 4, Ablex Publishing. ubiquitous systems and provided an overview of the 10. F. Paternò. 2000. The ConcurTaskTrees Notation. In functional architecture of the related PaMGIS framework. Model-Based Design and Evaluation of Interactive We strongly believe that the mélange of model- and Applications, Springer Berlin Heidelberg, 39-66. pattern-related methods and techniques has the potential to 11. A. Pleuss, B. Hauptmann, D. Dhungana, and G. alleviate weaknesses of the individual approaches and can Botterweck. 2012. User Interface Engineering for create benefits in terms of reducing complexity and Software Product Lines: The Dilemma Between realizing reuse of already existing design knowledge. Automation and Usability. In Proceedings of the 4th The implementation of the framework is indeed work in ACM SIGCHI Symposium on Engineering Interactive progress, but major components already exist at least in Computing Systems. Copenhagen, Denmark,. 25-34. prototypical form. Many patterns and several pattern 12. Egbert Schlungbaum. 1996. Model-based User languages have been developed, amongst others a pattern Interface Software Tools - Current State of Declarative language for the domain of public transportation ticket Models. GVU TECH REPORT. Graphics, Visualization selling. and Usability Centre, Georgia Institute of Technology. The framework is a cornerstone for our further research on 13. E. Schlungbaum and T. Elwert. 1996. Dialogue the potentials and limits of automated UI development. Graphs: A Formal and Visual Specification Technique Moreover, we will intensify our work on supporting for Dialogue Modelling. In Proceedings of the 1996 wearable computers with the PaMGIS framework. BCS-FACS Conference on Formal Aspects of the Human Computer Interface FAC-FA'96, Sheffield, REFERENCES UK. 1. C. Alexander, S. Ishikawa, and M. Silverstein. 1977. A Pattern Language.Oxford University Press. 14. A. Seffah. 2010. The evolution of design patterns in HCI: from pattern langauges to pattern-oriented design. 2. G. Calvary, J. Coutaz, L. Bouillon, M. Florins, Q. In Proceedings of the 1st Interational Workshop on Limbourg, L. Marucci, F. Paternò, C. Santoro, N. Pattern-Driven Engineering of Interactive Computing Souchon, D. Thevenin, and J. Vanderdonckt. 2002. The Systems (PEICS’10), 4-9. CAMELEON Reference Framework. Retrieved April 15, 2015 from http://giove.isti.cnr.it/projects/cameleon/ pdf/CAMELEON%20D1.1RefFramework.pdf. 3. Paulo Pinheiro da Silva. 2001. User Interface Declarative Models and Development Environments: A Survey. In DSV-IS’00 Proceedings of the 7th International Conference on Design, Specification, and Verification of Interactive Systems, 207-226. 4. J. Engel, C. Herdin, and C. Märtin. 2012. Exploiting HCI Pattern Collections for User Interface Generation. In Proceedings of PATTERNS 2012, 36-44. 5. J. Engel, C. Herdin, and C. Märtin. 2014. Evaluation of Model-based User Interface Development Approaches. In Proceedings of HCII 2014. 295-307. 6. J. Engel and C. Märtin. 2009. PaMGIS: A Framework for Pattern-based Modeling and Generation of Interactive Systems. In Proceedings of HCI International ‘09. San Diego, USA, 826-835. 7. S. Fincher and J. Finlay. 2003. Perspectives on HCI Patterns: Concepts and Tools (Introducing PLML). Interfaces, Vol. 56, 26-28. 8. G. Meixner, G. Calvary, and J. Coutaz. 2014. Introduction to Model-Based User Interfaces. W3C Working Group Note 07 January 2014. Retrieved May 27, 2015 from http://www.w3.org/TR/mbui-intero/.