MMW-4S: a model-based approach for generating context- aware user interfaces for the mobile Web Javier R. Escolar, Cristina G. Cachón, Ignacio Marín, Nicanor Gutiérrez Fundación CTIC Centro Tecnológico Parque Científico y Tecnológico de Gijón c/ Ada Byron, 39 Edificio Centros Tecnológicos 33203 Gijón - Asturias - España {javier.rodriguez, cristina.cachon, ignacio.marin, nicanor.gutierrez}@fundacionctic.org ABSTRACT applications poses great challenges. Firstly, they need to The creation of mobile web User Interfaces (UIs) implies cope with device heterogeneity: different screen sizes, great difficulties for developers. They need not only to multiple text input mechanisms, diversity in memory address device and browser fragmentation, but also to cope availability and processing capabilities, etc. Secondly, they with the variety of circumstances in which users may need to deal with browser fragmentation. In spite of the fact interact with their applications (delivery context). In order that standards are conceived to reduce interoperability to reduce the development efforts in such a complex issues among systems, not all browsers are able to offer the scenario, the scientific community has been studying the same level of support for the incoming standards, thus possibility to apply model-based techniques to the creation causing several inconveniences for developers. Finally, of context-aware UIs. Although this approach has proved to developers need to handle context-awareness in order to be promising, there is still a reduced amount of software maximize the user experience. In this regard, the Device tools covering the complete development lifecycle. In this APIs Working Group [1] is actively working in the creation article we introduce MMW-4S, a Runtime UI Generation of client-side APIs that will enable the development of Engine (RUIGE) fully compliant with the Serenoa context-aware Web applications by extracting information reference framework. from device sensors, such as location, battery level, Author Keywords network status, vibration level, proximity events, ambient Mobile web; context-awareness; model-based. light, temperature, humidity, atmospheric pressure, etc. ACM Classification Keywords A promising research approach to mitigate the complexity D.2 Software Engineering: Design tools and techniques. H5 of the development of UIs is the Model Based User Information interfaces and presentation: Group and Interface (MBUI) design methodology. In this article we organization interfaces. introduce MMW-4S, a RUIGE that is fully compliant with the Serenoa reference framework [2] and aims at facilitating General Terms the creation of context-aware UIs for the mobile Web. In Design; Languages. order to show the applicability of our solution and how it INTRODUCTION handles context-awareness, we present four adaptation use During the last years, the usage of Internet has experienced cases where the corresponding UIs are able to react to a dizzying growth and, accordingly, Web technologies have context changes. evolved at a rapid pace. Nowadays, users commonly access RELATED WORK the Web from a diverse set of mobile devices, such as One of the most well-known frameworks that defines the smartphones, tablets, book readers or even personal gaming fundamentals model-based UI design is CAMELEON [3]. consoles among others. Moreover, they are increasingly This framework structures the development life cycle into demanding rich applications able to tailor the UI in four abstraction levels, from task specification to the finally accordance to their specific context: location, time, user UIs. The Task and Concepts level considers the logical preferences, environment conditions, device capabilities, activities to be performed in order to reach users’ goals and network status and so on. the domain objects manipulated by these tasks. The From the developer perspective, programming mobile Web Abstract User Interface (AUI) level is an expression of the UI in terms of interaction spaces or presentation units, regardless of the interaction modalities (graphical, vocal, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are haptic, etc.) and the interactors available. The Concrete not made or distributed for profit or commercial advantage and that copies User Interface level (CUI) expresses the UI in terms of bear this notice and the full citation on the first page. To copy otherwise, “concrete interactors” that depend on the type of platform or republish, to post on servers or to redistribute to lists, requires prior and media available. Lastly, the Final User Interface level specific permission and/or a fee. (FUI) consists of source code to be interpreted or compiled RUIGE is composed of various sub-modules, each of them by actual target platforms. in charge of generating different outputs, such as mobile interfaces, vocal systems, avatar engines, etc. New sub- Various models have been proposed to represent different modules might be added to the architecture in order to aspects at different levels of abstraction. For instance, support additional interaction modes and target platforms ConcurTaskTrees [4] and ANSI/CEA-2018 [5] are intended for the same application definition. to define task models. MARIA [6] and USIXML [7] aim at representing AUI models and IDEAL2 [8] proposes a CUI Each RUIGE sub-module follows several stages in order to model tailored to mobile Web UIs. In what regards to generate the final application based on the abstract model-based approaches for multi-device web definition. Each stage is associated to a specific module: development, W3C has recently created the MBUI Working Group [9] in order to create standards as a basis for  Transformer: component in charge of the process of interoperability across authoring tools. converting the abstract language (ASFE-DL) into a generic language for the representation of the CUI of Some development platforms aimed at facilitating mobile each sub-module. This language will be the input to the Web development offer declarative models to describe UIs next stage. in a device independent manner. For instance, AWS Open  Generator: this module analyses the output of the Source (previously Volantis Mobility Server) [10] and IBM transformer and generates the executable code, which Portal Accelerator [11] uses XDIME2 [12], a standard- will be used at runtime. based XML vocabulary based on XHTML and XForms, and  Runtime: this component is intended to provide support Netbiscuits [13] uses BiscuitML [14] for the same purpose. for the execution of applications. Note that some sub- However, any of these platforms follow a pure Model- modules may require both deployment and execution Based approach. stages. This component is in charge of deploying the BACKGROUND application, if it has not been done before by the MyMobileWeb (MMW) [15] is an open-source standards- generator. For instance, uploading web applications in a based software platform released under LPGL version 3 web server, deploying an application in a servlet license. It has been designed to facilitate the development container or even installing (or running) a file in the of applications and portals for the mobile Web. It intends to user device. After the deploy stage, the application is provide an advanced environment for the development of executed by an execution platform; for instance, a web rich mobile Web applications which maximize user browser. experience regardless of the device used to access the Web. MMW-4S It has been conceived to solve inherent problems of mobile MMW-4S is an open-source module1 fully compliant with Web development in a holistic way and has been declared RUIGE and with the Serenoa reference framework. It is as a reference development platform by mTLD through their aimed at generating mobile Web applications for multiple dotMobi initiative. platforms and devices. Moreover, MMW-4S is able to Serenoa is a model-based framework aimed at developing a interoperate with the Adaptation Engine Serenoa reference platform for enabling the creation of context- component in order to adapt the generated UIs to the sensitive service front-ends (SFEs). From the point of view context, thus trying to maximize the user experience. Figure of Serenoa, a context-sensitive SFE provides a UI that 1 shows a simplified version of the MMW-4S architecture exhibits some capability to be aware of the context and to and its integration in the Serenoa platform. These sub- react to changes of this context in a continuous way. As a modules are highlighted in blue color. Basically, MMW-4S result, such a UI will be adapted to a person’s devices, receives the abstract description of the application from the tasks, preferences, and abilities, thus improving people’s Serenoa Service Repository, which subsequently gets such satisfaction and performance compared to traditional SFEs description from the Authoring Tools. The following based on manually designed UIs. sections provide details about the three main sub-modules of MMW-4S: transformer, generator and runtime. This paper presents the evolution performed by the authors over the MMW platform in order to adapt it to a pure Note that MMW-4S runtime needs to query the Serenoa model-based approach following the guidelines established Adaptation Engine in order to know the most appropriate by the Serenoa reference framework (MyMobileWeb For adaptation actions that the Runtime must apply in each Serenoa: MMW-4S). context. To accomplish that goal, the Runtime needs to produce/reference ad-hoc software modules (Context RUNTIME USER INTERFACE GENERATION ENGINE Delegates) that are the responsible for the extraction of RUIGE is one of the modules proposed by the Serenoa reference framework. It is a modular engine responsible for 1 the creation of final applications adapted to the context All MMW-4S code has been released as open-source starting from the AUI level by means of ASFE-DL [16]. under the LGPLv3 license and it is available at the Morfeo Forge: https://svn.forge.morfeo-project.org/serenoa/trunk/ contextual information coming from device sensors at based on jQuery offer the possibility to perform client-side runtime. These Context Delegates propagate the adaptations at runtime. information to the Context Manager, which is in charge of ADAPTATION SCENARIOS storing all the context properties and values for each device Once RUIGE behavior has been explained, it is necessary instance. The Adaptation Engine, in order to decide the to introduce some adaptation rules to be considered during most appropriate adaptation that should be applied in each the application lifecycle. In order to exemplify how MMW- specific context, needs to process adaptation rules 4S treats adaptation rules, we will suppose the following set expressed in AAL-DL [16] format and should communicate of rules, as coming from the Adaptation Engine. For the to the aforementioned Context Manager. sake of legibility they are expressed in natural language in this article, although they have been formally defined in AAL-DL. Adaptation to device features R1: If the device is a tablet, then master-detail presentations will be rendered in one single view. R1 must be considered during the transformation process when converting from ASFE-DL (AUI) to IDEAL2 + SCXML (CUI) by means of an XSLT. This XSLT generates two different versions of the IDEAL2 description. In the case of tablets, a master-detail view is expressed in just one IDEAL2 presentation. Otherwise, in case of mobile devices, the same view is expressed in two different IDEAL2 Figure 1. Architectural approach presentations, plus the additional flow logic expressed in Transformer SCXML. In order to carry out the adaptation, MMW-4S MMW-4S transformer carries out the transformation from sends the User-Agent to the Context-Manager to identify ASFE-DL to the CUI models used by MMW: IDEAL2 for which kind of device is accessing to the web. the UI view definition and SCXML [17] for the application Adaptation to the user profile flow description. It has been implemented as an XSL R2: If user is color-blind, then it will use an alternative transformation sheet (XSLT) due to the fact that both the color palette in the referenced images. input language (ASFE-DL) and the output languages Adaptation to the status of the device (IDEAL2 and SCXML) are XML-based. The current version R3: If the level of the battery is higher than a prefixed of MMW-4S Transformer supports the last version of ASFE- threshold, then a video must be shown but if its level is DL available at the time of writing, the last version of lower, it renders an image. IDEAL2 and the draft version of SCXML published on 16 May 2008. Both rules, R2 and R3 must be considered at execution time Generator by the runtime module in order to dynamically adjust the The UI Generator of MMW-4S is the part of the module in multimedia content (the color palette or the video/image in charge of generating JavaServer Pages (JSPs) from IDEAL2 each case) while the user is interacting with the application. documents. Each UI defined in IDEAL2 implies the In this case, the adaptation rules received in AAL-DL format generation of a specific JSP for each markup language in have been transformed into a rules language expressed in the market (WML, XHTML Basic/MobileProfile, HTML4 DRL, so as they can be executed at runtime by using the and HTML5). Note that the generated JSPs contain server- new rule engine developed within MMW-4S. Such rule side code that will be executed at runtime. These JSPs are engine is based on Drools Rule Engine [18]. In R2, the structured to guide the execution of the HTLM5 Rendering Context Manager handles user profiles in order to recognize Engine at runtime. special user features as color-blindness. In order to get the level of the battery in R3, an Android application was built. Runtime It simulates the access to the battery because, so far, there This module is intended to provide the most appropriate isn’t any API implementation in any browser to do it. FUI at runtime. Whenever a client device requests a specific Web page to the server, the corresponding JSP will Adaptation to the position of the user be executed. Note that JSPs are translated into Java Servlets R4: If the user is far away from the device, then bigger font- at runtime. Consequently, each web page access will result size will be shown but if the user is closer, the font-size will in the invocation of server-side code containing the logic be smaller. that generates the adapted Web content dynamically. The R4 must be considered by the generator during the HTML5 rendering engine is based on the jQueryMobile generation process when converting from IDEAL2 + library. Moreover, a set of pre-generated Javascript libraries SCXML (CUI) to HTML5 + CSS3 + Javascript (FUI). In this case, the adaptation rule is considered in the generation Specifying Task Models. Proceedings of Interact 97, 14- process to determine the Javascript code that should be 18 July 1997. delivered. However, the adaptation will be carried out in the 5. Rich, C. Building Task-Based User Interfaces With runtime process when the Javascript code is executed. In ANSI/CEA-2018. IEEE Computer, Vol. 42, No. 9, order to change the font-size, the frontal-web camera is August 2009. used by means of webRTC [19]. This API allows calculating the distance between the screen and the user and 6. Paternò, F., Santoro C., Spano, L.D., MARIA: A this value is periodically sent to the server to update the Universal Language for Service-Oriented Applications font-size. in Ubiquitous Environment", ACM Transactions on Computer-Human Interaction, Vol.16, N.4, November CONCLUSIONS AND FUTURE WORK 2009, pp.19:1-19:30, ACM Press. This article summarizes the ongoing research work aimed at creating an open-source RUIGE that uses Model-Based 7. Limbourg, Q., Vanderdonckt, Q., Michotte, B., techniques to facilitate the development of mobile Web UIS Bouillon, L., López-Jaquero, V.: USIXML: A Language tailored to the context. Moreover, we present four scenarios Supporting Multi-path Development of User Interfaces. to highlight the types of adaptations considered and we EHCI/DS-VIS 2004: 200-220 provide technical details about the implementation of such 8. Cantera, J.M, Díaz, J.L, Rodríguez, C.: IDEAL2 adaptations. Corelanguage. MyMobileWeb Working Draft, 31 December 2010, http://files.morfeo- Further research needs to be done in order to: (a) align the project.org/mymobileweb/public/specs/ideal2/ideal2- proposed method with incoming standards, such as those 20101231(2010). models being defined in the W3C MBUI Working Group; (b) exploit further context variations to perform adaptations, 9. W3C Model-Based User Interfaces (MBUI) Working taking into account the evolution of the recommendations Group: http://www.w3.org/2011/mbui. of the W3C Device APIs Working Group; (c) validate our 10. AWS Open Source web site: proposal in comparison to other research works http://www.antennasoftware.com/community/developer ACKNOWLEDGMENTS s/open-source The research is being performed thanks to the support of the 11. IBM Portal Accelerator web site: http://www- following research projects: 03.ibm.com/software/products/us/en/ibmmobiportacce/  MyMobileWeb: an open source project intended to 12. XDIME2 language reference: facilitate the creation of mobile Web applications http://www.antennasoftware.com/pdf/open_source_start funded by Ministerio de Industra, Turismo y Comercio _here.pdf (MITyC) within the framework of The National Plan 13. Netbiscuits Platform web site: for Scientific Research, Development and http://www.netbiscuits.com/ Technological Innovation 2008-2011 and by the 14. BiscuitML reference guide: European Regional Development Fund (ERDF). http://kb.netbiscuits.com/tactile/bml/index.html  Serenoa: a european project aimed at developing a 15. MyMobileWeb Project web site: novel, open platform for enabling the creation of http://mymobileweb.morfeo-project.org. context-sensitive SFEs. This project has received funding from the European Commission's Seventh 16. Ghiani G., Paternò F., Santoro C., Spano L.D.: A Set of Framework Programme under grant agreement nº Languages for Context-Aware Adaptation. CASFE 258030 (FP7-ICT-2009-5). 2012, CEUR WS Proceedings, Vol 970. 17. Barnett, J. et al.: State Chart XML (SCXML): State REFERENCES 1. W3C Device APIs (DAP) Working Group: Machine Notation for Control Abstraction. W3C Working Draft 16 February 2012, http://www.w3.org/2009/dap/. http://www.w3.org/TR/2012/WD-scxml-20120216/ 2. Serenoa Project web site: http://www.serenoa-fp7.eu. (2012). 3. Limbourg, Q., Vanderdonckt, J., Bouillon, L., Calvary, 18. Drools web site: http://www.jboss.org/drools/ G., Coutaz, J., Thevenin, D.: A unifying reference 19. WebRTC web site: http://www.webrtc.org/ framework for multi-target user interfaces. Interacting with Computers. 15, 289-308 (2003). 4. Paterno, F., Mancini, C., Meniconi, S.: ConcurTaskTrees: A Diagrammatic Notation for