Multidimensional Context-Aware Adaptation of Service Front-Ends Javier Caminero1, Mari Carmen Rodríguez-Gancedo1, Jean Vanderdonckt2, Fabio Paternò3, Joerg Rett4, Dave Raggett5, Jean-Loup Comeliau6, Ignacio Marín7 1 Telefónica R&D, Spain; 2 Université catolique de Louvain, Belgium; 3 Consiglio Nazionale delle Ricerche, Italy; 4 SAP AG, Germany; 5 GEIE ERCIM, France; 6 W4 S.A., France; 7 Fundación CTIC, Spain Email: fjcg@tid.es, mcrg@tid.es, jean.vanderdonckt@uclouvain.be, fabio.paterno@isti.cnr.it, joerg.rett@sap.com, dsr@w3.org, jean-loup.comeliau@w4global.com, ignacio.marin@fundacionctic.org ABSTRACT voice, gesture, etc.), geographical location, social context, The EU-funded Serenoa Project (www.serenoa-fp7.eu) is accessibility and user preferences. creating a novel open platform for developing context- Finally, the underlying technologies involved in the aware application user interfaces. Such user interfaces are development of user interfaces are multiple, keep changing aware of the changes in the context and can react to them in and emerging at a rapid pace. Development therefore often a continuous way. This includes adaptation to the user's requires technical experts, who are hard to find and retain. devices, tasks, preferences and abilities, thereby improving As a matter of fact, most of the effort in a traditional project the user's satisfaction and performance compared to is spent on infrastructure and technical tasks, and not on traditional approaches to user interface design. An business issues. This often contributes to frustration of additional benefit is a reduction in the cost and business stakeholders and end-users. There is also a steep development time for creation and maintenance of user learning curve for each new platform that has to be interfaces. mastered: Android with Java, iOS with ObjectiveC, Windows with C#, Linux with C/C++, the Web with HTML and JavaScript, and so forth. Translating user DESCRIPTION OF THE PROBLEM interface designs into each of these platforms is time- Even though software development environments, consuming and expensive. standards and engineering methodologies have significantly evolved and matured over the last decade, designing and implementing context-aware application user interfaces is MODEL-BASED DESIGN still complex, and therefore expensive. A variety of approaches have evolved in response to the Businesses are seeing increasing diversity in the kinds of above challenges. These include the Eclipse Integrated devices being used (i.e. workstations, desktops, laptops, Development Environment, the Java 2 Platform Enterprise smart phones, tablets, embedded devices, etc.), but this is Edition (J2EE), W3C's HTML5, JavaScript libraries (e.g. putting up the cost of developing and maintaining user jQuery), and OMG's Model-Driven Architecture (MDA). interface designs. Aligning business and Information These have been supported by work on software Technology (IT) is a constant challenge, not only because development methodologies, such as Model-Driven of the constantly evolving standards and underlying Engineering (MDE) or Agile project methodologies, such technologies, but also because of the accelerating pace at as Scrum or xP, or with UI design approaches such as User- which functional requirements change throughout the Centred Design (UCD). When it comes to designing user lifetime of applications. Business applications need to interfaces, Model-Based approaches bring many benefits by adjust rapidly to fulfill changing requirements, and this filling the gaps where traditional approaches are inadequate trend is accelerating as users require more and more for agile development, and struggle to cope with the reality mobility. that requirements aren't fully understood at the outset of a project, but rather emerge as designs are tried out and found Moreover, implementing Human-Computer Interaction found useful, relevant and necessary. (HCI) systems requires forecasting user-driven interaction combinations that are not always easy to anticipate. Adding Model-based user interface design is about separating out context management to the HCI domain increases design concerns at different levels of abstraction using both complexity further because it requires taking into account declarative and procedural knowledge. The declarative part multimodality (i.e. keyboard & mouse, touch screens, is expressed as task models, abstract user interface (AUI) models and concrete user interface (CUI) models, whilst the procedural part is expressed with rule languages for Permission to make digital or hard copies of all or part of this work for adaptation at both design-time and runtime. personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Other Model-Based approach benefits include interface implementation. The adaptation engine simplification through automation and code generation, a determines the optimal adaptation for the current context of clean separation of concerns, e.g., IT and business issues, use, based upon the context models and adaptation rules. and the fact that the model becomes a common language To achieve this goal, the context manager provides for all project stakeholders and helps IT align with information related to all the possible contextual business. Designers vary in how they approach the dimensions (i.e. the user preferences, the environment, challenges of user interface design. We would not take social relationships, etc.) The run- time engine generates advantage of the true potential of Model-Based user the final interactive application according to the context. interface design if we only supported a top-down Currently, this module is composed of a set of sub-modules methodology. A significant number of designers find it which cover several modalities (i.e. mobile web easier to start at the concrete level and refine the abstract applications, vocal interfaces, avatar- based interaction or level later. Subsequent revisions to the abstract level should desktop business applications). In fact, ‘Applications be possible without throwing away the design work done at prototypes’ showing the functionality of these sub-modules the concrete level. However, traditional Model-based and the whole Serenoa framework are under development. design also suffers from a lack of attention to adaptation to the context of use, and this is exacerbated by a lack of standards for expressing adaptation rules. AUTHORING TOOLS Authoring tools help designers, engineers and web authors to easily create context-sensitive user interfaces for DESCRIPTION OF THE SOLUTION different platforms and different interaction modalities, e.g. visual, aural and tactile. The authoring tools will be usable by non-expert programmers, and fulfill some additional key success factors, including the usability of the graphical interface, the availability on multiple platforms and the support for concurrent work by multiple users (business experts, user interface designers and programmers working as a team) appear as very important requirements. The authoring tools will support editing of model-based descriptions at both abstract and concrete levels, together with context-dependent transformation rules. Two types of authoring tools are being developed as proof of concept for the Serenoa architecture: Figure 1. Serenoa’s architecture Serenoa’s proposed architecture for supporting user • Eclipse-based Plug-in: this type is a 'plug-in' for interface adaptation is depicted in Figure 1. Eclipse, one of the most widely used Integrated Development Environments (IDEs) in the research and Firstly, at design-time, ‘Authoring tools’ are used to industrial communities for software development. Since support the design of model-based user interfaces. The most of the open source and research based authoring authoring tools support Serenoa’s abstract user interface environments/tools, i.e. MARIA and LEONARDI and other description language, as well as the Serenoa language for related libraries are developed in Java, we have also chosen expressing adaptation rules (see ‘Languages’). For those the Java programming language for Serenoa's authoring developers who find it easier to start the definition of the environment and tools. interface at the concrete level and then refine the abstract • HTML5-based Browser Application: the second level as they go, there is a design-time adaptation engine type being developed is an HTML5-based web browser which makes it possible to work at both levels. application that operates on the models that are held on the Secondly, the ‘Theoretical framework’ consists of the web server side. This approach allows live concurrent reference models and the ontology. These reference models editing by multiple users, so that they could see and discuss are aimed at guiding developers and designers during the the changes which a remote user is making in real time. A complete software life-cycle, listing alternative possibilities server-based design-time adaptation engine allows for implementing context-aware adaptation, and permitting designers to work at different levels of abstraction, the analysis and comparison of adaptive and adaptable synchronising changes in a cooperative workflow. applications. The ontology, based on the reference models, Thus, Model-Based languages, i.e. MARIA, UsiXML, and is intended to gather all the knowledge involved in IDEAL2 have been used as basis for not only developing advanced adaptation logic for user interfaces. It is used to the abstract UI and adaptation logic languages, but also the inform developers as well as to support adaptation generation of Abstract/Concrete UIs. processes in the runtime. Finally, the runtime phase will transform the description and associated rules of user interfaces into a final user LANGUAGES performance, reliability or rules that promote or demote The languages developed in Serenoa will cover the sets of second-order rules). specification of adaptive SFEs at different abstraction layers, and of the context-dependent transformation rules to be applied on the user interfaces. With the Serenoa THEORETICAL FRAMEWORK solution, the exploitation of both these languages will be The Theoretical ground of the Serenoa Project is structured supported not only at design time but also at runtime. At in three main components: a Context-aware Reference design time, the authoring tools will help the designers, Framework (CARF), a Context-aware Design Space engineers and web authors to easily create and edit context- (CADS) and a Context-aware Reference Ontology sensitive SFEs for different platforms (at both abstract and (CARFO). The CARF provides to stakeholders the core concrete levels) and relevant context-dependent concepts for defining and implementing adaptive and transformations rules. At runtime, the logical descriptions adaptable systems. The CADS provides means to analyze, of the SFEs and of the adaptation rules will be transformed evaluate and compare multiple applications regarding their in a final, adapted user interface implementation. coverage level of adaptation, especially concerning certain specific dimensions (such as: modality types). The CARFO • The Advanced Service Front-End Description not only formalizes the core concepts defined by CARF and Language (ASFE-DL) is aimed at enabling the their relationships, but also enables the request and retrieval development and authoring of context-aware SFEs. The of relevant information for defining and executing the user interfaces modeled through this language will be adaptation process. Both industrial and scientific domains adapted to the context by exploiting the rules defined can benefit of these theoretical models, once they provide through the Advanced Adaptation Logic Description support for the whole development life-cycle of adaptive Language (AAL-DL). By leveraging on past expertise on and adaptable applications, i.e. design, specification, user interface languages that Serenoa members have implementation and evaluation. already authored or co-authored, and on previous experiences they gathered by working in relevant industrial case studies to support requirements of most modern APPLICATION PROTOTYPES service-based user interfaces, the Serenoa consortium plan The Warehouse-Management Scenario for the intelligent to build a more complete language that will allow ASFE- picking prototype is aimed at providing a seamless context DL to meet the Serenoa requirements and to go beyond the (environment and task) adaptation experience to users in state-of-the-art in this field. The ASFE-DL has been one of the partner's Living Lab facilities in the field of already specified at the Abstract user interface level: it Future Retail Concepts (FRC). This scenario motivates how describes the UI through a number of abstract interaction proactive applications can provide unobtrusive and units and associated connections in a modality-independent adequate help (e.g. missing parts, location of necessary manner. In the next months the Serenoa project plans to parts, etc.) when the user needs it. Thereby, the service time cover also the Concrete user interface level. The ASFE-DL can be reduced while increasing the quality of service. At is currently being submitted as an input to standardization the FRC, the supply chain continues to Retail Management work at W3C. with five demos, e.g. on price strategy and smart vending • The Advanced Adaptation Logic Description and concludes in Retail with eight demos, e.g. on Mobile Language (AAL-DL) is a high-level language intended to Payment and RFID Shelves. This opens up the possibility, express advanced adaptation logic in a declarative manner. while following the modular design of the Serenoa The basic idea is that the user interfaces modeled through components, to create further prototypes, moving from ASFE-DL will be adapted to the context by exploiting the Logistics to Sales scenarios. rules defined through the AAL-DL. The AAL-DL rules With the E-commerce Scenario, we aim at illustrating how have been expressed through an Event-Condition-Action different online end-users can take advantage of adaptive (ECA)-based format where: i) events are changes that can SFEs while connecting to both a front-end application and a occur in the context state or in the UI state; ii) conditions back-end application. Typical user roles involved in the are Boolean predicates referring to context state or UI state; scenario include online shoppers and employees, acting iii) actions are changes affecting the interactive application. either as supervisors or customer representatives in charge In the current specification of the AAL-DL we have of following-up with online orders. Based on their roles, considered the definition of first-order adaptation rules such users can access different features, but their UIs are (simple adaptation rules like e.g., adapt this service front- capable of adapting based on different factors such as end for this platform) and second-order adaptation rules language, colour- blindness or type of device (either a (those that govern the application of adaptation rules by e.g. home computers or a mobile device). selecting first-order rules: the action part of a rule can be in turn another rule). In the next version of this language we The E-Health Scenario is intended to be shown through the will consider third-order adaptation rules (strategies that improvement of two existing developments: privilege some adaptation approach for usability, • The SARA project, intended to provide a user interface for chronic disease patients self-monitoring in the form of a (Windows based) tablet PC. The project wants to domains. Besides this, Serenoa tries to establish a formal evolve to provide multi-device support (Android tablet link between the research community and the industry (for devices, smartphones, etc.) and an expressive virtual instance by means of standardization and dissemination assistant in order to engage patients in the usage of the actions). application. This project is now in a pre-market phase, after As a part of the individual exploitation strategies the successful field tests using real patients from the consortium members demonstrate the Serenoa Andalusian health system. ‘Applications prototypes’ and Development Tools to • The HealthDrive project, aiming to leverage on product groups, customers and partners. This aims at consumer devices such as computers, tablet PCs and encouraging these industrial stakeholders for an adoption of phones to provide its users access to their personal file on the Serenoa technologies to enhance their existing products the Andalusian health system. In order to do so, all medical or to create new ones. information is digitized and shared by the institutions, with a publicly accessible interface for each user in which she From the point of view of the academic community, several can interact with doctors and see their health records. The branches of knowledge are involved with research of inclusion of ECAs (Embodied Conversational Agents) context-aware adaptation. As the closest related ones, we technology for guiding the navigation through all this can highlight: Human-Computer Interaction, Software personal information is also being considered. Engineering and Architecture, Distributed Systems and Ubiquitous Computing. In the context of Serenoa project, the scientific field may benefit from: concrete requirements ADDED VALUE AND IMPACT for designing and implementing adaptive systems, In a general way, the Serenoa project may contribute to the authoring tools and languages that support the creation of following aspects: adaptive applications, theoretical frameworks that provide a • Faster time-to-market: given the support provided catalogue of information to support the research in the by the Serenoa technologies (i.e. authoring tools, adaptation field, evaluation criteria and possible architectural languages and models) the implementation of adaptive and approaches. adaptable applications will be easier and more efficient. Furthermore, releasing some Serenoa components as open • Reduction of development effort: instead of source facilitates the adoption of the results by other spending a lot of resources looking for knowledge about members of the community, even any other projects, context-aware adaptation, stakeholders can rely in a communities, organizations or anyone interested in this centralized information source and a dedicated focused area. This issue implies that any of them could take development platform, provided respectively by the the results and evolve them into something more complete, theoretical framework, and by the technological framework allowing to increase the impact of the project outside the (i.e. tools, languages, etc.). community of the project. Specifically, Serenoa is expected to produce three types of results: theoretical frameworks, • Promote re-use: by means of standards and a languages, and application prototypes. consistent terminology, the applications can be implemented in a more flexible manner, i.e. with the • Theoretical frameworks: the reference models integration of components and with the extension and (CARF and CADS) will be made available for the public updates of existing applications. by means of written documents (as deliverables and scientific papers). They will also include a detailed • Efficient solutions: Serenoa is filling a gap in description about their methodology, creation, and context-aware adaptation systems by allowing non-expert application available online by means of a web page. The developers to develop efficient solutions. This is achieved, Ontology will be developed based on this theoretical in particular, by adopting a model-based approach. framework. • Increased agility: Because context is often • Languages: the XML schemas describing the dynamic by nature, adaptation needs to be tuned little by languages developed in Serenoa will be made public by little (and sometimes continuously), based on the feedback putting them available on the project web site. In addition, provided by the different stakeholders, starting with the such results will be submitted to W3C for standardization. end-user. Therefore, the capability for systems to change its The W3C working group on Model-Based User Interfaces behaviour rapidly and to take into account new adaptation (MBUI) has started its work and various Serenoa partners rules, thus fitting new contexts, is essential. Serenoa's are involved in this group. adaptation engine adds substantial agility to the development process of context-aware systems, which can • Application prototypes: the applications, which provide great benefits. will illustrate the Serenoa framework, lay on various software modules: context manager, adaptation engine, • Joining efforts: By analyzing the technological runtime engine and the rest of components. In general, the landscape and the current status of the market, Serenoa software prototypes will be made publicly available. We identifies actual users' needs and orient its efforts in order envision various policies for making them public: open to progress simultaneously in both scientific and industrial source, public executable code, videos, etc. The way how The project provides various scenarios as proofs of concept, they will be made public depends on the organization that including a warehouse management application, an is developing them and the type of prototype. example of e-commerce application and an avatar-based e- In order to accomplish these results and to arouse interest in health project. The project outcomes are expected to the scientific and industrial community, an action planning accelerate time-to-market of context-aware systems, in has been envisaged. Firstly, on August-September 2012, a order to increase agility and to promote reuse. delivery of the final version of the Serenoa's architecture Dissemination and exploitation efforts are expected to and the second release of the modules will be available for impact the tools used in the industry in the ICT sector and the evaluation. From then to the end of the project to influence academic actors. (September 2013) the development efforts will be mainly ACKNOWLEDGMENTS focused on the final release of the Serenoa's components This work received funding from the European and tools, the exploitation of them in the Serenoa use cases Commission’s Seventh Framework Program under grant and the final evaluation. agreement number 258030 (FP7-ICT-2009-5). REFERENCES SERENOA BENEFICIARIES 1. Ali, S., Rett, J., Lewandowski,A. (2011). A SOA based Serenoa is a framework that simplifies and accelerates the Context-Aware Order Picking System for Warehouses development of context-sensitive applications. This implies using Laser Range Finder and Wearable Computer. the reduction of the time-effort of the development process Lucca, Italy: IEEE WoWMoM, 2011, pp. 1 - 8. and its corresponding maintenance costs. The value of the framework is based on a solid model-based architecture, a 2. Calvary, Gaëlle, et al. (2003). A Unifying Reference good methodology and advanced tools to automatically Framework for multi-target user interfaces. Interacting generate applications. Different user roles are susceptible to with Computers, vol. 15, pp. 289-308. Computer- take advantage of the aforementioned features: Aided Design of User Interfaces. • Developer: both final applications developers and 3. Cavazza, Marc, et al. (2010). How was your day? An those who are interested in adaptive technologies. Affective Companion ECA Prototype. Association for Computational Linguistics (ACL), 2010, pp. 277-280. • Researcher: scientific community interested in context-aware adaptation and related topics. 4. Limbourg, Q., et al. (2005). USIXML: A language • End-User: indirect beneficiary who gets faster supporting multi-path development of user interfaces. results because of the time-to-market reduction. Springer, 2005, Engineering Human Computer Interaction and Interactive Systems, pp. 134-135. • Executive Director: professional who may be interested in the creation of new business opportunities. 5. Losada, B. et al. (2011). Agile Development of Interactive Software by means of User Objectives. Barcelona, Spain: ICSEA 2011 - The Sixth CONCLUSIONS International Conference on Software Engineering The Serenoa project is intended to fill a gap in the market Advances, 2011, pp. 539-545. for developing context-aware Service Front Ends. Providing simple tools and methodologies to address user- 6. MyMobileWeb (2008-2011). "Open source software centric systems for a better and more enjoyable user platform for the development of top-quality mobile experience potentially benefits a multitude of stakeholders, web applications and portals, providing an advanced from researchers and developers to end-users and CIOs. content & application adaptation environment". http://mymobileweb.morfeo-project.org, 2008-2011. Starting from a theoretical model consisting of a reference framework and a design space, Serenoa addresses this 7. Paternò, F., Santoro, C. and Spano, L.D. (1999). challenge by adopting a Model-Based approach and by MARIA: A universal, declarative, multiple abstraction- offering a set of tools that can be used both at design time level language for service-oriented applications in and at runtime for implementing context-aware systems. ubiquitous environments. ACM 2009, ACM Trans. Computer-Human Interaction, vol. 16, pp. 19:1-19:30. At design time, Serenoa’s main components include authoring tools that are able to describe the user interfaces 8. W3C (2011). Model-Based UI Working Group and their adaptation rules using two languages to design Charter, http://www.w3.org/2011/01/mbui-wg-charter systems at different abstraction levels. At runtime, 9. The SERENOA Project: Multidimensional Context- Serenoa’s main components include the Context Manager Aware Adaptation of Service Front-Ends J. Caminero, to capture the current context related information, an M.C. Rodríguez, J. Vanderdonckt, F. Paternò, J. Rett, Adaptation Engine to dynamically compute the optimal adaptation strategy and a Runtime Engine to generate the D. Raggett, J. Comeliau, I. Marín; Proc. Of the 8th LREC International Conference, Istanbul, Turkey, May final user interface for each target platform. 2012