End-User Customization of Multi-Device Ubiquitous User Interfaces Fabio Paternò, Giuseppe Zichitella HIIS Laboratory – CNR-ISTI Via Moruzzi 1, 56124 Pisa, Italy {fabio.paterno, giuseppe.zichitella}@isti.cnr.it ABSTRACT End-User Development [3] (EUD) can be defined as a set In this paper we discuss an approach to supporting end- of methods, techniques, and tools that allow users of users in customizing multi-device ubiquitous user software systems, who are acting as non-professional interfaces. In particular, we show a tool allowing end-users software developers, at some point to create, modify or to customize desktop-to-mobile adaptation by exploiting extend a software artefact. End-users have already model-based descriptions in the MARIA language. Some difficulties with single device applications, thus it is easy to results are presented along with indications for future work. understand how such difficulties increase when considering Author Keywords applications for multi-device environments. This is one End-user Development, Ubiquitous Applications, Multi- further reason for providing better support for EUD in Device Environments, Adaptation. ubiquitous applications. ACM Classification Keywords The vision of ubiquitous computing [9] is that the users H.5.2 User Interfaces. operate in intelligent environments, which are aware of users’ needs and able to assist, even proactively, the users INTRODUCTION in performing their activities and reaching their goals. To One of the main issues in current technological settings is this end, one important aspect is the possibility for a user how to design and develop interactive applications that can surrounded by multiple devices to freely move about and be accessed through a wide variety of devices (ranging continue the interaction with the available applications from small watches to very large screens, including various through a variety of interactive devices. Indeed, in such types of smartphones, PDAs and Digital TVs). This is environments one big potential source of frustration is that particularly important in Web application, which are the people have to start their session over again from the most common applications. beginning at each interaction device change. Continuous One important research area in this context is the model- task performance implies that interactive applications be based approach, in which declarative descriptions of the able to follow users and adapt to the changing context of user interface are used in order to avoid dealing with a use while preserving their state. Thus, migratory user plethora of low-level implementation details associated interfaces require integrated solutions able to address state with the wide number of available devices and persistence and user interface adaptation when the user implementation languages. Despite such potential benefits, changes the device. its adoption has mainly been limited to professional Model-based languages are utilized at design time to help designers, but new solutions are recently emerging that are the user interface designer cope with the increasing able to extend such approaches in order to achieve natural complexity of today’s applications and contexts. The development by enabling end-users to develop or modify underlying user interface models are mostly used to interactive applications still using conceptual models, but generate a final user interface code, which is then executed with continuous support that facilitates their development, at run time. Nevertheless, approaches utilizing the models analysis, and use [1]. at run time are receiving increasing attention. We agree with Sottet et al. [8], who call for keeping the models alive at run time to make the design rationale available and show Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are a solution for this purpose. not made or distributed Pre-proceedings for profit or of the 5th International commercial Workshop advantage on Model Driven and that copies Development of In the following we present some research work that bear thisUser Advanced notice and the Interfaces full citation (MDDAUI 2010):onBridging the firstbetween page. User To copy otherwise, Experience and UIor Engineering, republish,organized to post onat the 28th ACM servers or toConference redistributeon Human to lists,Factors in prior requires exploits model-based approaches for multi-device Computing Systems (CHI specific permission 2010),aAtlanta, and/or fee. Georgia, USA, April 10, 2010. ubiquitous applications. We show how we have enriched a software model-based platform for migratory user Copyright © 2010 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. Re-publication of material from this volume interfaces with a new tool for desktop-to-mobile adaptation, requires permission by the copyright owners. This volume is published by its editors. called parametric bidimensional semantic redesign. One of 1 41 its features is that it allows the end-users to customize the selected, data entered, …) and identifies the last element adaptation process. We present some initial results and then accessed in the source device. Thus, when a logical version discuss how we plan to extend them. of the interface for the target device is generated, it also contains the state detected in the source device version so that the user inputs (selections performed, data entered, …) MIGRATORY USER INTERFACES are not lost. In the last phase, the user interface Migration is the result of two main features: state implementation for the target device is generated and persistence across multiple devices and adaptation to the activated remotely at the point corresponding to the last device interaction resources. They have to be supported basic task performed in the initial device. while users interact with the applications made available by the intelligent environment. For this purpose, we have designed and developed a migration architecture [5], which supports a number of reverse and forward transformations that are able to transform existing desktop Web applications for various interaction platforms and support task continuity. The basic assumption is that there exists a huge amount of easily accessible content for desktop Web Figure 1. The main phases of the adaptation process. applications, which can be processed and transformed to support migratory interfaces, even across non-Web In the process of creating an interface version suitable for a implementation languages. The advantage of this solution platform different from the desktop, we use a semantic with respect to others (e.g. [4]) is that it does not require redesign module. This part of the migration environment that the applications be implemented using a particular automatically transforms the logical description of the toolkit in order to make them able to migrate. desktop version into the logical description for the new platform. Therefore, the goal of this transformation is to In this environment the client devices subscribe to the provide a description of the user interface suitable for the migration service by running a migration client that new platform. This means that intelligent rules are used for provides the environment with information regarding the adapting the description of the user interface to the new device characteristics. The devices access Web applications platform taking into account its capabilities (e.g.: using through the migration server, which includes proxy interface elements that are more suitable for the new functionalities. Migration can be triggered either by the user platform) but ensuring at the same time that the support for or it can be automatically triggered by the smart the original set of tasks is maintained. This solution allows environment when some specific event (such as very low the environment to exploit the semantic information battery or connectivity) is detected, or in a mixed solution contained in the logical description. In this case the in which the environment suggests possible migrations and semantic information is related to the basic tasks that the the user decides whether or not to accept them. user interface elements are expected to support. When the user accesses the application through an This software architecture for migratory user interfaces interaction platform other than the desktop, the server currently uses MARIA [7], a recent model-based language, transforms its user interface by building the corresponding which allows designers to specify abstract and concrete user logical description and using it as a starting point for interface languages according to the CAMELEON creating the implementation adapted to the accessing Reference framework [2]. This language represents a step device. Figure 1 shows how adaptation is obtained. There forward in this area because it provides abstractions also for are three main phases: reverse engineering, semantics-based describing modern Web 2.0 dynamic user interfaces and adaptation, and generation. In the first phase, the tool Web service accesses. In its first version it provides an automatically builds the logical description of the accessed abstract language independent of the interaction modalities page. It has rules able to handle HTML and CSS tags and and concrete languages for graphical desktop and mobile associate them with the corresponding logical elements. For platforms. In general, concrete languages are dependent on example, if DIV, or FIELDSET or IFRAME tags are used the typical interaction resources of the target platform but then it recognises that there is a group of logically independent of the implementation languages. connected elements in the page. We call the adaptation module semantic redesign since its purpose is to change the In MARIA an abstract user interface is composed of one or design still considering the interaction semantics of the multiple presentations, a data model, and a set of external implementation elements that are specified in the functions. Each presentation contains a number of user corresponding logical description. In addition to interface interface elements (interactors) and interactor compositions adaptation, the environment supports task continuity. To (indicating how to group or relate a set of interactors), a this aim, when a request for migration to another device is dialogue model describing the dynamic behaviour of such triggered, the environment also takes the state of the user elements, and connections indicating when a change of interface, which depends on the user input (elements presentation should occur. The interactors are classified in abstract terms: edit, selection, only_output, control, 42 interactive description, etc.. Each interactor can be The elements that determine the cost of the interactors are: associated with a number of event handlers, which can the font attributes (size, style, type), the vertical and change properties of other interactors or activate external horizontal space required by a text, image dimensions, functions. interline value, interactor type, and so on END-USER ADAPTATION CUSTOMIZATION Figure 2 shows the user interface that allows end users to In the research on migratory user interfaces, one issue that configure the adaptation process. The various parameters we are considering is how to provide users with more are grouped according to the related user interface aspect control on the migration process in order to improve its considered. For the fonts, it is possible to specify the usability. In this context more control can mean various minimum and maximum font in the target device, and the things. One important aspect is control on the rules that associated measure unit. For the radio buttons it is possible drive adaptation to the various platforms (the most common to indicate whether they should be transformed into an case is desktop-to-mobile adaptation). For example, the interactor that supports the same semantics but with using adaptation engine is able to split the desktop pages when less space screen. In this case, it is possible to specify the they require considerable amount of interaction resources threshold, in terms of number of choice options, which but some users may like to have more control on the should trigger the transformation and the type of interactor splitting algorithm. to use for its replacement. Similar parameters are available for the list boxes. Other parameters concern the maximum In particular, we have designed a new tool for adaptation: number of characters for a text, maximum and minimum Parametric Bidimensional Semantic Redesign. It supports dimensions for images. These parameters determine the adaptation from desktop-to-mobile devices and overcomes cost of rendering a presentation. This cost is compared with limitations of previous approaches in the area [6] because it the overall sustainable cost in the target device, which is allows users to configure the adaptation process and provides more control on costs calculation and the given by the screen resolution multiplied by horizontal and vertical tolerance. The higher the tolerance coefficient adaptation results. For example, while previous solutions values are, the more scrollable the generated user interface calculated the screen space requested by the user interface elements mainly in terms of its vertical use, the new will be. This means that end users have the possibility to algorithm calculates both the horizontal and the vertical specify to what extent the adapted content will be scrollable in the target device. The table tolerance provides an consumption of screen space. additional factor to consider when calculating the The adaptation tool takes as input the concrete description sustainable cost. In practise, this means that when there are of a desktop user interface in the MARIA language and tables, more scrolling will be acceptable before deciding to goes through a number of steps. For each step a number of split the presentation. specific rules are applied. First, it performs some basic The customization interface also allows the user to indicate transformations: if the user provides preferences regarding two additional parameters: what type of scrolling the minimum and maximum fonts for the target device then it transforms all the textual content in order to fit in the (horizontal or vertical) to avoid has the priority, and the splitting algorithm version to apply. Indeed, the tool given range. Next, it calculates the cost of all the interactors supports two ways to determine how splitting should be and composition operators in the provided specification. If performed. In both cases it analyses the cost of the the resulting total cost is sustainable for the target device then the corresponding logical description is generated composition operators (grouping or relation), which includes those of the composed interactors, and the cost of otherwise it starts the process to reduce the cost in order to the tables (both data and layout tables). Then, the decision make it sustainable. First, basic elements are adapted for the of the set of elements to allocate to the newly generated target device: the images are reduced in space while mobile presentation is given in one case by the most preserving their aspect ratio, some interactors are replaced expensive element. In the other case the algorithm first with others that are semantically equivalent but needs less calculates what elements are able to make the current screen space, long texts are reduced in such a way that the part exceeding a limit is shown only on request, image and presentation sustainable by the target device if removed, and then selects among them the one that has the lowest text in tables are reduced in size. After these basic cost. The rationale for this second option is that it allows transformations the overall cost is calculated again and if it is not yet sustainable by the target device then the part users to obtain a sustainable presentation by removing the related to page splitting is activated. The purpose of this least amount of information possible, thus preserving as much as possible the original design. phase is to split the original desktop presentation into two or more presentations, which are sustainable for the target In terms of results of the adaptation process we have mobile device. For this purpose the algorithm considers the conducted a study comparing our tool with two publicly interactor compositions, and associates some of them to available tools for desktop-to-mobile adaptation: Mowser newly generated mobile presentations, removing them from (http://mowser.com) and Skweezer the current presentation in order to decrease its overall cost. (http://www.skweezer.com). The results were encouraging because our tool has shown to be more flexible since it 3 43 allows end users to customize the adaptation parameters In this paper we have presented first results that allow end- and is able to adapt a higher number of types of interface users to customize desktop-to-mobile adaptation in order to elements than the other two tools (e.g. tables and long texts change the results that can be obtained by automatic user do not receive specific adaptation transformations with the interface generation. other two tools). We plan to further extend this work in various directions. The customization user interface can be improved in order to make the effects of the various customization parameters more understandable. In addition, in this work we have considered only desktop-to-mobile adaptation but other types of transformations can benefit from the approach proposed, e.g. graphical-to-vocal adaptation. ACKNOWLEDGMENTS This work has been supported by the EU project OPEN (http://www.ict-open.eu/) REFERENCES 1. Berti, S., Paternò, F., Santoro C., “Natural Development of Ubiquitous Interfaces”, Communications of the ACM, September 2004, pp.63-64, ACM Press. 2. Calvary, G., Coutaz, J., Bouillon, L., Florins, M., Limbourg, Q., Marucci, L., Paternò, F., Santoro, C., Souchon, N., Thevenin, D., and Vanderdonckt, J. 2002. The CAMELEON reference framework. CAMELEON Project. Deliverable 1.1 3. Lieberman, H., Paternò, F., Wulf W. (eds), End-User Development, Springer Verlag, ISBN-10 1-4020-4220- 5, 2006. 4. Melchior, J., Grolaux, D.,Vanderdonckt, J.,Van Roy, P., A Toolkit for Peer-to-Peer Distributed User Interfaces: Concepts, Implementation, and Applications, pp. 69.78, EICS’09, July 15–17, 2009, Pittsburgh, Pennsylvania, USA. 5. Paternò, F., Santoro, C., Scorcia, A., Ambient Intelligence for Supporting Task Continuity across Multiple Devices and Implementation Languages, the Computer Journal, the British Computer Society, 2009. 6. Paternò, F., Santoro, C., Scorcia A Automatically Adapting Web Sites for Mobile Access through Logical Descriptions and Dynamic Analysis of Interaction Resources. AVI 2008, Naples, May 2008, ACM Press, pp. 260-267. 7. Paternò F., Santoro C., Spano L.D., "MARIA: A Universal Language for Service-Oriented Applications Figure 2. The customization user interface. in Ubiquitous Environment", ACM Transactions on Computer-Human Interaction, Vol.16, N.4, November 2009, pp.19:1-19:30. CONCLUSIONS Ubiquitous environments call for adaptive systems in order 8. Sottet J., Ganneau V., Calvary G., Coutaz J., Demeure to adapt to the varying interaction resources. Model-based A., Favre J., Demumieux R.: Model-Driven Adaptation approaches can provide useful support in this context. for Plastic User Interfaces. INTERACT (1) 2007: 397- However, there is a need for providing users with more 410. control on ubiquitous interfaces, according to the end-user 9. Weiser M., "The Computer for the 21st Century" - development paradigm. Scientific American Special Issue on Communications, Computers, and Networks, September, 1991. 44