=Paper=
{{Paper
|id=Vol-462/paper-2
|storemode=property
|title=AnAmeter: The First Steps to Evaluating Adaptation
|pdfUrl=https://ceur-ws.org/Vol-462/paper2.pdf
|volume=Vol-462
}}
==AnAmeter: The First Steps to Evaluating Adaptation==
https://ceur-ws.org/Vol-462/paper2.pdf
AnAmeter: The First Steps to Evaluating Adaptation
F. Tarpin-Bernard1 , I. Marfisi-Schottman1, H. Habieb-Mammar2
1
LIESP
Université de Lyon, LIESP,
INSA-Lyon, F-69621, Villeurbanne, France
Tel: +33 (0) 4 72 43 70 99
Fax: +33 (0) 4 72 43 79 92
franck.tarpin-bernard@insa-lyon.fr
iza.marfisi@insa-lyon.fr
2
Human-Centered Software Engineering Group
Concordia University, 1455 Maisonneuve West
Montreal Quebec Canada H3G 1M8
mammar@encs.concordia.ca
Abstract. This paper presents the online AnAmeter framework that helps
characterize the different types of adaptations a system features by helping the
evaluator fill in a simple form. The provided information is then processed to
obtain a quantitative evaluation of three parameters called global, semi-global
and local adaptation degrees. By characterizing and quantifying adaptation,
AnAmeter provides the first steps towards the evaluation of the quality of a
system’s adaptation. AnAmeter is an open tool available as freeware on the web
and has been applied to a selection of well known systems. To build this
evaluation grid we also collected a number of systems that cover the full range
of adaptation types.
Keywords: adaptation degree, evaluating adaptation, adaptivity, adaptability,
characterization, quantification.
1 Introduction
People using computer systems are of various ages and have all different kinds of
interests and background knowledge. In addition to traditional desktops, the variety of
computing platforms includes mobile telephones, personal digital assistants (PDAs),
pocket PCs, wearable and immersive environments and many more. In this context,
novel adaptive and adaptable systems are emerging. Faced with this huge set of
propositions, it is very difficult to characterize to what extent a specific application is
adaptive or adaptable. Likewise, it is difficult to identify the new adaptation features
that should be implemented in a system in order to increase its adaptation degree. For
these reasons, it is necessary to characterize all the different kinds of adaptations that
can possibly exist and define a proper way of quantifying the degrees of these
12 F. Tarpin-Bernard, I. Marfisi-Schottman, and H. Habieb-Mammar
adaptations. In order to accomplish a good evaluation framework, a measure of
usability should be added (see Fig. 1). These indicators could be used either to
improve a system by identifying its strengths and weaknesses, or for objectively
comparing several systems of the same family.
1st step 2nd step 3rd step
Identify the types of Measure the degree Test usability
adaptation + of adaptation + = measuring
= characterization = = quantification usability
Fig. 1: The three steps to evaluating a system’s adaptation
In this paper, we present a first proposal, AnAmeter, for characterizing and
quantifying the adaptation of a system. This tool is largely based on our analysis of
the multiple facets of adaptation we will develop in the first section. Then, we present
the core of AnAmeter: a grid that helps characterize the adaptations by crossing the
adaptation factors and the aspects of adaptation. Based on this grid, we build a
quantification technique that provides a measure of the adaptation degree. Finally, the
interest, limits and potential extensions of AnAmeter are discussed.
2 The multiple Facets of adaptation
Many studies have tried to build a taxonomy of adaptive interfaces [1, 2, 3]. Based on
these works and on a first design space provided by Vanderdonckt & al. [4], we
consider that the most important questions to characterize adaptation are:
Who initiates the adaptation?
What are the factors to which the system can adapt?
What aspects of the system are adapted?
In the next part, we look at each of these questions and elaborate a list of answers.
2.1 Who initiates the adaptation?
To analyze systems adaptations in a functional way we chose to use the two possible
sources responsible for initiating the adaptation as identified by Kobsa & al. [5]:
The system itself. In this case, the adaptation is automatically initiated and the
system is called adaptive.
The user. In this case, the adaptation is requested by the user and the system is
called adaptable.
2.2 What are the factors to which the system can adapt?
Usually, the need for adaptation is associated to the notion of context. In the human
computer interaction (HCI) field, a context is generally described according to three
dimensions [3]: the user, the platform and the environment. However, a specific user
placed in the same environment using the same interaction platform could require
AnAmeter: The First Steps to Evaluating Adaptation 13
some adaptation relevant to his/her activity. That is why we define four sets of factors
of adaptation: user, interaction platform, environment and the activity.
In the next part, we itemize these four factors of adaptation into sub-factors. The
lists of sub-factors are not meant to be fully exhaustive, but as we will see later, they
will help evaluate the levels of adaptation provided by a system. This classification
work was done thanks to a detailed review of the state of the art and "context of use"
definitions provide by standards like IEC CDV TR 61997 [6] and ISO 9241-11 [7].
We also analyzed more than 50 systems found in articles or available in public
distribution (complete list on http://liesp.insa-lyon.fr/AnAmeter/References.php).
2.2.1 User
A User model usually refers to various user characteristics [8, 9]. We can group these
characteristics into four sub-factors:
Knowledge and level of experience: The knowledge refers to the user's
theoretical understanding of the subjects treated in the system. The level of
experience refers to the necessary skills to use the system itself. For example, the
systems can adapt the content of the lessons or give more helping tips to users
that are not familiar with the system.
Socio-demographic characteristics and user role: Characteristics such as age,
gender, weight, height, wage, profession, hobbies, cultural preferences and user
role… are useful factors of adaptation for all kinds of systems. When looking for
tourist activities, a GPS system will filter the information showing only the
attractions compatible with the user’s interests and his propensity to spend.
Cognitive abilities and emotional state: The different modes of perceiving,
memorizing, learning, judging etc. and the emotional state [10] (happy, sad,
worried, frustrated, panicked, confident…) may also be considered for adapting.
Perceptual and motor abilities: These characteristics are useful to enable the
systems to be used with disabilities (vision, manipulation, etc.). These disabilities
can range from slight myopia or color blindness to total deafness and paralysis.
2.2.2 Interaction platform
The interaction platform describes the physical characteristics of the devices. The
major characteristics that a system may take into account are the following:
Computing power and autonomy: Systems often need to be adapted to the
platform’s processing power and the memory capacity. For some portable
devices it is also worthwhile to adapt to the battery level by shutting of certain
services for example.
Input/Output device: Some systems are available on a wide variety of platforms.
Certain web browsers adapt to the different screen sizes and input devices such as
mice, keyboards and pens when used on desktops, laptops, or telephones.
Software environment: Computer systems are almost always used alongside other
systems on the same platform. These systems can adapt to cohabitate,
synchronize and even cooperate with each other.
Connectivity: More and more systems are now using network connections. The
connectivity factor is therefore very important. Systems can adapt to cope with
the lack of connection or slow connectivity or even type of network.
14 F. Tarpin-Bernard, I. Marfisi-Schottman, and H. Habieb-Mammar
2.2.3 Environment
The third factor is the environment, a term used to cover the physical, social and
organizational elements that are outside of the interactive system (platform & user).
Human environment: In some cases, systems can adapt to the other people who
are interacting with the user (directly or through the system). This kind of
adaptation can be used for multi-user applications (games, forums, chats), for
applications that adapt to the other users’ actions (commercial websites such as
Amazon that propose frequently bought products or Google that considers the
popularity of websites) or even for applications that detect humans present in the
same physical area.
Machine environment: This type of environment is defined by any reachable
material such as external servers, extra output and input devices (video projector,
motion detector…) which are not part of the Interaction Platform but could be
connected on the fly to the main system.
Ambient characteristics: Systems can also adapt to the luminance, temperature,
the level of noise and the movements of the device.
Spacio-temporal characteristics: Many GPS navigation systems propose potential
interesting tourist areas by using geographic latitude and longitude measures.
Localisation can also be expressed semantically if the system identifies a specific
area such as a room or on a larger scale, a country or time zone.
2.2.4 Activity
The fourth factor is the activity itself. At a micro level, it includes task characteristics
and at a macro level it includes the general activity and the user’s goal.
Task characteristics: The frequency, complexity, dangerousness and
confidentiality character of the task can be taken into account to adapt the system
by using icons and fast links for frequent tasks (favorite links) or extra warning
messages and backup copies for dangerous or confidential tasks.
Task flow: Here, the task is considered as a part of a tasks flow. For example, if
the user usually does task B after task A, the system might set a quick or
automatic launch to task B each time task A is done. The system can also keep a
historic of the different tasks done so that the user can access them faster.
User’s goal: For each activity (combination of tasks), the user can have a
different set of goals. For example, Photoshop® could adapt according to
whether the user is editing photos, looking at a slide show or sorting photos…
General activity: The general nature of the activity weighs heavily in a successful
adaptation. Someone wanting to have fun, for example, will not have the same
way of using a system as someone who wants to learn or work.
2.3 What aspects of the system are Adaptable?
Many aspects of applications can be adapted. We ordered these aspects by using a
common approach of HCI engineering: the PAC (presentation, abstraction, control)
model, developed by Coutaz [11]. This model has the advantage of clearly separating
the functional aspect of the system called “abstraction” from the interface components
called “presentation”. The “control” is in charge of linking these two worlds and thus
AnAmeter: The First Steps to Evaluating Adaptation 15
externalizing the means and rules of communication. In the next section, we clarify
these aspects by using an example of a GPS system.
2.3.1 Abstraction
In this part we will be talking about the adaptation of the information and the data
proposed by the system and the way the different services behave.
Data & information: A GPS system in a car will give different information when
asked for the hotels in the surrounding area. The hotels proposed will depend on
the localization of the car.
Service behavior: A company time-table planner for example will authorize the
boss to take holidays whenever he wants but will send an approval email and
mark the holidays as “to be confirmed” for any other employee.
2.3.2 Control
The control module is in charge of giving access to the services and data available in
the system by interacting in different ways with the user.
Filtering services and data: For various reasons, adaptation might mean limiting
the number of services offered or providing only a partial access to a complex
service. On our GPS system, for instance, the services to find a tourist attraction
are only available when the system is set on “vacation” mode.
Interaction mode: Systems can choose to accept input and deliver output via
many devices. For example, when the car is running, the information on the
screen of the GPS system is read out loud by a voice synthesizer.
2.3.3 Presentation
Spacio-temporal organization: The elements of information can be arranged in a
variety of ways. For example, the GPS system will present the descriptions of the
hotels in a specific order by calculating the distance to the hotel.
Presentation aspects: Finally, we get to the outermost layer of the surface, which
includes elements such as colors, shapes, buttons, boxes, menus, volume, sound
effects... For example, our GPS system will change the colors and the brightness
of the screen when night falls.
3. Characterizing and Quantifying Tool
As mentioned in the introduction, the AnAmeter tool characterizes the adaptation and
measures the quantity of this adaptation. It is important to keep in mind that it does
not yet measure the usability of the adaptation. For a complete evaluation, it would be
necessary to at least measure the utility of the adaptation to make sure the users really
need it, the quality of the implementation to make sure the adaptation is easy to use
but also the efficiency of the adaptation to make sure it actually helps the users work
faster or better. This third step (see Fig. 1) is not in the scope of this paper but will
benefit from our work. We present AnAmeter, an open system to support an iterative
and participative building approach to develop a standard evaluating tool.
16 F. Tarpin-Bernard, I. Marfisi-Schottman, and H. Habieb-Mammar
3.1 Characterizing adaptation
Using the classification presented in the previous section, we can build a first
characterizing grid by crossing the adaptation aspects versus the factors of adaptation.
This grid can be used to break down types of adaptability as well as the types of
adaptivity (Fig. 2). Each factor (respectively aspect) is divided into sub-factors
(respectively sub-aspects). The sub-factors and sub-aspects are also broken down into
elements. For ease of presentation, we have not drawn these last subdivisions but each
cell of the main grid contains a smaller grid composed of these elements which refer
to the finest grain of description. Each lower level cell corresponds to the question
“Does this aspect adapt to this factor?” If this is the case then the cell should be
checked. For example, the system tested in Fig. 2 adapts the “size of the text” and the
“type and color of the background” to the users “myopia”.
Fig. 2 : V1.0 of the main grid and a smaller grid containing aspect and factor elements.
Some of these questions might not make very much sense in certain situations or
for a specific type of system. This is why we add a N/A (non-applicable) option. The
complete evaluation requires filling out two grids (one for the system’s adaptability
degree and one for the system’s adaptivity degree) and therefore answering a long list
of questions. In order to ease the work of the evaluator and speed up the process, we
have built an online tool for handling the grid that only requires the evaluator to check
AnAmeter: The First Steps to Evaluating Adaptation 17
boxes. The tool is available online at http://liesp.insa-lyon.fr/AnAmeter. To make sure
the system was applicable to all adaptive or adaptable systems, we trialled it on a web
browser (Google), a writing and calculating system (Office 2007), an online
bookstore (Amazon) and a communication system (smart phone XDA O2). For the
first evaluations we carried out, filling out one grid took about 60 minutes.
3.2 Quantifying adaptation
Now that we have built a grid to characterize the adaptability and the adaptivity of a
system, we want to quantify these adaptations. Once each cell of the smaller grid
relevant to sub-aspect B and sub-factor C is filled in, an adaptation degree AB/C
ranging from 0 to 3 is automatically calculated according to the number and
distribution of the boxes checked using the rules detailed in Table 1. For example,
Fig. 2 shows the small grid of the sub-aspect “presentation aspects” and the sub-factor
“perceptual/motor abilities”. Once the evaluator clicks on the OK button, the
adaptation degree Apresentation aspects / perceptual, motor abilities will be automatically calculated
according to the number and position of the ticks entered in the grid.
Table 1 : Scoring process for the adaptation degree.
Degree Meaning Reading in the grid Example
The system does not have this
AB/C= 0 No checked boxes.
type of adaptation.
One aspect is adapted to one
AB/C= 1 One checked box.
factor.
One aspect is adapted to several Checked boxes only on
AB/C= 2 factors or several aspects are one row or only on one
adapted to one factor. column.
Checked boxes on at least
More than two aspects adapt to
AB/C= 3 two rows and two
more than two factors.
columns.
When all the cells in the main grid relevant to the aspect B and the factor C are
filled in with a score of 0, 1, 2 or 3, a local adaptation degree, LAB/C is determined by
calculating the average of these scores. The N/A cells will not be considered in the
calculations. The results is then converted into a percentage as shown in equation n°1
(100% corresponds to a score of 3 in all the cells).
AB.item / C .item 100 n cells relevant to B and C
LAB / C m N/A cells relevant to B and C (1)
(n m) 3
Once all the local adaptation degrees AB/j relevant to an aspect B are calculated, the
semi-global aspect adaptation degree AAB can be found with equation n°2. In the
same way, equation n°3 is used to determine the semi-global factor adaptation degree
18 F. Tarpin-Bernard, I. Marfisi-Schottman, and H. Habieb-Mammar
FAc relevant to the factor C.
LAB / j
AAB j {factors} (2)
n
LAi / C
FAC i {aspects} (3)
n
Finally, the global adaptation degree, GA, is determined by taking the average of
the semi-global adaptation degrees - either of all the aspects or of all the factors - as
shown in equation n°4.
AAi FA j j {factors}
GA (4)
n n i {aspects}
To enable easy understanding of these adaptation degrees, we then identify the
aspects and the factors by using the first letter of their name. Also, the adaptation
degree relevant to adaptivity (self-adaptive) will be marked with an apostrophe
(LA'C/A, GA'…). Fig. 3 illustrates an example of these equations:
Fig. 3 : Example of local, semi-global and global adaptation degrees.
4. Discussion
The advantages of the AnAmeter tool are:
1) Its simplicity. The tester fills out the grid by answering simple Boolean questions
of the following type: does a precise aspect of the system adapt to a precise factor?
Clear examples with references are available for each type of adaptation.
2) Its precision. The tool provides precise local evaluations. This is very useful for
AnAmeter: The First Steps to Evaluating Adaptation 19
systems that specialize on a few adaptations. It is also possible to evaluate the
system from two fundamental different points of view: adaptability (user-initiated
adaptation) or adaptivity (automatic system-initiated adaptation).
3) Its completeness. We have tried to build a complete evaluation protocol that
systematically considers all the possible adaptations so that none are left out. We
hope the list of adaptations and the 300 examples of their implementation
available on the AnAmeter platform will also inspire system designers.
4) Its ease for comparing adaptations. AnAmeter can be used to compare two
systems of the same family but also to measure the evolution of a system by
evaluating the effect of adding or withdrawing adaptations.
5) Its extensibility and flexibility. Our idea was to offer a robust basis for the
community to build on. The architecture of our tool makes it easy to extend by
adding other elements, by dividing the sub-categories or extending the measuring
scale.
6) Its accessibility. AnAmeter is freely accessible on the web along with a selection
of completely tested systems (http://liesp.insa-lyon.fr/AnAmeter). This makes it
possible for the same system to be tested by several evaluators who could then
combine their results to obtain a mean value for the adaptation degree.
Although AnAmeter has many advantages, the fact that the approach tries to be as
complete as possible extends the time required to evaluate a system to approximately
one hour. Indeed, this first version of the grid contains 22 aspect elements and 59
factor elements which add up to more than a 1000 Boolean questions to answer for a
highly adaptive or adaptable system. Of course, for most of the systems, entire
sections of the grid will be left out or marked as non-applicable, greatly reducing the
amount a work. By creating an online tool that enables easy manipulation of the grid
and calculates the adaptation degree automatically, we have lightened the task but it is
still represents quite an investment of time and effort. We hope it will be possible to
improve the grid with the scores and the comments of people who use it.
5. Future work
We believe that building an evaluating tool, widely accepted by the community can
only be done in a cooperative way with the help of the members of this community.
AnAmeter was created to serve as a basis for building on and this is why we created
an open, extensible and flexible online framework.
In the near future, we plan on adding a measure of the adaptations usability to
establish a global evaluation mark as seen in the first section (see Fig. 1). In order to
do so, indicators such as utility, quality of the implementation or the added efficiency
brought by the adaptation will have to be measured. These indicators can be found
with different techniques such as interviewing users, analyzing tracking data or
setting up a test session with specific tasks to be accomplished. If the adaptations of
the system have already been identified and quantified with AnAmeter, the evaluators
will have precious information to build an efficient protocol for this third step. Indeed,
they will already have a global view of the important adaptations featured by the
20 F. Tarpin-Bernard, I. Marfisi-Schottman, and H. Habieb-Mammar
systems and have an idea of the factors and aspects to modulate so as to observe
usability indicators.
The next step is to test the AnAmeter tool for ease of use by asking other people to
use it to evaluate systems on their own and send us feedback. Now that AnAmeter is
available on the web is should be easy to launch an evaluation campaign.
Finally, we plan to ask people with different user profiles to test the same system
in order to see if the results coincide or not to measure AnAmeter’s reliability.
6. Conclusion
In this paper, we present AnAmeter, a tool to characterize the multiple facets of
adaptability and a quantification technique to measure the adaptability degree of an
interactive system. We discuss the multiple facets of adaptation, primarily the aspects
and factors of adaptation that serve as parameters. Then, we suggest the use of a
scoring matrix to evaluate local, semi-global and global adaptation of an interactive
system. We provide a first version of the scoring technique and simple formulas for
calculating these adaptation degrees. The AnAmeter tool is presented as a starting
point for the community to cooperatively build a widely accepted framework for
evaluating any kind of adaptable or adaptive systems.
7. References
1. Brusilovsky, P., “User Modeling and User-Adapted Interaction”, Adaptive Hypermedia, Vol.
11, Issue 1-2, 87--110 (2001)
2. Jameson, A., “Adaptive interfaces and agents”, J. A. Jacko & A. Sears (Eds.), The Human
Computer Interaction Handbook, Mahwah, NJ: Erlbaum, 305--330 (2003)
3. Calvary G., J. Coutaz , D. Thevenin, Q. Limbourg, L. Bouillon, J. Vanderdonckt, “A
unifying reference framework for multi-target user interfaces”, Interacting With Computers,
Vol. 15 No. 3, 289--308 (2003)
4. Vanderdonckt J., D. Grolaux , P. Van Roy, Q. Limbourg, B. Macq and B. Michel, “A Design
Space For Context-Sensitive User Interfaces”, Proceedings of IASSE (2005)
5. Kobsa, A., J. Koenemann and W. Pohl, “Personalized Hypermedia Presentation Techniques
for Improving Online Customer Relationships”, The Knowledge Engineering Review, 16(2),
Cambridge University Press, UK, 111--155 (2001)
6. IEC CDV TR 61997, “Guidelines for the user interfaces in multimedia equipment for general
purpose use” (2000)
7. ISO 9241, “Ergonomic requirements for office work with visual display terminals” (1998)
8. De Bra, P., A. Aerts, B. Berden, B. De Lange, B. Rousseau, T. Santic, D. Smits and N. Stash,
“AHA! The Adaptive Hypermedia Architecture”, Proceeding of the ACM Hypertext
Conference, 81--84 (2003)
9. Habieb-Mammar H., F. Tarpin-Bernard “CUMAPH: Cognitive User Modeling for Adaptive
Presentation of Hyper-documents: An experimental study”, AH 2004, Eindhoven University
of Technology, Netherlands, 23--26 (2004)
10. Picard, R.W., “Affective Computing”. In The MIT press, United State (2008)
11. Coutaz J., “PAC: an object oriented model for implementing user interfaces”, Bull., vol. 19,
37--41 (1987)