This paper presents an approach to integrate usability evaluations into Model-Driven Web development processes. Our main motivation is to define a generic usability evaluation process which can be instantiated into any concrete Web development process that follows a Model-Driven Development (MDD) approach. A preliminary version of a Web Usability Model was defined in order to support this usability evaluation process at several stages. This Web Usability Model decomposes the usability sub-characteristics (from the Software Quality Model proposed in the ISO/IEC 25000 SQuaRE standard) into other sub-characteristics and measurable attributes. Web metrics are intended to be associated to measurable attributes in order to quantify them. Our approach is intended to perform usability evaluations at several stages of a Web Development process. In this paper, we show how usability evaluations at final user interface (UI) can provide feedback about changes in order to improve usability issues at intermediate artifacts (Platform-Independent Models and Platform-Specific Models) or at transformations rules among these intermediate artifacts.
D.2.9 [Management]: Software quality assurance, D.2.8 [Metrics]: product metrics. H5.2 [User Interfaces]: Evaluation/methodology
Usability in Web applications is a crucial factor since the ease or difficulty that users experience with this kind of systems will determine their success or failure. Web applications are increasing its importance in industrial domains; thereby, the need for usability evaluation methods that are specifically crafted for the Web domain has become critical.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Usability evaluations methods for Web applications can be
supported by a quality model which defines usability as a
quality characteristic that is decomposed into specific attributes
that are easier to measure. Although there are several proposes
in this field, most of these approaches [
This paper presents an approach to integrate usability
evaluation into any Model-Driven Web Development method
by defining a usability evaluation process. This Web Usability
Model has been defined by decomposing the usability
subcharacteristics (from the Software Quality Model proposed in
the ISO/IEC 25000 SQuaRE standard) into other
subcharacteristics and measurable attributes taking into account
ergonomic criteria proposed in Bastien and Scapin [
This paper is organized as follows. Section 2 discusses related work that report usability evaluation processes for Web applications. Section 3 presents our approach to integrate usability evaluations into Model-Driven Web Development. Section 4 presents our Web Usability Model that supports our approach. Section 5 shows a brief example of how the usability evaluation process can be instantiated into a concrete Web development method. We mainly focus on evaluations at final user interface. Finally, Section 6 presents discussions and further work.
There are several approaches that deal with Web usability
evaluation, for instance, Ivory [
Olsina and Rossi [
Calero et al. [
Seffah et al. [
Moraga et al. [
The majority of these approaches evaluate Web applications in order to suggest changes at design or implementation stages. It implies that more efforts and resources must be invested into code maintenance. This fact does not occur in a MDD approach where only the maintenance of models is required since source code can be automatically generated from the intermediate artifacts (PIM and PSM models).
In previous work, Abrahão and Insfran [
Since the adoption of Model-Driven Development (MDD) in
the industrial domain has increased recently, our approach is
intended to integrate usability issues into a Model-Driven Web
Development. Web development methods that follow this
approach such as OO-H [
The usability of a Web application obtained as a result of a MDD process can be assessed at different abstraction levels (PIM, PSM and CM). Our proposal is intended to use a Web Usability Model, which is a set of sub-characteristics decomposed into measurable attributes that can be quantified by metrics. The Web Usability Model can be applied to assess the models from each abstraction level (see Fig.1). However, not all the measurable attributes can be evaluated at all the abstraction levels. The higher abstraction level, the less attributes can be considered. In addition, feedback that is obtained after the artifact evaluation has different targets depending on the abstraction level: 1.
At the PIM level it is possible to assess models that specify the Web application independently of platform details such as: presentation models, navigational models, dialogue models, etc. (1 in Fig.1). The set of measurable attributes that can be evaluated at this level is mainly related to how the information will be accessed by users and how this information will be presented by abstract UI patterns (i.e. navigability, information density, etc). However, this set of attributes may differ depending on the PIM expressiveness from each Web development method. This evaluation will generate a usability report in order to provide feedback about how to correct these PIM models.
At the PSM level it is possible to assess the concrete interface models related to a specific platform (2 in Fig.1). The set of measurable attributes that can be evaluated at this level is wider since it includes attributes related with specific software components (widgets) that cannot be considered at PIM level (i.e. behavior of explore bars, visual feedback from radio buttons, etc).This evaluation will generate a usability report in order to provide feedback to previous stages about how to correct the PIM and PSM models, as well as the transformation rules among them.
At the CM level it is possible to evaluate the final user interface (3 in Fig.1). The set of measurable attributes that can be evaluated at this level is the widest since more aspects related to the end-user perspective can be considered (i.e. browser compatibility, metaphor recognition, subjective appealing, etc). This evaluation will also generate a usability report in order to provide feedback to previous stages about how to correct the PIM and PSM models, as well as the transformation rules among them, and code generation rules among PSM and CM.
The former evaluations can be applied in an iterative way until
the models (PIM, PSM, and CM) have the required level of
usability. In order to integrate these evaluations into a
framework, a usability evaluation process should be defined as
an inspection method that guides evaluators on how the Web
Usability Model can be applied. This inspection method could
be defined in order to be compliant with the Quality Evaluation
Division proposed in the ISO/IEC 2504n SQuaRE series [
Establish evaluation requirements such as the purpose of evaluation, identification of Web application type, and selection of the more relevant sub-characteristics of the Web Usability Model taking into account the users’ needs.
Specify the evaluation concerning with the establishment of the artifacts to be evaluated (PIM, PSM or CM); selection of metrics associated to the attributes selected from the Web Usability Model; specification of the calculation formulas of these metrics taking into account the abstraction level of the artifact and the modeling primitives from the concrete Model-Driven Web development method; establishment of rating levels for these metrics; establishment of criteria for global assessment; and the definition of templates to report usability problems.
Design the evaluation plan describing the evaluator tasks schedule.
Execute the evaluation by applying the selected Web metrics in order to detect usability problems.
Generate the usability reports providing feedback in order to improve the intermediate artifacts (PIM and PSM) or transformation rules. 6.
Analysis of changes suggested by usability reports and selection of the alternatives taking into account criteria such as level and priority of usability problems, resources needed to apply changes, etc.
It should be noted that this process is defined to be instantiated into any concrete Model-Driven Web Development method. The instantiation implies to know the modeling primitives of the concrete Model-Driven Web development method in order to be able to specify the calculation formula of the metrics and to understand the traceability between models. This traceability helps the evaluator to establish the source of the usability problems (PIMs, PSMs or transformations rules among them).
The SQuaRE standard [
Our Web Usability Model is an adaptation and extension from
the usability model for model-driven development presented in
Abrahão and Insfran [
The main quality characteristics of the software quality model
are: functionality, security, interoperability, reliability,
operability (usability) and efficiency. Although the term
operability and ease of use have been proposed in SQuaRE to
rename usability and operability sub-characteristic,
respectively, we prefer to use the term usability and operability
in this work to avoid misunderstandings in terminology.
Usability can be decomposed into the five sub-characteristics
proposed in SQuaRE [
The former three sub-characteristics were decomposed into
other sub-characteristics or measurable attributes, taking into
account the ergonomic criteria proposed in Bastien and Scapin
[
The last two sub-characteristics are related to the perception of
the end-user (attractiveness) or evaluator (compliance) using
the Web Application:
iv. Attractiveness refers to the attributes of a Web application
that are related to the aesthetic design. They can be
quantified by measuring the UI uniformity in terms of font
style (color, face and size), background color, and position
of elements.
v. Compliance can be measured by assessing the agreement of
the proposed Web Usability Model with respect to the
standard SQuaRE and several Web design style guides.
Once the sub-characteristics have been identified, Web metrics
are associated to the measurable attributes in order to quantify
them. Values obtained from these Web metrics will allow us to
interpret if measurable attributes contribute to achieving certain
usability level in the Web application. The metrics included in
our model were mainly extracted from the survey presented in
Calero et al. [
As an example, we show some definitions of new proposed metrics that can be associated to attributes of the Web Usability Model: •
Number of different font styles for textual links: This metric is defined as number of different font style combinations (size, face, and color) for all textual links in the same navigation category. (Scale type: absolute value greater or equal to 1). The interpretation is: more than one style combination in the same navigation category means that font style uniformity is not insured. This metric is associated to the font style uniformity attribute, which belongs to the attractiveness sub-characteristic (iv). •
Proportion of elements that show current user state: This metric is defined as the ratio between the number of elements that show feedback about the current user state and the total number of elements that are required to have this feedback capability. (Scale type: ratio between 0 and 1). The interpretation is: values closer to 1 indicate that user can obtain feedback about his/her current state in the Web application. This metric is associated to the immediate feedback attribute, which belongs to the user guidance subcharacteristic (ii. e).
Web metrics definitions from the Web Usability Model are generic, and their calculation formula must be instantiated by identifying variables from this formula in the modeling primitives of the concrete Web development method for each abstraction level (PIM, PSM or CM). Not all the metrics can be defined at all the abstraction levels, for instance, the former metric can be applied at PIM level (if style properties are defined at the abstract UI) or at CM level (if style properties are defined in Cascading Style Sheets files). However, the second metric only can be defined at PSM or CM level since the feedback depends on the widget behavior from the concrete interface.
In this section, we show an overview of how the previous
usability process can be instantiated into a concrete Web
development methodology. As an example, we selected the
OOH [
The OO-H method [
Next, we show as an example, a brief description about the steps involved in our usability evaluation process.
Step 1 (See Section 3): The purpose is to evaluate the usability of a Web application developed following the OO-H method. The selected Web application is a task management system developed for a Web development company located in Alicante, Spain. Finally, the attributes chosen were font style uniformity to evaluate the attractiveness sub-characteristic, and immediate feedback to evaluate the user guidance sub-characteristic, at least to some extent.
Step 2 (See Section 3): The artifacts selected for this evaluation were the final UIs (Code Model). The metrics selected to evaluate the previous attributes were Number of different font styles for text links and Proportion of elements that show current user state (see explanation of each metric in Section 4). The rating level for the former metric was established at no UP for values equal to 1; low UP for values equal to 2; medium UP for values equal to 3; and critical UP for values greater than 3. The rating level for the second metric was established at no UP for values equal to 1; low UP for values in the range [0.8, 1}; medium UP for values [0.5, 0.8} and critical UP for values [0, 0.5}. The usability report is defined as a list of usability problems (UP) detected with the next fields: description of the UP, level of the UP (critical, medium, or low), source of the problem (model), occurrences, and recommendations to correct it. More fields can be defined such as priority, impact, etc. Step 3 (See Section 3): In this case, the evaluator is the same developer. The task assigned was the evaluation of all the user interfaces (CM) in order to present a usability report which will contain the usability problems detected with all the proposed fields filled in.
Step 4 (See Section 3): As an example, we only show the execution of the evaluation of one user interface (CM). Figure 2 shows a user interface automatically generated (Code Model) that represents the task management functionality of the Web application.
User interface automatically
Number of different font styles for textual links 1: The textual links that appears in the user interface (Fig. 2) are Tasks, Reports, Contacts and Exit from the top menu; and New Folder, All tasks, Pending tasks, Ended tasks, and Tasks out of date from the left menu. In the first navigation category (top menu), the value of the metric is 2 since the links Tasks, Reports, Contacts are displayed in a different style from the Exit link, which is displayed in a different color and it is also underlined. In the second navigation category (left menu), the value of the metric is also 2 since the links New Folder, Pending tasks, Ended tasks, and Tasks out of date Contacts are displayed in a different style from the All tasks, which is displayed in a different font face and font size. The rating level of the metric (see Step 2) indicates the existence of a low usability problem (UP001) for each menu.
Proportion of elements that show current user state 1: The user interface must show the current user state, it means, the current section and the current task that is being performed. There are three types of elements that show the current user state in the Web application: the tabs from the top menu (Tasks, Reports, and Contacts); the shape changes of the cursor when it is pointing on a textbox; and the highlight effects of a textbox when it has focus. Since the tabs are the only type of element that does not explicitly show the section in which the user is 1 It should be note that both metrics must be applied to all the user interfaces of the entire Web application. currently interacting, the value of the metric is 2/3=0.66. The rating level of the metric (see Step 2) indicates the existence of a medium usability problem (UP002).
Steps 5 and 6 (See Section 3): The usability problems detected after applying the previous metrics, can be explained in a usability report that contains the UP001 (See Table 1) and the UP002 (See Table 2).
links Tasks, Reports, Contacts and all tasks in the Abstract Presentation Diagram. In this PIM model font style properties can be defined.
The transformation rule that maps the representation of the tabs: Task, Reports and Contacts (PIM level) with the specific widget of the platform that shows the tabs (PSM). 1 occurrence for each UI that shows these tabs.
The widget target of the transformation rule should be changed for other widget with a highlight feature when a tab is clicked. After analyzing and applying the proposed recommendations, a more usable Web application can be obtained without to need maintenance of source code.
This paper has presented a proposal in progress to integrate a
usability evaluation process into Model-Driven Web
development processes. The purpose of our work is to give an
outline of a generic usability evaluation process supported by a
Web Usability Model. A preliminary version of a usability
evaluation process supported by a Web usability Model has
been presented. Our Web Usability Model decomposes the
usability sub-characteristics (from the Software Quality Model
proposed in the ISO/IEC 25000 SQuaRE standard) into other
sub-characteristics and measurable attributes taking into
account ergonomic criteria. Web metrics were associated to
measurable attributes in order to quantify them. Finally, a brief
example has been shown in order to illustrate how the usability
evaluation process can be instantiated into a concrete Web
development method that follows the MDD approach. Although
our example only shows a CM evaluation providing feedback to
PIM models or transformations between PIM and PSM models,
the usability evaluation process can evaluate intermediate
artifacts (PIM and PSM models) by selecting metrics that their
calculation formula has been defined to be applied to concrete
PIM and PSM models (i.e., depth and breadth of a navigational
map [
We believe that the inherent features of model-driven
development processes (e.g., traceability between models by
means of model transformations) provide a suitable
environment for performing usability evaluations. Specifically,
if the usability of an automatically generated user interface can
be assessed, the usability of any future user interface produced
by this approach could be predicted. In other words, we are
talking about a user interface that is usable by construction [
In this way, usability can be taken into account throughout the entire Web development process, enabling Web applications to be developed with better quality thereby reducing effort at the maintenance stage.
• • • • •
values obtained by individual metrics, and perform analyses of the impact on how the attributes affect (negatively or positively) other attributes of the Web Usability Model.
This work is financed by META project (ref. TIN2006-15175C05-05), the Quality-driven Model Transformation Project from the Universidad Politécnica de Valencia. The authors thank Jaime Gomez from Universidad de Alicante for his valuable help in providing the generated Web application and its models used to illustrate our usability evaluation process.