=Paper=
{{Paper
|id=Vol-285/paper-5
|storemode=property
|title=Towards Metrics for Web Accessibility Evaluation
|pdfUrl=https://ceur-ws.org/Vol-285/paper04.pdf
|volume=Vol-285
|dblpUrl=https://dblp.org/rec/conf/interact/LimaLO07
}}
==Towards Metrics for Web Accessibility Evaluation==
https://ceur-ws.org/Vol-285/paper04.pdf
DEGAS 2007 | Proceedings 10 September 2007 | Rio de Janeiro y Brazil
Towards Metrics for Web Accessibility Evaluation
Sinésio Teles de Lima
Mestrado em Gestão do Conhecimento e da Tecnologia da Informação
Universidade Católica de Brasília
SGAN 916 Norte Av. W5 - Sala A121 - Asa Norte - Brasília - DF - Brasil - CEP 70790-160
stlima@gmail.com
+55 61 3448-7137
Fernanda Lima
Universidade Católica de Brasília
ferlima@ucb.br
+55 61 3448-7121
Káthia Marçal de Oliveira
Universidade Católica de Brasília
kathia@ucb.br
+55 61 3448-7125
can define, among other things, measurable factors, or
ABSTRACT
Nowadays, accessibility is a crucial factor for Web site indicators that will permit visibility as to the specified
development and use, and yet, people with visual service levels that were either violated or respected.
disabilities face many accessibility barriers that hinder The present work proposes the use of software metrics
the adequate understanding of Web contents. In Brazil, to evaluate the accessibility that can be used to define
the Federal Government published a law that formalizes service level indicators for Web accessibility SLAs. The
the mandatory accessibility to Governmental Web sites’ following sections present Web accessibility concepts,
content. In this context, it is necessary to define ways to our approach to Web accessibility evaluation and the
evaluate accessibility to guarantee the quality of these work in progress.
sites in this respect. Therefore, this article proposes a
means of Web accessibility evaluation through metrics. WEB ACCESSIBILITY
Web accessibility means that people with visual,
Author Keywords physical, speech, cognitive or neurological disabilities
Web accessibility, Metrics. are given the opportunity to perceive, understand,
navigate and interact with the Web [7]. According to
ACM Classification Keywords [12], to perceive is “to become aware of something via
K.5.2 [Legal Aspects of Computing]: Governmental the senses”; to understand is to grasp the meaning of
Issues – Regulation; K.4.2 [Computers and Society]: something; and to interact is “to have and effect on each
Social Issues; D.2.8 [Metrics]. other”.
INTRODUCTION According to W3C [1], evaluating Web sites for
Organizations use Web technology to make products accessibility can be done using approaches such as:
and services available to potential clients. In this preliminary review, conformance evaluation using
context, Web content accessibility is an important accessibility tools to determine if a Web site meets
component of quality that must be observed among accessibility standards such as the Web Content
services offered by Web site providers. In Brazil, the Accessibility Guidelines (WCAG) and involving users
Law Decree 5.296 explicitly states this importance, by in the evaluation.
compelling Governmental Web site contents to be in The WCAG [4] proposes 14 general guidelines which
accordance with Web accessibility guidelines. are composed of checkpoints that explain specific
Therefore, it is crucial to define a means of evaluating accessibility aspects and techniques to use them; and of
accessibility in order to allow the elaboration of better priorities that show how critical their implementation is.
software contracts between clients and providers. In this The current version of WCAG recommendations is 1.0.
aspect, Service Level Agreements (SLAs) are used to In the 2.0 WCAG Working Drafts [5], verification
establish minimum quality patterns pertaining to points and priorities have changed to ‘success criteria’
services supplied by providers to their clients [10]. The and ‘conformance levels’. Furthermore, the guidelines
SLA’s main component is a service catalog where one were regrouped into four basic principles: “(i) Content
11
DEGAS 2007 | Proceedings 10 September 2007 | Rio de Janeiro y Brazil
must be perceivable, (ii) Interface components in the Q4. What is the user’s efficacy in specific task
content must be operable, (iii) Content and controls execution while interacting with the Web sites?
must be understandable, (iv) Content should be robust
enough to work with current and future user agents Q5. What is the percentage of user productivity in
(including assistive technologies)”. Each principle specific task execution while interacting with the Web
contains general guidelines organized into levels and sites?
success criteria. The W3C Consortium describes the
relationship between the two versions [3]. Q6. What is the degree of user satisfaction in specific
Other methods to evaluate Web accessibility can be task execution while interacting with the Web sites?
found in the literature [6][11]. One of these methods, Based on the comparison between the two WCAG
called participatory observation [11], proposes the versions, metrics were defined to answer these
evaluation of interactive sessions with disabled users questions (Table 1). The data was obtained in two ways:
and an observer, in an effort to identify barriers in Web using a questionnaire filled out during participatory
access and strategies found by the user to try to observation sessions and using an automatic tool called
overcome them. TAW [14]. The tool was chosen, among other reasons,
because of its capacity to evaluate pages of arbitrary
OUR APPROACH TO EVALUATE WEB depth in a Web site and generating HTML reports with
ACCESSIBILITY quantity of violations per page.
In this research, the Goal-Question-Metric (GQM)
approach [2] was used to define what was going to be Three Brazilian Federal Government Web sites were
evaluated. The main idea of GQM is that measurement chosen on which to apply these metrics
should be goal-oriented. Initially, an explicit (www.receita.fazenda.gov.br, www.previdencia.gov.br,
measurement goal is defined. Subsequently this goal is and www.ibge.gov.br). The choices were based on the
refined into several questions that break it down into its results of a Brazilian contest named iBEST Contest,
major components. Then, each question is refined into where the best Brazilian sites, divided into different
metrics that, when measured, will provide information categories, receive prizes annually. The three sites
to answer these questions. By answering the questions above received the best prizes for the Government
we will be able to analyze if the goal has been attained. category.
The GQM goal for this work is: One task was defined for each site. The first task was to
To analyze Web sites, find out whether a person is going to receive income tax
returns, by filling out two fields: a number similar to a
for the purpose of evaluating, Social Security Number and a dynamically generated
with respect to accessibility, image shown as a capcha field (a code shown as a
distorted image for security purposes). The second task
from the view points of users with disabilities, was to identify the necessary documents to apply for a
in the context of Brazilian Federal Government. Social Security application. The third task was to find
the Contact part of the site and identify the subjects that
According to the definitions presented in the previous can be dealt with by email.
section, it was possible to identify that accessibility
evaluation means, mainly, to evaluate the user’s During participatory observation sessions, the tasks
capacity to perceive, operate and understand Web site were executed by ten users with varying degrees of
content. Furthermore, there is a concern, associated visual disability and different Web use expertise. All the
with accessibility, with respect to user capacity to reach sessions were conducted in environments with personal
the desired objective with an acceptable effort in a computers with keyboard, mouse, speakers, Web
satisfactory manner. This aspect is explored by ISO/IEC browsers (Internet Explorer or Firefox Mozilla) and
9126 [8] with respect to quality in use, i.e., the software screen reader software (Jaws or Virtual Vision).
product’s capacity to guarantee that users will reach Our approach was presented to the Brazilian scientific
their specific goals with efficacy, productivity, security community in previous events related to multimedia and
and satisfaction, in specific contexts of use. Based on Web [9] and software quality [10]. These papers show
these premises, the following questions were elaborated some of the preliminary results. The current results
to evaluate accessibility: (Table 1) show that the Web content levels of
perception and understanding increase whenever the
Q1. What is the degree of perception with respect to quantity of violations to the WCAG 1.0 checkpoints
Web site contents? decreases. Analyzing the results of questions 1, 2 and 3,
one can conclude that task 1 had the lowest degree of
Q2. What is the degree of ease of operation of Web site perception, operation and understanding, while task 3
contents? obtained the best levels of the accessibility principles.
Q3. What is the degree of understanding of Web site
contents?
12
DEGAS 2007 | Proceedings 10 September 2007 | Rio de Janeiro y Brazil
WORK IN PROGRESS 3. Candwell, B. et al. (Editors) Comparison of WCAG
Through the analysis of metrics and obtained data, it 1.0 checkpoints to WCAG 2.0. 2006.
was possible to define a preliminary parallel that http://www.w3.org/TR/WCAG20/appendixD.html.
indicates the following: the lower the degree of
perception, operation and understanding of web content, 4. Chisholm, W., Vanderheiden, G. and Jacobs, I.
the lower will be the efficacy, the productivity, and the (Editors) Web Content Accessibility Guidelines 1.0.
satisfaction of the users, during task execution with 1999. http://www.w3.org/TR/WCAG10/.
those contents. 5. Henry, S. L. (Editor) WCAG 2 FAQ. 2007.
The next steps aim to improve the analysis of the http://www.w3.org/WAI/WCAG20/wcag2faq.
metrics, in order to produce indicators that can be used 6. Henry, S. L. Just Ask: Integrating Accessibility
to compose a service catalog for a Web accessibility Throughout Design.
SLA. Accessibility SLAs will be useful by contributing http://www.w3.org/WAI/eval/users.html#ut-access.
to accessibility initiatives, in the future, as formal
instruments, between Web content developers and 7. Henry, S. L. and EOWG Group (Editors).
clients. Introduction to Web Accessibility. 2005
http://www.w3.org/WAI/intro/accessibility.php.
REFERENCES
1. Abou-Zahra and EOWG. Evaluating Web Sites for 8. International Organization for Standardization.
Accessibility: Overview. ISO/IEC 9126-1. Software Engineering - Product
http://www.w3.org/WAI/eval/Overview.html quality – Part 1: Quality model, 2001.
2. Basili, V. R., Caldiera, G. and Rombach, H. D. The 9. Lima, S. T.; Lima, F.; Oliveira, K M. Investigação da
Goal Question Metric Paradigm. Encyclopedia of Acessibilidade em Sítios Web do Governo Brasileiro:
Software Engineering, v. 1, 528-32. John Wiley & Uma Abordagem para o Estabelecimento de SLA de
Sons, 1994. Acessibilidade. Proc Brazilian WebMedia 2006
Symposium, SBC Press (2006).
13
DEGAS 2007 | Proceedings 10 September 2007 | Rio de Janeiro y Brazil
10. Lima, S. T.; Lima, F.; Oliveira, K M. Avaliação da 12. Merriam-Webster Dictionary
Acessibilidade de Sítios Web por meio de métricas de http://www.mw.com/dictionary
software. Proc Brazilian Software Quality 2007
13. Office for Government Commerce Itil. The Key to
Symposium, SBC Press (2007).
Manage IT Services: Service Delivery – Version 1.2;
11. Melo, A.M., Baranauskas, M.C.C. and Bonilha, Crow, 2001.
F.F.G. Avaliação de acessibilidade na Web com a
14. Web Accessibility Test (TAW). 2006.
participação do usuário: um estudo de caso. Proc
http://www.tawdis.net/taw3/cms/en
Brazilian IHC Symposium, SBC Press (2004).
14