Organizations use Web technology to make products and services available to potential clients. In this context, Web content accessibility is an important component of quality that must be observed among services offered by Web site providers. In Brazil, the Law Decree 5.296 explicitly states this importance, by compelling Governmental Web site contents to be in accordance with Web accessibility guidelines. Therefore, it is crucial to define a means of evaluating accessibility in order to allow the elaboration of better software contracts between clients and providers. In this aspect, Service Level Agreements (SLAs) are used to establish minimum quality patterns pertaining to services supplied by providers to their clients [10]. The SLA's main component is a service catalog where one can define, among other things, measurable factors, or indicators that will permit visibility as to the specified service levels that were either violated or respected.
The present work proposes the use of software metrics to evaluate the accessibility that can be used to define service level indicators for Web accessibility SLAs. The following sections present Web accessibility concepts, our approach to Web accessibility evaluation and the work in progress.
Web accessibility means that people with visual, physical, speech, cognitive or neurological disabilities are given the opportunity to perceive, understand, navigate and interact with the Web [7]. According to [12], to perceive is "to become aware of something via the senses"; to understand is to grasp the meaning of something; and to interact is "to have and effect on each other".
According to W3C [1], evaluating Web sites for accessibility can be done using approaches such as: preliminary review, conformance evaluation using accessibility tools to determine if a Web site meets accessibility standards such as the Web Content Accessibility Guidelines (WCAG) and involving users in the evaluation.
The WCAG [4] proposes 14 general guidelines which are composed of checkpoints that explain specific accessibility aspects and techniques to use them; and of priorities that show how critical their implementation is. The current version of WCAG recommendations is 1.0. In the 2.0 WCAG Working Drafts [5], verification points and priorities have changed to 'success criteria' and 'conformance levels'. Furthermore, the guidelines were regrouped into four basic principles: "(i) Content 12 must be perceivable, (ii) Interface components in the content must be operable, (iii) Content and controls must be understandable, (iv) Content should be robust enough to work with current and future user agents (including assistive technologies)". Each principle contains general guidelines organized into levels and success criteria. The W3C Consortium describes the relationship between the two versions [3].
Other methods to evaluate Web accessibility can be found in the literature [6] [11]. One of these methods, called participatory observation [11], proposes the evaluation of interactive sessions with disabled users and an observer, in an effort to identify barriers in Web access and strategies found by the user to try to overcome them.
In this research, the Goal-Question-Metric (GQM) approach [2] was used to define what was going to be evaluated. The main idea of GQM is that measurement should be goal-oriented. Initially, an explicit measurement goal is defined. Subsequently this goal is refined into several questions that break it down into its major components. Then, each question is refined into metrics that, when measured, will provide information to answer these questions. By answering the questions we will be able to analyze if the goal has been attained.
To analyze Web sites, for the purpose of evaluating, with respect to accessibility, from the view points of users with disabilities, in the context of Brazilian Federal Government.
According to the definitions presented in the previous section, it was possible to identify that accessibility evaluation means, mainly, to evaluate the user's capacity to perceive, operate and understand Web site content. Furthermore, there is a concern, associated with accessibility, with respect to user capacity to reach the desired objective with an acceptable effort in a satisfactory manner. This aspect is explored by ISO/IEC 9126 [8] with respect to quality in use, i.e., the software product's capacity to guarantee that users will reach their specific goals with efficacy, productivity, security and satisfaction, in specific contexts of use. Based on these premises, the following questions were elaborated to evaluate accessibility: Q1. What is the degree of perception with respect to Web site contents? Q2. What is the degree of ease of operation of Web site contents? Q3. What is the degree of understanding of Web site contents?
Q4. What is the user's efficacy in specific task execution while interacting with the Web sites? Q5. What is the percentage of user productivity in specific task execution while interacting with the Web sites? Q6. What is the degree of user satisfaction in specific task execution while interacting with the Web sites?
Based on the comparison between the two WCAG versions, metrics were defined to answer these questions (Table 1). The data was obtained in two ways: using a questionnaire filled out during participatory observation sessions and using an automatic tool called TAW [14]. The tool was chosen, among other reasons, because of its capacity to evaluate pages of arbitrary depth in a Web site and generating HTML reports with quantity of violations per page.
Three Brazilian Federal Government Web sites were chosen on which to apply these metrics (www.receita.fazenda.gov.br, www.previdencia.gov.br, and www.ibge.gov.br). The choices were based on the results of a Brazilian contest named iBEST Contest, where the best Brazilian sites, divided into different categories, receive prizes annually. The three sites above received the best prizes for the Government category.
One task was defined for each site. The first task was to find out whether a person is going to receive income tax returns, by filling out two fields: a number similar to a Social Security Number and a dynamically generated image shown as a capcha field (a code shown as a distorted image for security purposes). The second task was to identify the necessary documents to apply for a Social Security application. The third task was to find the Contact part of the site and identify the subjects that can be dealt with by email.
During participatory observation sessions, the tasks were executed by ten users with varying degrees of visual disability and different Web use expertise. All the sessions were conducted in environments with personal computers with keyboard, mouse, speakers, Web browsers (Internet Explorer or Firefox Mozilla) and screen reader software (Jaws or Virtual Vision).
Our approach was presented to the Brazilian scientific community in previous events related to multimedia and Web [9] and software quality [10]. These papers show some of the preliminary results. The current results (Table 1) show that the Web content levels of perception and understanding increase whenever the quantity of violations to the WCAG 1.0 checkpoints decreases. Analyzing the results of questions 1, 2 and 3, one can conclude that task 1 had the lowest degree of perception, operation and understanding, while task 3 obtained the best levels of the accessibility principles.
Through the analysis of metrics and obtained data, it was possible to define a preliminary parallel that indicates the following: the lower the degree of perception, operation and understanding of web content, the lower will be the efficacy, the productivity, and the satisfaction of the users, during task execution with those contents.
The next steps aim to improve the analysis of the metrics, in order to produce indicators that can be used to compose a service catalog for a Web accessibility SLA. Accessibility SLAs will be useful by contributing to accessibility initiatives, in the future, as formal instruments, between Web content developers and clients.