DEGAS 2007 | Proceedings 10 September 2007 | Rio de Janeiro y Brazil Towards Metrics for Web Accessibility Evaluation Sinésio Teles de Lima Mestrado em Gestão do Conhecimento e da Tecnologia da Informação Universidade Católica de Brasília SGAN 916 Norte Av. W5 - Sala A121 - Asa Norte - Brasília - DF - Brasil - CEP 70790-160 stlima@gmail.com +55 61 3448-7137 Fernanda Lima Universidade Católica de Brasília ferlima@ucb.br +55 61 3448-7121 Káthia Marçal de Oliveira Universidade Católica de Brasília kathia@ucb.br +55 61 3448-7125 can define, among other things, measurable factors, or ABSTRACT Nowadays, accessibility is a crucial factor for Web site indicators that will permit visibility as to the specified development and use, and yet, people with visual service levels that were either violated or respected. disabilities face many accessibility barriers that hinder The present work proposes the use of software metrics the adequate understanding of Web contents. In Brazil, to evaluate the accessibility that can be used to define the Federal Government published a law that formalizes service level indicators for Web accessibility SLAs. The the mandatory accessibility to Governmental Web sites’ following sections present Web accessibility concepts, content. In this context, it is necessary to define ways to our approach to Web accessibility evaluation and the evaluate accessibility to guarantee the quality of these work in progress. sites in this respect. Therefore, this article proposes a means of Web accessibility evaluation through metrics. WEB ACCESSIBILITY Web accessibility means that people with visual, Author Keywords physical, speech, cognitive or neurological disabilities Web accessibility, Metrics. are given the opportunity to perceive, understand, navigate and interact with the Web [7]. According to ACM Classification Keywords [12], to perceive is “to become aware of something via K.5.2 [Legal Aspects of Computing]: Governmental the senses”; to understand is to grasp the meaning of Issues – Regulation; K.4.2 [Computers and Society]: something; and to interact is “to have and effect on each Social Issues; D.2.8 [Metrics]. other”. INTRODUCTION According to W3C [1], evaluating Web sites for Organizations use Web technology to make products accessibility can be done using approaches such as: and services available to potential clients. In this preliminary review, conformance evaluation using context, Web content accessibility is an important accessibility tools to determine if a Web site meets component of quality that must be observed among accessibility standards such as the Web Content services offered by Web site providers. In Brazil, the Accessibility Guidelines (WCAG) and involving users Law Decree 5.296 explicitly states this importance, by in the evaluation. compelling Governmental Web site contents to be in The WCAG [4] proposes 14 general guidelines which accordance with Web accessibility guidelines. are composed of checkpoints that explain specific Therefore, it is crucial to define a means of evaluating accessibility aspects and techniques to use them; and of accessibility in order to allow the elaboration of better priorities that show how critical their implementation is. software contracts between clients and providers. In this The current version of WCAG recommendations is 1.0. aspect, Service Level Agreements (SLAs) are used to In the 2.0 WCAG Working Drafts [5], verification establish minimum quality patterns pertaining to points and priorities have changed to ‘success criteria’ services supplied by providers to their clients [10]. The and ‘conformance levels’. Furthermore, the guidelines SLA’s main component is a service catalog where one were regrouped into four basic principles: “(i) Content 11 DEGAS 2007 | Proceedings 10 September 2007 | Rio de Janeiro y Brazil must be perceivable, (ii) Interface components in the Q4. What is the user’s efficacy in specific task content must be operable, (iii) Content and controls execution while interacting with the Web sites? must be understandable, (iv) Content should be robust enough to work with current and future user agents Q5. What is the percentage of user productivity in (including assistive technologies)”. Each principle specific task execution while interacting with the Web contains general guidelines organized into levels and sites? success criteria. The W3C Consortium describes the relationship between the two versions [3]. Q6. What is the degree of user satisfaction in specific Other methods to evaluate Web accessibility can be task execution while interacting with the Web sites? found in the literature [6][11]. One of these methods, Based on the comparison between the two WCAG called participatory observation [11], proposes the versions, metrics were defined to answer these evaluation of interactive sessions with disabled users questions (Table 1). The data was obtained in two ways: and an observer, in an effort to identify barriers in Web using a questionnaire filled out during participatory access and strategies found by the user to try to observation sessions and using an automatic tool called overcome them. TAW [14]. The tool was chosen, among other reasons, because of its capacity to evaluate pages of arbitrary OUR APPROACH TO EVALUATE WEB depth in a Web site and generating HTML reports with ACCESSIBILITY quantity of violations per page. In this research, the Goal-Question-Metric (GQM) approach [2] was used to define what was going to be Three Brazilian Federal Government Web sites were evaluated. The main idea of GQM is that measurement chosen on which to apply these metrics should be goal-oriented. Initially, an explicit (www.receita.fazenda.gov.br, www.previdencia.gov.br, measurement goal is defined. Subsequently this goal is and www.ibge.gov.br). The choices were based on the refined into several questions that break it down into its results of a Brazilian contest named iBEST Contest, major components. Then, each question is refined into where the best Brazilian sites, divided into different metrics that, when measured, will provide information categories, receive prizes annually. The three sites to answer these questions. By answering the questions above received the best prizes for the Government we will be able to analyze if the goal has been attained. category. The GQM goal for this work is: One task was defined for each site. The first task was to To analyze Web sites, find out whether a person is going to receive income tax returns, by filling out two fields: a number similar to a for the purpose of evaluating, Social Security Number and a dynamically generated with respect to accessibility, image shown as a capcha field (a code shown as a distorted image for security purposes). The second task from the view points of users with disabilities, was to identify the necessary documents to apply for a in the context of Brazilian Federal Government. Social Security application. The third task was to find the Contact part of the site and identify the subjects that According to the definitions presented in the previous can be dealt with by email. section, it was possible to identify that accessibility evaluation means, mainly, to evaluate the user’s During participatory observation sessions, the tasks capacity to perceive, operate and understand Web site were executed by ten users with varying degrees of content. Furthermore, there is a concern, associated visual disability and different Web use expertise. All the with accessibility, with respect to user capacity to reach sessions were conducted in environments with personal the desired objective with an acceptable effort in a computers with keyboard, mouse, speakers, Web satisfactory manner. This aspect is explored by ISO/IEC browsers (Internet Explorer or Firefox Mozilla) and 9126 [8] with respect to quality in use, i.e., the software screen reader software (Jaws or Virtual Vision). product’s capacity to guarantee that users will reach Our approach was presented to the Brazilian scientific their specific goals with efficacy, productivity, security community in previous events related to multimedia and and satisfaction, in specific contexts of use. Based on Web [9] and software quality [10]. These papers show these premises, the following questions were elaborated some of the preliminary results. The current results to evaluate accessibility: (Table 1) show that the Web content levels of perception and understanding increase whenever the Q1. What is the degree of perception with respect to quantity of violations to the WCAG 1.0 checkpoints Web site contents? decreases. Analyzing the results of questions 1, 2 and 3, one can conclude that task 1 had the lowest degree of Q2. What is the degree of ease of operation of Web site perception, operation and understanding, while task 3 contents? obtained the best levels of the accessibility principles. Q3. What is the degree of understanding of Web site contents? 12 DEGAS 2007 | Proceedings 10 September 2007 | Rio de Janeiro y Brazil WORK IN PROGRESS 3. Candwell, B. et al. (Editors) Comparison of WCAG Through the analysis of metrics and obtained data, it 1.0 checkpoints to WCAG 2.0. 2006. was possible to define a preliminary parallel that http://www.w3.org/TR/WCAG20/appendixD.html. indicates the following: the lower the degree of perception, operation and understanding of web content, 4. Chisholm, W., Vanderheiden, G. and Jacobs, I. the lower will be the efficacy, the productivity, and the (Editors) Web Content Accessibility Guidelines 1.0. satisfaction of the users, during task execution with 1999. http://www.w3.org/TR/WCAG10/. those contents. 5. Henry, S. L. (Editor) WCAG 2 FAQ. 2007. The next steps aim to improve the analysis of the http://www.w3.org/WAI/WCAG20/wcag2faq. metrics, in order to produce indicators that can be used 6. Henry, S. L. Just Ask: Integrating Accessibility to compose a service catalog for a Web accessibility Throughout Design. SLA. Accessibility SLAs will be useful by contributing http://www.w3.org/WAI/eval/users.html#ut-access. to accessibility initiatives, in the future, as formal instruments, between Web content developers and 7. Henry, S. L. and EOWG Group (Editors). clients. Introduction to Web Accessibility. 2005 http://www.w3.org/WAI/intro/accessibility.php. REFERENCES 1. Abou-Zahra and EOWG. Evaluating Web Sites for 8. International Organization for Standardization. Accessibility: Overview. ISO/IEC 9126-1. Software Engineering - Product http://www.w3.org/WAI/eval/Overview.html quality – Part 1: Quality model, 2001. 2. Basili, V. R., Caldiera, G. and Rombach, H. D. The 9. Lima, S. T.; Lima, F.; Oliveira, K M. Investigação da Goal Question Metric Paradigm. Encyclopedia of Acessibilidade em Sítios Web do Governo Brasileiro: Software Engineering, v. 1, 528-32. John Wiley & Uma Abordagem para o Estabelecimento de SLA de Sons, 1994. Acessibilidade. Proc Brazilian WebMedia 2006 Symposium, SBC Press (2006). 13 DEGAS 2007 | Proceedings 10 September 2007 | Rio de Janeiro y Brazil 10. Lima, S. T.; Lima, F.; Oliveira, K M. Avaliação da 12. Merriam-Webster Dictionary Acessibilidade de Sítios Web por meio de métricas de http://www.mw.com/dictionary software. Proc Brazilian Software Quality 2007 13. Office for Government Commerce Itil. The Key to Symposium, SBC Press (2007). Manage IT Services: Service Delivery – Version 1.2; 11. Melo, A.M., Baranauskas, M.C.C. and Bonilha, Crow, 2001. F.F.G. Avaliação de acessibilidade na Web com a 14. Web Accessibility Test (TAW). 2006. participação do usuário: um estudo de caso. Proc http://www.tawdis.net/taw3/cms/en Brazilian IHC Symposium, SBC Press (2004). 14