We applied a combination of accessibility evaluation methods (AEMs) in this research to provide an overview of the web accessibility level of chatbots included on a selected sample of Spanish city council websites. The methodology proposed is user-centred design to provide the most extensive web accessibility analysis results. For this purpose, the use of heuristics, automatic tools and end -user evaluations has been combined. For these AEMs, we included two types of assessors: university undergraduate students as trained evaluators over one academic semester and end users with visual accessibility needs. The sample to be evaluated for this pilot research comprises chatbots embedded in five Spanish city council websites. The added value of combining AEMs is to achieve a better implementation of the Web Content Accessibility Guidelines (WCAG) framework and precise feedback from expert users with visual accessibility needs. Results indicate key accessibility barriers in the sample analysed, making the chatbots and their websites inaccessible to users with accessibility needs. While the accessibility training process of university undergraduate students is useful for skills development and increases the accuracy of WCAG evaluation, the combination of AEMs improves the quality of the results, making input from end users with accessibility needs essential to achieve a holistic and precise analysis.
public web sites in several parts of the world: 17 studies in Finland[
At least in the European context, the state of the art has improved significantly in the last two
decades. In 2005, the European Public Administration Network (EPAN) [
In the Spanish context, [
Quite recently, the interaction between citizens and government institutions has undergone a significant transformation. Technological advances related to natural language processing have brought new possibilities for improving the efficiency and accessibility of public services. One of the most outstanding developments is the implementation of chatbots in city council websites, which have proven to be an effective and easy tool to facilitate communication and citizen support [16].
These chatbots can integrate audio, visual, text-based, and graphical elements, including buttons or other user interface components as part of the interaction, which can enhance access to support for individuals with accessibility needs [17]. They can pefrorm tasks through text-based interactions or by interpreting human speech and responding with synthesised voices, offering flexibility, personalisation, and alternative communication methods [18]. However, research indicates that these tools often lack accessibility features [19, 20].
The main objective of this research was to understand the current state of accessibility of city council chatbots and their websites. For that purpose, we used a combination of accessibility evaluation methods (AEMs) to assess the web accessibility of chat bots on Spanish city council websites. Our methodology combined heuristic evaluations supported by automated tools and enduser evaluations. We included two types of evaluators: university undergraduate students acting as trained evaluators after one semester of academic work and end users with visual accessibility needs. The pilot study evaluated chatbots embedded in five Spanish city council websites.
Previous research indicates the importance of using several AEMs [21], heuristic evaluations supported by automatic tools can be intricate and may sometimes yield incorrect results, such as false positives. [22] notes that the ambiguity of the evaluation process can lead different evaluators to concentrate on various aspects that might not be directly related to the criterion being assessed, also the determination of success or error is subjective and varies from evaluator to evaluator. The combination of heuristic evaluations with end-user evaluations strengthens the process allowing the identification of accessibility barriers [23].
Considering the limited research in the accessibility assessment of public administration websites
and chatbots we acknowledge in this pilot research we used a to-pdown user-centred methodology:
1. Full analysis of websites. A review of city council websites with WCAG 2.1 by trained
students in the "Usability and Accessibility" course, part of the degrees in Information
Technology Engineering and Computer Engineering at Universidad Nacional de Educación
a Distancia (UNED). To include a centred user perspective in the course, students had to
evaluate their city council website by applying a service -learning approach, so students
understand their local context and reality [15]. Students had to fill in a WCAG checklist
applying heuristic evaluation supported by automatic tools, in this case, the browser plugin
for WAVE, and rate each WCAG criterion with the values: (1) not achieved, the feature to
test is missing, (2) partially achieved, the feature to test is available but not integrated, (3) not
applicable, (4) largely achieved, the feature to test is available and partially integrated, and
(5) fully achieved, the feature to test is available and fully integrated. The checklist included
levels A and AA which are those required by the public administration legislation [
of users with visual accessibility needs in the context of research at Universidad Complutense de Madrid [24]. For that purpose, this study used key aspects of the POUR principles from WCAG: (1) perceivable, if non-text elements have alt text, content maintains structure across formats, and there is enough contrast for readability, (2) operable, whether users have enough time to understand chatbot messages and whether navigation is clear, with focus indicators showing where the user is on the screen, (3) understandable, if the chatbot's language is easy to read and if it assists users with things like correcting spelling errors and, (4) robust, the chatbot works well with assistive tools like screen readers. In addition to WCAG, two more criteria were considered: the chatbot should support voice-based interaction and be capable of understanding and responding to user commands like ‘repeat that’ or ‘connect me to an agent’. The end-user evaluations were conducted in July 2024 using mainly smartphones. 2.2.
Following the inverted pyramid approach, we analysed five city council websites and their chatbots from cities in Spain, selecting five chatbots each developed by a different company to ensure a diverse sample of technical and accessibility approaches, see Table1.
These chatbots were chosen for their variety in design and development, all being recent projects by companies with digital accessibility expertise, increasing the likelihood that they meet current accessibility standards.
The city council websites were evaluated by students in the "Usability and Accessibility" course as one of the assignments during the course using a quantitative approach [25] where students rated each of the criteria based on a Likert scale. In the case o f the end users, a qualitative method was
used to capture the interaction via observation [26]. Four participants with different levels of visual impairment wererecruited. Several of them have priorexperienceevaluating mobileapp accessibility through previous research at Universidad Complutens e de Madrid:
Víctor Alberto. Can only perceive light and shadows; uses Apple’s VoiceOver screen reader on his smartphone.
Jesús. Blind; uses VoiceOver on smartphone and operates both Windows and Mac desktop computers.
Gemma. Uses VoiceOver on her smartphone but engages with her device less frequently. Margarita. Has 10% vision in one eye and only light perception in the other; uses her smartphone extensively with both VoiceOver and a magnifying glass.
In both analyses of data, results are shown divided by strengths and barriers. 3.1
Results shown in the following tables were coded to range from 1 (very lonwot- achieved) to 5 (very high – fully achieved). Thresholds to select criteria are 4.6 or above for strengths and 3.6 or below for barriers, while if a criterium has 3 or more non -applicable values (i.e., 3) is not included as a barrier for lack of consistency within the sample.
• • • • • • • • • • • • • • The “Perceivable” principle ensures that information and user interface components are presentable to users in ways they can perceive (See Table 2).
Orientation: content can be viewed and operated in both portrait and landscape orientations.
Use of Colour: colour is not used as the only visual means of providing information. Resize Text: text can beresized without assistive technology, maintaining contentand functionality.
Images of Text: when possible, text is used to convey information instead of images of text. Reflow: content can be displayed without losing information or functionality, and without requiring both vertical and horizontal scrolling.
Text Spacing: users can adjust text spacing to improve readability.
synchronised media does not include captions or audio descriptions, and both audio and video media do not provide an alternative.
Info and Relationships: information, structure, and relationships presented visually lack being programmatically determined.
Identify Input Purpose: The use of code does not indicate the purpose of common input. Contrast and Non-Text Contrast: the visual presentation of text and images of text has problems with the contrast ratio. The same applies to the visual presentation of non -text elements. Principle 2, “Operable”, ensures that all interface components and navigation are usable, regardless of how users interact with the content (see Table 3).
• •
Keyboard and no Keyboard Trap: all content functionalities can be operated via a keyboard interface without needing specific timing for individual keystrokes.
Focus Order and Visible: any user interface operable by the keyboard has a mode where the keyboard focus indicator is visible. 3.2
Pause, Stop, Hide: users have problems with content that moves, scrolls, or auto -updates, with limited options to pause, stop, or hide such content. 3.1.3.
Principle 3, “Understandable”, requires that users can comprehend both the content and how the service functions, while principle 4, “Robust”, emphasises that the content should be interpreted reliably by various user agents, including assistive technologies (see Table 4).
Multiple Ways: users can get to the same content in multiple ways.
Label in Name: for user interface components with labels that include text or images of text, the name includes the text that is visually presented.
On Focus and on Input:when a user interface component receives focus, it does not trigger a change in context or when changing the setting of any user interface component does not automatically cause a change in context.
Consistent Navigation and Identification, Labels or Instructions: consistently order navigation that repeatsacross pages, labels or instructions areprovided whenever user input is required. • • • • • • • • • 3.2
Error Suggestion: when an input error is detected correction suggestions are not provided to the user.
Parsing: in content using markup languages, elements do not have complete start and end tags, are not properly nested, or have duplicate attributes.
Introduction: participants were given a brief overview of the session, including its goals and the evaluation process, to ensure clarity and understanding. • • • • • • • • • • • • • •
Task execution: participants were assigned specific tasks to perform using the chatbots. Observing how they interact with the interface allows researchers to assess usability and ease of navigation.
Final questions: participants respond to a series of questions aimed at identifying visual accessibility barriers, gathering feedback on interface usability, and suggesting improvements.
By analysing both the participants' behaviour and their feedback, the study seeks to evaluate whether the chatbots align with accessibility standards and deliver an inclusive experience for users with different degrees of visual impairment. The participants are asked to complete the following tasks using their own mobile devices, either via a website: • • •
Attempt to register or update their information in the municipal census.
Ask about available parking spaces, specifically those located in the city centre.
These tasks reflect common, real-life interactions with municipal chatbots, helping to evaluate their usability and accessibility in practical, everyday situations. The evaluation of the five -city council chatbots highlights both positive features and significant accessibility issues affecting users with visual impairments.
Sufficient time for input: users are given adequate time to type and send messages, reducing stress and improving usability.
Accurate cursor placement: most chatbots correctly position the cursor in the input field, aiding smooth interaction.
Logical focus navigation: screen readers follow an orderly focus path, helping users track conversations more easily.
Strong text contrast: all but one chatbot offers good contrast between text and background, supporting users with low vision.
Correct language recognition: all chatbots detect and operate in Spanish, enabling clear communication for native speakers.
Improper labelling: buttons and links often lack descriptive labels, making them difficult for screen readers to interpret.
Incomplete link reading: screen readers frequently announce only the link title, missing important context.
Orientation inconsistency: some chatbots do not display content properly when switching between vertical and horizontal views.
Poor text size flexibility: most fail to support scalable text, limiting readability for users needing larger fonts.
Lack of meaningful error feedback: errors are indicated only visually (e.g., underlining) without explanatory text for screen readers.
Layout issues with resizing: chatbots do not adapt well to window resizing, causing confusion and navigation difficulties.
Colour-only information cues: relying solely on colour to convey information excludes users who cannot perceive colours.
Limited voice interaction: while some chatbots accept voice input, most don’t respond via voice, reducing accessibility.
Difficulty repeating or escalating: chatbots struggle with simple commands like repeating a message or connecting to a human agent.
Task execution barriers: tasks like census registration or parking inquiries are hindered by complex steps and external pages, making the process frustrating and difficult to complete.
The findings of this research reveal that, despite advances in technology and the growing use of digital tools in Public Administration, the city council websites and chatbots analysed fall short of meeting WCAG [20]. Despite some positive elements, the ov erall accessibility of these websites and chatbots is limited. This lack of compliance leads to poor user experiences and effectively excludes those with accessibility needs particularly those with visual needs from accessing essential public services.
The evaluation of websites and chatbots using a combination of AEMs within a user -centred
designed methodology including trained students who evaluate their local city council website and
end users with visual accessibility shows results aligned with the literature in the area [
The use of several AEMs showcased three key aspects of this research: (1) the short experience of trained students evaluating websites even when applying a service -learning approach [15], (2) the evaluation by end-users revealed unexpected usability challenges and participant frustration, emphasising that usability alone is not enough and accessibility must be prioritised [24], and (3) the fact that websites and chatbots have been designed and implemented by different providers and with different objectives, could be showing inconsistencies when integrated [20].
The study had several limitations. The sample of websites and chatbotsevaluated, and the number of both student-evaluating websites and end-userswith accessibility needs were small. Mostwebsites and their chatbots were from Comunidad de Madrid, and bothe heuristic evaluations and end-user tests were conducted in controlled settings, not reflecting real -world usage conditions. For future research, we recommend including a bigger sample, websites evaluated by more than one evaluator allowing interrater reliability of the results [21], including AAA criteria, and a broader group of users with accessibility needs as end-users, not only visual, expanding the geographic scope of chatbot analysis, and conducting evaluations in everyday contexts to better understand real-life accessibility challenges.
Also, while the AEMs included in this pilot research cover a well -established standard such as WCAG, it is necessary to include standards more focused on virtual assistants and chatbots, which not always are set in web environments, such as The Natural Language Interface Accessibility User Requirements (NAUR) [27] outline the essential accessibility considerations for designing natural language interfaces. These guidelines offer a broader perspective on accessibility in communication processes than WCAG [28].
This user-centred methodology not only provided a diagnosis of accessibility shortcomings but
also helped raise awareness about the importance of digital inclusion in public services. Accessibility
must be a core principle in technological developmen—tespecially in public services. Ensuring equal
access to city council websites and chatbots not only supports the rights of disabled users but also
builds public trust and promotes social inclusion [
We are grateful for the time and dedication of the participants in this study. This publication is part of the R&D&I project HumanAI-UI, Grant PID2023-148577OB-C22 (Human-Centered AI: UserDriven Adaptative Interfaces-HumanAI-UI - IA centrada en las personas: interfaces adaptativas dirigidas por el usuario) funded by MICIU/AEI/10.13039/501100011033 and by FEDER/UE. Francisco Iniesto’s work is supported by the scholarship César Nombela 2024 -T1/COM-31590 from the Community of Madrid – Comunidad de Madrid.
[15] F. Iniesto, C. Rodrigo, Exploring the accessibility evaluation of city council websites by computer science students using a service-learning approach, in: INTERACCION 2024: XXIV Congreso Internacional de Interacción Persona-Ordenador \ XXIV International Conference on Human Computer Interaction, ACM, New York, NY, USA, 2024. doi:10.1145/3657242.3658591. [16] M. E. Cortés-Cediel, A. Segura-Tinoco, I. Cantador, M. P. Rodríguez Bolívar, Trends and challenges of e-government chatbots: Advances in exploring open government data and citizen participation content, Gov. Inf. Q. 40.4 (2023) 101877. doi:10.1016/j.giq.2023.101877. [17] K. Lister, T.Coughlan, F.Iniesto, N. Freear, P. Devine, Accessible conversational user interfaces, in: W4A '20: 17th Web for All Conference, ACM, New York, NY, USA, 2020. doi:10.1145/3371300.3383343. [18] A. Pradhan, K. Mehta, L. Findlater, "Accessibility Came by Accident", in: CHI '18: CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 2018. doi:10.1145/3173574.3174033. [19] F. Iniesto, T. Coughlan, K. Lister, P. Devine, N. Freear, R. Greenwood, W. Holmes, I. Kenny, K. Mcleod, R. Tudor, Creating ‘A Simple Conversation’: Designing a Conversational User Interface to Improve the Experience of Accessing Support for Study, ACM Trans. Access.
Comput. (2022). doi:10.1145/3568166. [20] T. Coughlan, F. Iniesto, What should I know? Analysing behaviour and feedback from student use of a virtual assistant to share information about disabilities, High. Educ. (2025) 101002. doi:10.1016/j.iheduc.2025.101002. [21] F. Iniesto, C. Rodrigo, The use of WCAG and automatic tools by computer science students: a case study evaluating MOOC accessibility, JUCS - J. Univers. Comput. Sci. 30.1 (2024) 85–105. doi:10.3897/jucs.101704. [22] G. Brajnik, Validity and reliability of web accessibility guidelines, in: Proceeding of the eleventh international ACM SIGACCESS conference, ACM Press, New York, New York, USA, 2009. doi:10.1145/1639642.1639666. [23] P. Acosta-Vargas, S. Luján-Mora, T. Acosta, L. Salvador-Ullauri, Toward a Combined Method for Evaluation of Web Accessibility, in: Proceedings of the International Conference on Information Technology & Systems (ICITS 2018), Springer International Publisnhgi , Cham, 2018, pp. 602–613. doi:10.1007/978-3-319-73450-7_57. [24] M. Zapata, A. Díaz, Visual Accessibility in Chatbots of the Spanish Public Administration, in: Communications in Computer and Information Science, Springer Nature Switzerland, Cham, 2025, pp. 236–244. doi:10.1007/978-3-031-94156-6_25. [25] M. Vigo, M. Arrue, G. Brajnik, R. Lomuscio, J. Abascal, Quantitative metrics for measuring web accessibility, in: the 2007 international cross -disciplinary conference, ACM Press, New York, New York, USA, 2007. doi:10.1145/1243441.1243465. [26] S. McLeod, Observation methods, Simply Psychology. URL: https://www.simplypsychology.org/observation.html [27] Natural Language Interface Accessibility User Requirements. NAUR, Web Accessibility
Initiative (WAI). URL: https://www.w3.org/TR/naur/ [28] F. Iniesto, C. Rodrigo, The Application of Natural Language Interface Accessibility User Requirements by Computer Science Students: the Case Study of an IT Support Chatbot, in Proceedings of the TEEM’24: international conference technological ecosystems ofr enhancing multiculturalism, Springer, 2025 (In Press).