Conversational AI for Web Inclusivity: Technologies, Design Patterns and Development Toolkits

Abstract

Digital services can represent an important channel for granting access to knowledge, education, and work. Expecially to the citizens living with disabilities. However, as of now, the Web is conceived essentially for visual fruition and is inadequate for all those users living with permanent or situational impairments. Conversational AI is emerging as a technology apt for the development of inclusive and accessible applications, but there is still a lack of guidance specific to the design of inclusive Conversational AI systems. This research proposes to identify guidelines, interaction patterns, and enabling technology for a new paradigm for accessible conversational web browsing.

Keywords

• eol>Software accessibility
• card tools
• conversational interfaces
• HCI design and evaluation methods

1. Introduction

Digital inclusion is a primary right for all citizens. Digital services, in fact, can represent an important channel for granting access to knowledge, education, and work. Thus, the development of accessible digital services, will become essential to guarantee the inclusion, and right to access information to every member of society, most notably to citizens with disabilities. However, right now, the Web is conceived for visual fruition and is inadequate for those living with permanent or with situational impairments. Despite providing a level of support, assistive technologies, such as screen readers, cannot always grant proper assistance because of the still limited compliance of websites to accessibility guidelines.

Conversational AI (CAI) is emerging as a technology for inclusive interaction with digital services, as it can provide an interaction paradigm that is independent of the visual channel. Still, there is a lack of widely accepted guidelines, methodology, and development platforms to guide developers and designers in delivering efective conversational interfaces that are also fully accessible to disabled users. Recent work explores ways to leverage CAI to augment the web. ConWeb[ 1 ] is a platform ofering a browser extension that enables users to browse the web through voice interaction. The ConWeb browsing paradigm was developed with Blind and Visually Impaired (BVI) users to define interaction patterns specific to their needs.

In this context, my research will investigate how CAI can improve the accessibility of digital services. It will address this research question: which are the guidelines, interaction patterns, enabling technologies needed for a new paradigm for conversational web browsing? The resulting methodological and technological framework will support the design of websites that can be accessed through Natural Language interaction.

2. Related Works

Conversational user interfaces (CUI) are a broad class of interfaces that often refer to chatbots and voice user interfaces (VUI). The interaction is generally a two-way dialogue between humans and machines. Alexa, Siri, Microsoft Cortana, and Google Assistants are examples of voice-enabled intelligent assistants with which users can interact to perform daily tasks.

In the context of web navigation, non-visual navigation methods have already been explored in the HCI literature through diferent approaches, such as web augmentation or end-user programming[ 2, 3, 4 ]. Other approaches to make the Web more accessible to BVI users include segmentation: Borodin et al. [ 5 ] introduce a multi-modal browser that leverages segmentation techniques to propose to users only the relevant parts of a webpage. Cambre et al., instead, proposed Firefox Voice[ 6 ], a browser assistant that enabled users to perform tasks on the browser through voice interaction. Through voice commands, it could perform both tasks at the webpage level and that of the browser application. Ripa et al.[ 4 ] present an end-user development environment approach that leverages semantic annotation to divide the page into relevant content blocks and generate voice assistants. Others propose the idea of a Conversational Web. ConWeb[ 1 ] is a browser extension that acts as a middleware between the Web client and Web servers. The page is parsed and used to generate the dialogue necessary to answer users’ requests. It uses an NLP pipeline and a headless browser to extract user intents and entities and transform the requests into actions on the webpage respectively. However, this solution still has some limitations as the actions are limited only to reading text and link navigation. Indeed these methods, while they provide new technical solutions for non-visual navigation, do not provide practitioners with generalised development for a Web that is conversational by design.

In HCI research, guidelines help address critical factors for the development of new design solutions, such as accessibility or usability. With the increasing popularity of conversational interfaces (CUI), it also emerged the need for specific guidelines aimed at helping their design. Nielsen heuristics[ 7 ] have been applied to CUI[ 8, 9 ]. Still, both industry and academic HCI are researching ways to codify design knowledge regarding CUI, in the form of heuristics, best practices and so on. For example, Murad et al.[ 10 ], starting from an analysis of Nielsen, Norman[ 11 ] and Shneiderman[ 12 ] guidelines, adds a set of principles specifically for VUI to ensure transparency and taking context into account. Microsoft[ 13 ], IBM[ 14 ], and Google have also published guidelines for the design of voice assistants. However, concerns have been raised regarding their validity for BVI users[ 15 ].

BVI users have been found to perceive commercial voice assistants as verbose and not very helpful. To address these issues, Branham et al. [ 16 ] discusses design implications to make voice assistants more accessible. They advocate for more personalization, both in the sense of customizable voice commands and customizable interaction preferences, to modify the length of turns during a conversation or speech speed. Corbett et al. [ 17 ], instead, provides a case study to highlight navigation challenges and define two design principles to increase discoverability and learnability by contextualised help and training. Lastly, Pucci et al. [ 18 ] propose a set of interaction patterns specifically researched for BVI users when interacting with Web conversational agents. The proposed paradigm presents the webpage to users in a tree-like model that goes beyond the sequential reading provided by screen readers, also ofering summarization and skimming patterns.

Despite all these eforts, guidelines for CUI that consider accessibility principles are still sparse and may consider only some interaction aspects. Furthermore, practitioners share diferent views regarding the need for generally shared guidelines for CUI, as it is a field that is still evolving and may be subject to changes, still[ 19 ]. Thus, consensus and general adoption of such guidelines are yet to be achieved.

3. Contribution and Future Work

In the first months of my Ph.D., I started analyzing card toolkits and guidelines available in the academic context and the industry. Card-based design tools indeed support the translation of theoretical research findings into design practices. They can be used to communicate methods, frameworks, or theories. The analysis aimed at surveying the toolkits for CUI design, to understand what are the reference principles currently followed by designers and developers. Furthermore, by this analysis, I aimed to understand which toolkits currently guide the design of inclusive conversational AI and how. To retrieve the toolkits relevant to inclusive design and CAI, a Google search and a query on Google Scholar were performed with the keywords: “inclusive cards”, “conversational AI cards”, and “responsible AI cards”.

Through the search queries, ten decks of cards were identified. Cards not containing guidelines relevant to CAI were excluded, leaving seven toolkits. The retrieved cards were mapped across their domain, content, and ideal moment of use, as shown in Table 1. By “domain” it is intended the field they were designed for, whether Inclusivity or AI. “Content” indicates the type of information codified in the cards. Examples are problem statements to elicit ideas, methods to suggest possible approaches, technologies to be used, or insights providing inspiration on a particular domain or problem. Lastly the “ideal moment of use” indicates in which phase of the design process it is best to employ the toolkit. The phases considered were: research, ideation, prototyping, development, and evaluation.

From the analysed sample, only three toolkits propose insights specific to AI: the AI Ideation Cards[ 24 ], Responsible Bots[ 25 ], and the Human-AI Interaction guidelines[26]. The AI Ideation Cards[ 24 ] provide scenarios and possible methods useful when designing digital products that leverage AI. They cover a broad range of tasks that can be supported by AI, and also, more specifically, by CAI, but they do not prioritize inclusivity in their definitions. This deck provides diferent application scenarios for CAI and lists some relevant issues to consider when designing an application based on CAI, such as privacy and inclusivity. However, since it supports only the ideation phase, it lacks concrete guidelines for the development of CAI applications.

Responsible Bots[ 25 ], proposed by Microsoft, instead, addresses the prototyping and development phases. It presents a set of guidelines specific to the development of conversational agents that are transparent and reliable. It also addresses inclusive design practices by referring to the Inclusive Activities Cards[ 23 ], also by Microsoft.

Lastly, the Human-AI Interaction guidelines[26] provide general principles for responsible AI-based interactive systems. The guidelines were defined through a four-step process that used heuristic evaluations to validate the guidelines. The Human-AI Interaction guidelines cover all moments of interaction between humans and machines. Since they are conceived as general, while the guidelines are also valid for CAI applications, they do not provide principles specific to conversational interaction.

The toolkits related to inclusivity[ 22, 21, 20, 23 ], instead, provide insights mostly useful for the ideation stage. The provided design knowledge is meant to be a starting point to reflect on problems faced by users with disabilities. Only Microsoft Inclusivity cards[ 23 ] cover also other design phases, spanning from research, to ideation, to prototiping. This toolkit provides principles for usability, error prevention, feedback on system status, and so on.

From the toolkits’ analysis, it emerged that a methodological framework for the development of inclusive Conversational AI applications is still lacking. In the AI domain, only the Responsible bot toolkit addresses the topic of inclusivity, but just by referring to an external set of guidelines. In the inclusive design domain, instead, with the exception of the Microsoft Inclusivity cards, the toolkits seem to focus more on providing inspirational content to start the ideation phase, than practical design guidelines.

Secondly, if we specifically consider guidelines for CAI, the conducted analysis highlights that the available CAI guidelines focus on how to present the conversational agent to users, for example giving guidelines regarding the tone of voice, how to be transparent about what the bot can do, or recover from a failed interaction. However, how to structure complex interactions with web pages is still not fully addressed in the literature. Thus, starting from the patterns identified by Pucci et. al [ 18 ], my research will aim at expanding the current methodological framework for CAI to define a new methodological and technological framework to drive the development of CAI that are inclusive by design and can serve the fruition of web content through voice interaction.

To achieve these objectives, my research will also focus on expanding user-centred methods, such as the User Centred Design (UCD) sprint [27], to include in ideation and testing phases principles specific to inclusive design and CAI. The UCD sprint ofers a methodology to develop new technology with a human-centred approach. The methodology is structured in three phases, Discovery, Design and Reality check, that are meant to guide the participants in understanding users and their needs. Currently, it does not include considerations specific for the design of AI technologies, nor for technologies catered to marginalised groups.

The methodological framework will be tested and validated in partnership with the Municipality of Milan, to assess the validity of current CAI design guidelines and toolkits discussed in this work and to identify extensions in the directions discussed in this paper. Using the Municipality’s web resources as a case study, I will investigate the limits of current AI and inclusivity guidelines, and how to expand them to support the development of accessible conversational services. The aim will be to define guidelines for a methodological framework guiding the design of CAI solutions addressing inclusivity.

Lastly, as a relevant aspect, this research will also explore technical solutions, from EndUser Development paradigms and platforms to the automatic generation of code, to support developers in implementing websites that “by design” can also be equipped with a conversational channel. Specifically, given the focus of my research on inclusive Web resources for the Public Administration, a possible solution that will be evaluated will consist in the expansion of the Web development kit proposed by the Designers Italia initiative[28]. The kit provides developers with design guidelines and ready-to-use Web components to develop consistent Web applications for the Public Administration. Future work will explore the feasibility of expanding this kit through semantic tagging and related Web browser extensions for generating a dialogue system for conversational Web browsing. [26] S. Amershi, D. Weld, M. Vorvoreanu, A. Fourney, B. Nushi, P. Collisson, J. Suh, S. Iqbal, P. N.

Bennett, K. Inkpen, J. Teevan, R. Kikin-Gil, E. Horvitz, Guidelines for human-ai interaction, CHI ’19, ACM, New York, NY, USA, 2019, p. 1–13. doi:10.1145/3290605.3300233. [27] M. Larusdottir, V. Roto, Å. Cajander, Introduction to user-centred design sprint, in: C. Ardito, R. Lanzilotti, A. Malizia, H. Petrie, A. Piccinno, G. Desolda, K. Inkpen (Eds.), Human-Computer Interaction – INTERACT 2021, Springer, Cham, 2021, pp. 253–256. [28] Designers Italia, Designers italia, 2023. URL: https://designers.italia.it/.

References

[1] M. Baez , F. Daniel , F. Casati , Conversational web interaction: Proposal of a dialog-based natural language interaction paradigm for the web , in: Chatbot Research and Design: Third International Workshop, CONVERSATIONS 2019, Amsterdam, The Netherlands, November 19-20 , 2019 , Revised Selected Papers, Springer-Verlag, Berlin, Heidelberg, 2019 , p. 94 - 110 . doi: 10 .1007/978- 3- 030 - 39540- 7 _ 7 .

[2] J. P. Bigham , T. Lau , J. Nichols , Trailblazer: Enabling blind users to blaze trails through the web , in: Proceedings of the 14th International Conference on Intelligent User Interfaces , IUI '09 , ACM , New York, NY, USA, 2009 , p. 177 - 186 . doi: 10 .1145/1502650.1502677.

[3] I. Ramakrishnan , V. Ashok , S. M. Billah , Non-visual web browsing: Beyond web accessibility , volume 10278 , 2017 , pp. 322 - 334 . doi: 10 .1007/978- 3- 319 - 58703- 5_ 24 .

[4] G. Ripa , M. Torre , S. Firmenich , G. Rossi, End-User Development of Voice User Interfaces Based on Web Content , 2019 , pp. 34 - 50 . doi: 10 .1007/978- 3- 030 - 24781- 2 _ 3 .

[5] Y. Borodin , F. Ahmed , M. A. Islam , Y. Puzis , V. Melnyk , S. Feng , I. V. Ramakrishnan , G. Dausch, Hearsay: A new generation context-driven multi-modal assistive web browser , in: Proceedings of the 19th International Conference on World Wide Web, WWW '10 , ACM , New York, NY, USA, 2010 , p. 1233 - 1236 . doi: 10 .1145/1772690.1772890.

[6] J. Cambre , A. C. Williams , A. Razi , I. Bicking , A. Wallin , J. Tsai , C. Kulkarni , J. Kaye , Firefox voice: An open and extensible voice assistant built upon the web , CHI '21 , ACM , New York, NY, USA, 2021 . doi: 10 .1145/3411764.3445409.

[7] J. Nielsen , R. Molich , Heuristic evaluation of user interfaces , in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '90 , ACM , New York, NY, USA, 1990 , p. 249 - 256 . doi: 10 .1145/97243.97281.

[8] M. Maguire , Development of a heuristic evaluation tool for voice user interfaces , in: A . Marcus , W. Wang (Eds.), Design, User Experience , and Usability. Practice and Case Studies , Springer International Publishing, Cham, 2019 , pp. 212 - 225 .

[9] C. Murad , C. Munteanu , B. R. Cowan , L. Clark , Revolution or evolution? speech interaction and hci design guidelines , IEEE Pervasive Computing 18 ( 2019 ) 33 - 45 . doi: 10 .1109/MPRV. 2019 . 2906991 .

[10] C. Murad , C. Munteanu , L. Clark , B. R. Cowan , Design guidelines for hands-free speech interaction , MobileHCI '18 , ACM , New York, NY, USA, 2018 , p. 269 - 276 . doi: 10 .1145/ 3236112.3236149.

[11] D. Norman , The Design of Everyday Things, The MIT Press, MIT Press, 2013 . URL: https://books.google.it/books?id=heCtnQEACAAJ.

[12] B. Shneiderman , Designing the User Interface: Strategies for Efective Human-Computer Interaction , 3rd ed., Addison-Wesley Longman Publishing Co., Inc., USA, 1997 .

[13] S. Amershi , D. Weld , M. Vorvoreanu , A. Fourney , B. Nushi , P. Collisson , J. Suh , S. Iqbal , P. N. Bennett , K. Inkpen , J. Teevan , R. Kikin-Gil , E. Horvitz, Guidelines for human-ai interaction , CHI '19 , ACM , New York, NY, USA, 2019 , p. 1 - 13 . doi: 10 .1145/3290605.3300233.

[14] R. J. Moore , S. An , G.-J. Ren , The ibm natural conversation framework: a new paradigm for conversational ux design, Human-Computer Interaction 38 ( 2023 ) 168 - 193 . doi: 10 . 1080/07370024. 2022 . 2081571 .

[15] A. Pradhan , K. Mehta , L. Findlater, ” accessibility came by accident”: Use of voice-controlled intelligent personal assistants by people with disabilities , CHI '18 , ACM , New York, NY, USA, 2018 , p. 1 - 13 . doi: 10 .1145/3173574.3174033.

[16] S. M. Branham , A. R. Mukkath Roy , Reading between the guidelines: How commercial voice assistant guidelines hinder accessibility for blind users , in: Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility , ASSETS '19 , ACM , New York, NY, USA, 2019 , p. 446 - 458 . doi: 10 .1145/3308561.3353797.

[17] E. Corbett , A. Weber , What can i say? addressing user experience challenges of a mobile voice user interface for accessibility , MobileHCI '16 , ACM , New York, NY, USA, 2016 , p. 72 - 82 . doi: 10 .1145/2935334.2935386.

[18] E. Pucci , I. Possaghi, C. M. Cutrupi , M. Baez , C. Cappiello , M. Matera , Defining patterns for a conversational web , CHI '23 , ACM , New York, NY, USA, 2023 . doi: 10 .1145/3544548. 3581145.

[19] K. H. Khemani , S. Reeves , Unpacking practitioners' attitudes towards codifications of design knowledge for voice user interfaces , in: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI '22 , ACM , New York, NY, USA, 2022 . doi: 10 .1145/3491102.3517623.

[20] Frog , Cards for humanity, 2023 . URL: https://cardsforhumanity.frog.co/.

[21] Ontario Government , Inclusive design cards, 2023 . URL: http://www.ontario.ca/page/ inclusive-design-cards.

[22] Google , Inclusive design works, 2023 . URL: https://inclusivedesignworks.app/.

[23] Microsoft , Microsoft inclusive design, 2023 . URL: https://inclusive.microsoft.design/.

[24] AIxDesign , Aixdesign, 2023 . URL: https://aixdesign.co/shop.

[25] P. Bailey , R. W. White , H. Liu , A. Kumaran , Responsible bots: 10 guidelines for developers of conversational ai , ACM Transactions on The Web ( 2019 ).