Benchmarking between the DQL Index and the Web Application Accessibility Index using Automatic Test Tools Tetyana Zhyrova1, Nataliia Kotenko1, Bohdan Bebeshko1, Karyna Khorolska1, and Svitlana Shevchenko2 1 State University of Trade and Economics, 19 Kioto str., Kyiv, 02000, Ukraine 2 Borys Grinchenko Kyiv University, 18/2 Bulvarno-Kudriavska str., Kyiv, 04053, Ukraine Abstract Every day, humanity is witnessing digital transformation, which involves implementing digital technologies to transform business processes and services from non-digital to digital. The success of this transformation is measured by the Digital Quality of Life (DQL) Index, and VPN service provider Surfshark publishes the study results. As Index DQL grows, so does the number of users because digital accessibility is becoming more popular daily. The increase in digital information, the continuous development of IT, and the transition of life online require software to be accessible to people with disabilities. There is a question about the correlation between software availability level and Index DQL. This research is dedicated to the solution to this issue. Ten European countries were randomly selected, and automated testing of the accessibility of government websites was carried out using the following tools: Tenon, Wave, and SiteImprove. The term Web application accessibility index was defined. For the first time, an analysis of the correspondence between Index DQL indicators and the web application accessibility index was carried out. Even though the EU countries and their partner states are constantly working to improve conditions for people with disabilities, a high DQL Index does not indicate a high level of accessibility of state websites, which was confirmed by the study. Keywords 1 Digital quality of life, accessibility testing, automatic test tools. 1. Introduction processes and services from non-digital to digital. Digital services are benefiting in almost every field of activity. The availability of the Internet Every day, humanity is witnessing digital and digital spaces has led to the emergence of e- transformation, which involves implementing commerce services and various options for digital technologies to transform business working on the Internet. This includes, among processes and services from non-digital to digital. other things, moving data to the cloud, using This includes, among other things, moving data to technological devices and tools for the cloud, using technology devices and tools for communication and collaboration, and communication and collaboration, and automating processes [5]. High digital quality of automating processes. Digital transformation life facilitates and accelerates the resolution of makes life easier and helps quickly solve many several issues by citizens of one or another problems. It is intended to improve the quality of country. The Digital Quality of Life (DQL) Index digital life that spending on digital transformation is calculated by looking at the impact of five core technologies and services is constantly increasing pillars: worldwide, as evidenced by the Statista website 1. Internet Affordability: This pillar determines (see Fig. 1) [[1]–4]. how long an individual has to work to afford Digital transformation means implementing the internet, specifically the cheapest mobile digital technologies to transform business and broadband internet. CPITS-2022: Cybersecurity Providing in Information and Telecommunication Systems, October 13, 2022, Kyiv, Ukraine EMAIL: zhyrova@knute.edu.ua (T. Zhyrova); kotenkono@knute.edu.ua (N. Kotenko); b.bebeshko@knute.edu.ua (B. Bebeshko); k.khorolska@knute.edu.ua (K. Khorolska); s.shevchenko@kubg.edu.ua (S. Shevchenko) ORCID: 0000-0001-8321-6939 (T. Zhyrova); 0000-0002-2675-6514 (N. Kotenko); 0000-0001-6599-0808 (B. Bebeshko); 0000-0003-3270- 4494 (K. Khorolska); 0000-0002-9736-8623 (S. Shevchenko) ©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 110 2. Internal Quality: this pillar determines the According to a study published by the VPN stability and speed of the internet for both service provider Surfshark, there is a mobile data and broadband. It also considers corresponding distribution of DQL among the the growth of speed and strength. countries of the world and Europe (see Fig. 2). 3. Electronic Infrastructure: this pillar measures The Digital Quality of Life 2021 index how developed a country’s e-infrastructure is. analyses 110 countries worldwide in terms of five Better infrastructure results in smooth online core pillars: internet affordability, internet quality, experiences such as banking, shopping, e-infrastructure, e-security, and e-government. entertainment, education, etc. These pillars consist of 14 indicators that are 4. Electronic Security is a measure of a nation’s interrelated and work together to measure the cyber security and includes various strategies overall digital quality of life [5]. Countries were the country has developed to decrease cyber- evaluated based on index scores with the best crimes and protect the privacy of its people. possible value equal to one. Denmark is the best 5. Electronic Government is the pillar that at 0.83, followed by South Korea (0.76), Finland reviews the advancements of the government (0.76), Israel (0.74), the United States of America in the digital sector. This results in lesser (0.74), and Singapore (0.72). Ukraine ranks only corruption, better transparency for the public, 47th in this ranking. and efficient public services [5]. 3 2.8 Spending in trillion U.S. dollars 2.5 2.4 2.1 2 1.8 1.5 1.5 1.31 1.18 0.96 1 1 0.5 0 2017 2018 2019 2020 2021* 2022* 2023* 2024* 2025* Figure 1: Spending on digital transformation technologies and services worldwide from 2017 to 2025 in trillion U.S. dollars Figure 2: Digital quality of life index in Europe for 2021 111 With the growth of DQL, the number of users 1. There are no clear and unambiguous of government Internet applications is growing, definitions of such concepts as “disability,” among which a particular share is occupied by “accessibility,” “person with disabilities,” or people with special needs, whose number is “person with special needs,” which has some constantly increasing. Such applications have ambiguity in research and needs clarification several requirements. Note that to solve these and agreement at the global level. issues, on October 26, 2016, the EU Parliament 2. There are no clear definitions and concepts of and the Council of the European Union passed the such terms as “accessibility web design,” EU Web Accessibility Directive, and on June 7th, “inclusive software,” or “universal web 2019, the European Union formally adopted the design.” European Accessibility Act. The Directive aims to 3. To ensure a high level of content accessibility create a more standardized and harmonized for people with disabilities, WAI (Web framework around the accessibility of websites Accessibility Initiative), within the framework and mobile applications of public sector bodies of the World Wide Web Consortium (W3C), [6]. The Directive complements the European developed the Web Content Accessibility Accessibility Act, which covers a wide range of Guidelines (WCAG) standard. Version products and services also in the private sector. WCAG 2.1 was adopted as the standard of the Further European legislation supports people with European Union EN 301 549 [21]. The disabilities in other areas, including electronic recommendations for ensuring the communications, audio-visual media services, accessibility of web content are the most eBooks, eCommerce, and ICT equipment [7]. universal, and many countries have adapted Following the specified directive, the following these requirements to their laws. terms were established: on September 23, 2019, 4. There are many programs for automated web all new public sector websites and apps were application accessibility testing. In the vast required to conform to the directive; by majority, they do not check all WCAG 2.1 September 23, 2020, all new and existing public requirements, so there is a need for several websites must conform to the directive; by June automatic testing tools. 23, 2021, all new and existing mobile apps must 5. No scientific publications highlight the issue conform to the directive [6]. Thus, the question of the DQL Index, its factors, the relationship arises about the relationship between the DQL between the DQL Index, the availability of index and the availability of web applications of web applications, etc. state importance. Therefore, the study of scientific and practical literature, publications, and publications on the 2. Related Works Internet proved that the issue of the relationship between the level of accessibility of software and the Digital Quality of Life index needs thorough The issue of software availability is quite research and analysis, and there is also a need to broad and multifaceted. It has worried several improve accessibility testing standards. Thus, this scientists but is still not revealed and not article aims to analyze accessibility standards, completed. Scientists: M. Goldberg [8], M. research existing automated web application Campoverde-Molina 9, K. Ordonez 10, G. E. accessibility testing tools, and correlate test Constain 11, J.S. Silva 12, K. Brown [13]3, results with the Digital Quality of Life index. M. Sashnova [[14]], etc. Nielsen Norman Group [[15]], which is a world leader in user experience research, needs special attention. For the past few 3. Index Correspondence Research years, their research has focused on website accessibility. By Directive (EU) 2016/2102 Of The Several scientists are working on the issue of European Parliament And Of The Council of 26 inclusive software: A. Savidis [[16]], J. Ohene- October 2016 on the accessibility of the websites Dzhan [[17]], I. Niculescu [[18]], K.M. Martinez and mobile applications of public sector bodies [[19]], V. Boronos [20] and others. A small [22], it is the applications of public sector bodies number of publications are devoted to studying that require accessibility. the digital quality of life. The standard contains a number of The analysis of publications in this direction recommendations on what needs to be done to showed that: make the web-content accessible to people with 112 disabilities. Recommendations are grouped Ireland, Luxembourg, Poland, Italy, Greece, according to four principles: perception, Ukraine, Albania, Montenegro [27–36]. manageability, comprehensibility and reliability. Table 1 contains the results of testing the These recommendations are called success accessibility of Government website home pages criteria, and according to them, the application with the appropriate automated testing tools. can receive one of three levels: A, AA or AAA. Testing of Siteimprove Browser Extensions Accessibility testing at the appropriate level is was carried out according to the following performed by automated systems and testers who criteria: check whether the content of the site meets the  Error. Occurrences of an issue that have been relevant criteria and assess the usability of the automatically determined to be in platform [[23]3]. contravention of the Web Content The Accessibility Conformance Testing Accessibility Guidelines (WCAG) 2. (ACT) Rules Format 1.0 defines a format for  Warning. Occurrences of an issue that have writing accessibility test rules. These test rules can been automatically determined to be in be used for developing automated testing tools contravention of best practices under the Web and manual testing methodologies. It provides a Content Accessibility Guidelines (WCAG) 2. common format that enables any party involved in  Review. Issues that cannot be checked accessibility testing to document and share their automatically but requires a manual inspection testing procedures in a robust and understandable to determine if each item lives up to the manner. This enables transparency and success criteria. harmonization of testing methods, including Testing of WAVE Browser Extensions was methods implemented by accessibility test tools carried out according to the following criteria: [[24]].  Error. Occurrences of an issue which have There is a range of tools for automated testing, been automatically determined to be in including Google ADT, Tenon, aXe, Wave, contravention of the Web Content SiteImprove, and others. We have chosen the Accessibility Guidelines (WCAG) 2. following tools for automated testing of the  Contrast Errors. Text is present that has a availability of government websites: Tenon, contrast ratio less than 4.5:1, or large text Wave, and SiteImprove. This choice is due to the (larger than 18 point or 14 point bold) has a GDS accessibility team’s audit of the most used contrast ratio less than 3:1. WCAG requires accessibility tools [[25]]. that page elements have both foreground AND Technology is constantly changing, so the web background colors defined (or inherited) that application's user experience will vary depending provide sufficient contrast. When text is on the technical capabilities of browsers and presented over a background image, the text devices. Not all browsers display web pages the must have a background color defined same way. There are often noticeable differences (typically in CSS) that provides adequate text in how browsers handle technologies like CSS, contrast when the background image is HTML, and JavaScript. Because, according to disabled or unavailable. WAVE does not statistics provided on the Statcounter GlobalStats identify contrast issues in text with CSS website, the browser market share in Europe was transparency, gradients, or filters. distributed as follows: Chrome 59.29%, Safari  Alerts. Occurrences of an issue that have been 19.97%, Edge 5.86%, Firefox 5.83%, Samsung automatically determined to be in Internet 3.62%, Opera 3.02% [[26]]. That is why, contravention of best practices under the Web within the framework of this study, testing the Content Accessibility Guidelines (WCAG) 2. availability of web applications of state  Structural Elements. The number of structural importance was carried out in Google Chrome. elements on the page, such as headings of We have chosen 10 European countries different levels, ordered lists and unordered randomly to ensure the sample’s lists, navigation, etcetera. representativeness: Denmark, Switzerland, Tenon testing was carried out according to the United Kingdom of Great Britain and Northern following criteria: Total Issues, Error Density. 113 Table 1 Test results according to the relevant criteria Siteimprove 2021 Digital Quality of Browser WAVE Browser Extensions Tenon Extensions Life Index Error Density Total Issues № Country Structural Elements Contrast Warning Review Errors Errors Errors Alerts 1 Denmark 1 14 3 20 11 2 7 56 2 7 2 Switzerland 8 16 8 23 19 16 114 139 10 21 3 Great Britain 10 5 4 23 0 1 2 40 2 3 4 Luxembourg 15 8 4 21 20 21 5 45 22 8 5 Poland 25 9 4 21 15 11 9 34 9 11 6 Italy 27 12 2 16 7 0 8 37 18 8 7 Greece 38 7 2 20 0 0 53 5 21 6 8 Ukraine 47 27 4 20 11 9 22 128 14 19 9 Albania 64 22 6 21 16 40 31 66 37 5 10 Montenegro 78 11 4 20 0 0 43 54 2 11 Fig. 3 shows an infographic of the research that have been checked, k is the number of errors results compared to the DOL Index. for each criterion. To determine the accessibility index of the Fig. 4 shows an infographic that demonstrates website, we use the formula: the correspondence between the DQL Index and 𝑛∙𝐸 the accessibility index of the home pages of the 𝐼 = ∑𝑛 ∙ 100%, (1) Government website. 𝑖=1 𝑘𝑖 According to the results of this study, it can be Where n is the number of evaluation criteria of concluded that there is no correspondence the web application, E is the number of elements between these indices. 90 78 80 70 64 Siteimprove 60 Browser Extensions 50 47 2021 Digital 38 37 Quality of Life 40 Index 30 25 27 27 19 2022 18 21 22 WAVE Browser 20 14 16 15 15 14 16 Extensions 11 12 11 11 8 10 10 8 9 9 7 7 10 5 1 2 02 0 02 Total Issues 0 Figure 2. Correspondence diagram of the number of errors according to the relevant criteria and the DQL Index 114 Montenegro 0.06 0.4668 Albania 0.04 0.5015 Ukraine 0.12 0.564 Greece 0.02 0.5879 Italy 0.06 0.6352 Poland 0.06 0.636 Luxembourg 0.05 0.6867 Great Britain 0.16 0.7065 Switzerland 0.11 0.7093 Denmark 0.07 0.8313 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Index Web Accessibility 2021 Digital Quality of Life Index Figure 4. The correspondence between the DQL Index and the accessibility index of the home pages of the Government website. and Technology, 2019, pp. 39–44. doi: 4. Conclusion 10.1109/picst47496.2019.9061376. [3] I. Bogachuk, V. Sokolov, V. Buriachok, Monitoring Subsystem for Wireless Systems The DQL index is calculated by analyzing the based on Miniature Spectrum Analyzers, in impact of five main components: Internet V International Scientific and Practical accessibility, Internet quality, e-infrastructure, e- Conference Problems of Infocom- security, and e-government. It does not consider munications. Science and Technology, 2018, the accessibility requirements of digital products, pp. 581–585. doi: 10.1109/infocommst. in particular web pages. It should be noted that EU 2018.8632151. countries and their partner states are constantly [4] Y. Sadykov, V. Sokolov, P. Skladannyi, working to improve conditions for people with Technology of Location Hiding by Spoofing disabilities, but the DQL Index does not indicate the Mobile Operator IP Address, in IEEE a high level of accessibility of state websites. This International Conference on Information and is evidenced by the results of the conducted Telecommunication Technologies and Radio research. For the first time, an analysis of the Electronics, 2021, pp. 22–25. doi: correspondence between Index DQL indicators 10.1109/UkrMiCo52950.2021.9716700. and the web application accessibility index was [5] 2021 Digital Quality of Life Index, carried out. Even though the EU countries and https://surfshark.com/dql2021. their partner states are constantly working to [6] EU Web Accessibility Compliance and improve conditions for people with disabilities, a Legislation https://www.deque.com/blog/eu- high DQL Index does not indicate a high level of web-accessibility-compliance-and- accessibility of state websites, which was legislation/. confirmed by the study. [7] Web Accessibility https://digital- strategy.ec.europa.eu/en/policies/web- 5. References accessibility. [8] M. Goldberg, et al., Are Accessible Software [1] Spending on Digital Transformation Accountable: A Commentary, Assistive Technologies and Services Worldwide from Technology, 34:1, 2022, pp. 61–63, doi: 2017 to 2025, https://www.statista.com/ 10.1080/10400435.2021.2024627. statistics/870924/worldwide-digital- [9] M. Campoverde-Molina, S. Luján-Mora, transformation-market-size/. L. Valverde, Process Model for Continuous [2] M. Vladymyrenko, et al., Analysis of Testing of Web Accessibility, in IEEE Implementation Results of the Distributed Access, vol. 9, 2021, pp. 139576–139593, Access Control System, in VI In-ternational doi: 10.1109/ACCESS.2021.3116100. Scientific and Practical Conference [10] K. Ordonez, J. Hilera, S. Cueva, Model- Problems of Infocommunications. Science Driven Development of Accessible 115 Software: A Systematic Literature Review. Economic Annals-XXI, 172(7–8), 2018, Univ Access Inf Soc 21, 2022, pp. 295–324. pp. 38-43. doi: 10.21003/ea.V172-07. doi: 10.1007/s10209-020-00751-6. [21] Countries that Have Adopted WCAG [11] G. E. Constain, et al., Framework for the Standards, https://www.3playmedia.com/ Design of Accessible Software to Support blog/countries-that-have-adopted-wcag- Users With Autism: A Proposal Oriented standards-map/. from the HCI, 2022 17th Iberian Conference [22] Directive (EU) 2016/2102 Of The European on Information Systems and Technologies Parliament And Of The Council of 26 (CISTI), 2022, pp. 1–5. doi: 10.23919/ October 2016, https://eur-lex.europa.eu/ CISTI54924.2022.9820360. legal-content/en/txt/html/?uri=celex: [12] J. S. Silva, R. Gonçalves, F. Branco, 32016L2102&from=EN. Accessible Software Development: A [23] T. Zhyrova, et al., Testing the Accessibility Conceptual Model Proposal, Universal of Web-Applications, Computer Systems Access in the Information Society 18, 2019, and Information Technologies, no. 3, 2021, pp. 703–716. doi: 10.1007/s10209-019- pp. 89–95. doi: 10.31891/CSIT. 00688-5 [24] Web Content Accessibility Guidelines, [13] K. Brown, et al., Students with Disabilities as https://www.w3.org/. Partners: A Case Study on User Testing an [25] Accessibility Tool Audit, https://alphagov. Accessibility Website, International Journal github.io/accessibility-tool-audit/. for Students as Partners, 4(2), 2020, pp. 97– [26] Browser Market Share in Europe, August 109. doi: 10.15173/ijsap.v4i2.4051. 2022, https://gs.statcounter.com/browser- [14] M. Sashnova, et al., Detection of market-share/all/europe. Accessibility and Quality of Websites of the [27] Denmark Government Official Website, Leading Universities of the World, Journal of https://denmark.dk/. Theoretical and Applied Information [28] Switzerland Government Official Website, Technology, 99(12), 2021, pp. 2845–2857. https://www.admin.ch/gov/en/start.html. [15] Nielsen Norman Group, [29] The United Kingdom of Great Britain and https://www.nngroup.com/. Northern Ireland Government Official [16] A. Savidis, C. Stephanidis, Inclusive Website, https://www.gov.uk/. Development: Software Engineering [30] Luxembourg Government Official Website, Requirements for Universally Accessible https://gouvernement.lu/en.html. Interactions, in Interacting with Computers, [31] Poland Government Official Website, vol. 18, no. 1, 2006, pp. 71–116. doi: https://www.gov.pl/. 10.1016/j.intcom.2005.06.005. [32] Italy Government Official Website [17] J. Ohene-Djan, R. Shipsey, Principles for https://www.governo.it/en. Inclusive Software Design of Learning [33] Greece Government Official Website Technologies, in 2018 Eighth IEEE https://www.gov.gr/en/sdg/. International Conference on Advanced [34] Ukraine Government Official Website Learning Technologies, 2008, pp. 989–990, https://www.rada.gov.ua/. doi: 10.1109/ICALT.2008.254. [35] Albania Government Official Website [18] I. Niculescu, et al., Towards Inclusive https://www.kryeministria.al/en/. Software Engineering Through A/B Testing: [36] Montenegro Government Official Website, A Case-Study at Windows, in 2021 https://www.gov.me/en/. IEEE/ACM 43` International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2021, pp. 180–187. doi: 10.1109/icse-seip52600.2021.00027. [19] C. M. Martínez, et al., Responsive Inclusive Design (RiD): A New Model for Inclusive Software Development, Univ Access Inf. Soc., 2022. doi: 10.1007/s10209-022-00893- 9. [20] V. Boronos, et al., Digital transformation of Ukraine: challenges of theory and practice in implementation of digital quality of life. 116