=Paper=
{{Paper
|id=Vol-3288/short8
|storemode=property
|title=Benchmarking between the DQL Index and the Web Application Accessibility Index using Automatic Test Tools (short paper)
|pdfUrl=https://ceur-ws.org/Vol-3288/short8.pdf
|volume=Vol-3288
|authors=Tetyana Zhyrova,Nataliia Kotenko,Bohdan Bebeshko,Karyna Khorolska,Svitlana Shevchenko
|dblpUrl=https://dblp.org/rec/conf/cpits/ZhyrovaKBKS22
}}
==Benchmarking between the DQL Index and the Web Application Accessibility Index using Automatic Test Tools (short paper)==
https://ceur-ws.org/Vol-3288/short8.pdf
Benchmarking between the DQL Index and the Web Application
Accessibility Index using Automatic Test Tools
Tetyana Zhyrova1, Nataliia Kotenko1, Bohdan Bebeshko1, Karyna Khorolska1,
and Svitlana Shevchenko2
1
State University of Trade and Economics, 19 Kioto str., Kyiv, 02000, Ukraine
2
Borys Grinchenko Kyiv University, 18/2 Bulvarno-Kudriavska str., Kyiv, 04053, Ukraine
Abstract
Every day, humanity is witnessing digital transformation, which involves implementing digital
technologies to transform business processes and services from non-digital to digital. The
success of this transformation is measured by the Digital Quality of Life (DQL) Index, and
VPN service provider Surfshark publishes the study results. As Index DQL grows, so does the
number of users because digital accessibility is becoming more popular daily. The increase in
digital information, the continuous development of IT, and the transition of life online require
software to be accessible to people with disabilities. There is a question about the correlation
between software availability level and Index DQL. This research is dedicated to the solution
to this issue. Ten European countries were randomly selected, and automated testing of the
accessibility of government websites was carried out using the following tools: Tenon, Wave,
and SiteImprove. The term Web application accessibility index was defined. For the first time,
an analysis of the correspondence between Index DQL indicators and the web application
accessibility index was carried out. Even though the EU countries and their partner states are
constantly working to improve conditions for people with disabilities, a high DQL Index does
not indicate a high level of accessibility of state websites, which was confirmed by the study.
Keywords 1
Digital quality of life, accessibility testing, automatic test tools.
1. Introduction processes and services from non-digital to digital.
Digital services are benefiting in almost every
field of activity. The availability of the Internet
Every day, humanity is witnessing digital
and digital spaces has led to the emergence of e-
transformation, which involves implementing
commerce services and various options for
digital technologies to transform business
working on the Internet. This includes, among
processes and services from non-digital to digital.
other things, moving data to the cloud, using
This includes, among other things, moving data to
technological devices and tools for
the cloud, using technology devices and tools for
communication and collaboration, and
communication and collaboration, and
automating processes [5]. High digital quality of
automating processes. Digital transformation
life facilitates and accelerates the resolution of
makes life easier and helps quickly solve many
several issues by citizens of one or another
problems. It is intended to improve the quality of
country. The Digital Quality of Life (DQL) Index
digital life that spending on digital transformation
is calculated by looking at the impact of five core
technologies and services is constantly increasing
pillars:
worldwide, as evidenced by the Statista website
1. Internet Affordability: This pillar determines
(see Fig. 1) [[1]–4].
how long an individual has to work to afford
Digital transformation means implementing
the internet, specifically the cheapest mobile
digital technologies to transform business
and broadband internet.
CPITS-2022: Cybersecurity Providing in Information and Telecommunication Systems, October 13, 2022, Kyiv, Ukraine
EMAIL: zhyrova@knute.edu.ua (T. Zhyrova); kotenkono@knute.edu.ua (N. Kotenko); b.bebeshko@knute.edu.ua (B. Bebeshko);
k.khorolska@knute.edu.ua (K. Khorolska); s.shevchenko@kubg.edu.ua (S. Shevchenko)
ORCID: 0000-0001-8321-6939 (T. Zhyrova); 0000-0002-2675-6514 (N. Kotenko); 0000-0001-6599-0808 (B. Bebeshko); 0000-0003-3270-
4494 (K. Khorolska); 0000-0002-9736-8623 (S. Shevchenko)
©️ 2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
110
2. Internal Quality: this pillar determines the According to a study published by the VPN
stability and speed of the internet for both service provider Surfshark, there is a
mobile data and broadband. It also considers corresponding distribution of DQL among the
the growth of speed and strength. countries of the world and Europe (see Fig. 2).
3. Electronic Infrastructure: this pillar measures The Digital Quality of Life 2021 index
how developed a country’s e-infrastructure is. analyses 110 countries worldwide in terms of five
Better infrastructure results in smooth online core pillars: internet affordability, internet quality,
experiences such as banking, shopping, e-infrastructure, e-security, and e-government.
entertainment, education, etc. These pillars consist of 14 indicators that are
4. Electronic Security is a measure of a nation’s interrelated and work together to measure the
cyber security and includes various strategies overall digital quality of life [5]. Countries were
the country has developed to decrease cyber- evaluated based on index scores with the best
crimes and protect the privacy of its people. possible value equal to one. Denmark is the best
5. Electronic Government is the pillar that at 0.83, followed by South Korea (0.76), Finland
reviews the advancements of the government (0.76), Israel (0.74), the United States of America
in the digital sector. This results in lesser (0.74), and Singapore (0.72). Ukraine ranks only
corruption, better transparency for the public, 47th in this ranking.
and efficient public services [5].
3 2.8
Spending in trillion U.S. dollars
2.5 2.4
2.1
2 1.8
1.5
1.5 1.31
1.18
0.96 1
1
0.5
0
2017 2018 2019 2020 2021* 2022* 2023* 2024* 2025*
Figure 1: Spending on digital transformation technologies and services worldwide from 2017 to 2025
in trillion U.S. dollars
Figure 2: Digital quality of life index in Europe for 2021
111
With the growth of DQL, the number of users 1. There are no clear and unambiguous
of government Internet applications is growing, definitions of such concepts as “disability,”
among which a particular share is occupied by “accessibility,” “person with disabilities,” or
people with special needs, whose number is “person with special needs,” which has some
constantly increasing. Such applications have ambiguity in research and needs clarification
several requirements. Note that to solve these and agreement at the global level.
issues, on October 26, 2016, the EU Parliament 2. There are no clear definitions and concepts of
and the Council of the European Union passed the such terms as “accessibility web design,”
EU Web Accessibility Directive, and on June 7th, “inclusive software,” or “universal web
2019, the European Union formally adopted the design.”
European Accessibility Act. The Directive aims to 3. To ensure a high level of content accessibility
create a more standardized and harmonized for people with disabilities, WAI (Web
framework around the accessibility of websites Accessibility Initiative), within the framework
and mobile applications of public sector bodies of the World Wide Web Consortium (W3C),
[6]. The Directive complements the European developed the Web Content Accessibility
Accessibility Act, which covers a wide range of Guidelines (WCAG) standard. Version
products and services also in the private sector. WCAG 2.1 was adopted as the standard of the
Further European legislation supports people with European Union EN 301 549 [21]. The
disabilities in other areas, including electronic recommendations for ensuring the
communications, audio-visual media services, accessibility of web content are the most
eBooks, eCommerce, and ICT equipment [7]. universal, and many countries have adapted
Following the specified directive, the following these requirements to their laws.
terms were established: on September 23, 2019, 4. There are many programs for automated web
all new public sector websites and apps were application accessibility testing. In the vast
required to conform to the directive; by majority, they do not check all WCAG 2.1
September 23, 2020, all new and existing public requirements, so there is a need for several
websites must conform to the directive; by June automatic testing tools.
23, 2021, all new and existing mobile apps must 5. No scientific publications highlight the issue
conform to the directive [6]. Thus, the question of the DQL Index, its factors, the relationship
arises about the relationship between the DQL between the DQL Index, the availability of
index and the availability of web applications of web applications, etc.
state importance. Therefore, the study of scientific and practical
literature, publications, and publications on the
2. Related Works Internet proved that the issue of the relationship
between the level of accessibility of software and
the Digital Quality of Life index needs thorough
The issue of software availability is quite
research and analysis, and there is also a need to
broad and multifaceted. It has worried several improve accessibility testing standards. Thus, this
scientists but is still not revealed and not article aims to analyze accessibility standards,
completed. Scientists: M. Goldberg [8], M. research existing automated web application
Campoverde-Molina 9, K. Ordonez 10, G. E. accessibility testing tools, and correlate test
Constain 11, J.S. Silva 12, K. Brown [13]3, results with the Digital Quality of Life index.
M. Sashnova [[14]], etc. Nielsen Norman Group
[[15]], which is a world leader in user experience
research, needs special attention. For the past few 3. Index Correspondence Research
years, their research has focused on website
accessibility. By Directive (EU) 2016/2102 Of The
Several scientists are working on the issue of European Parliament And Of The Council of 26
inclusive software: A. Savidis [[16]], J. Ohene- October 2016 on the accessibility of the websites
Dzhan [[17]], I. Niculescu [[18]], K.M. Martinez and mobile applications of public sector bodies
[[19]], V. Boronos [20] and others. A small [22], it is the applications of public sector bodies
number of publications are devoted to studying that require accessibility.
the digital quality of life. The standard contains a number of
The analysis of publications in this direction recommendations on what needs to be done to
showed that: make the web-content accessible to people with
112
disabilities. Recommendations are grouped Ireland, Luxembourg, Poland, Italy, Greece,
according to four principles: perception, Ukraine, Albania, Montenegro [27–36].
manageability, comprehensibility and reliability. Table 1 contains the results of testing the
These recommendations are called success accessibility of Government website home pages
criteria, and according to them, the application with the appropriate automated testing tools.
can receive one of three levels: A, AA or AAA. Testing of Siteimprove Browser Extensions
Accessibility testing at the appropriate level is was carried out according to the following
performed by automated systems and testers who criteria:
check whether the content of the site meets the Error. Occurrences of an issue that have been
relevant criteria and assess the usability of the automatically determined to be in
platform [[23]3]. contravention of the Web Content
The Accessibility Conformance Testing Accessibility Guidelines (WCAG) 2.
(ACT) Rules Format 1.0 defines a format for Warning. Occurrences of an issue that have
writing accessibility test rules. These test rules can been automatically determined to be in
be used for developing automated testing tools contravention of best practices under the Web
and manual testing methodologies. It provides a Content Accessibility Guidelines (WCAG) 2.
common format that enables any party involved in Review. Issues that cannot be checked
accessibility testing to document and share their automatically but requires a manual inspection
testing procedures in a robust and understandable to determine if each item lives up to the
manner. This enables transparency and success criteria.
harmonization of testing methods, including Testing of WAVE Browser Extensions was
methods implemented by accessibility test tools carried out according to the following criteria:
[[24]]. Error. Occurrences of an issue which have
There is a range of tools for automated testing, been automatically determined to be in
including Google ADT, Tenon, aXe, Wave, contravention of the Web Content
SiteImprove, and others. We have chosen the Accessibility Guidelines (WCAG) 2.
following tools for automated testing of the Contrast Errors. Text is present that has a
availability of government websites: Tenon, contrast ratio less than 4.5:1, or large text
Wave, and SiteImprove. This choice is due to the (larger than 18 point or 14 point bold) has a
GDS accessibility team’s audit of the most used contrast ratio less than 3:1. WCAG requires
accessibility tools [[25]]. that page elements have both foreground AND
Technology is constantly changing, so the web background colors defined (or inherited) that
application's user experience will vary depending provide sufficient contrast. When text is
on the technical capabilities of browsers and presented over a background image, the text
devices. Not all browsers display web pages the must have a background color defined
same way. There are often noticeable differences (typically in CSS) that provides adequate text
in how browsers handle technologies like CSS, contrast when the background image is
HTML, and JavaScript. Because, according to disabled or unavailable. WAVE does not
statistics provided on the Statcounter GlobalStats identify contrast issues in text with CSS
website, the browser market share in Europe was transparency, gradients, or filters.
distributed as follows: Chrome 59.29%, Safari Alerts. Occurrences of an issue that have been
19.97%, Edge 5.86%, Firefox 5.83%, Samsung automatically determined to be in
Internet 3.62%, Opera 3.02% [[26]]. That is why, contravention of best practices under the Web
within the framework of this study, testing the Content Accessibility Guidelines (WCAG) 2.
availability of web applications of state
Structural Elements. The number of structural
importance was carried out in Google Chrome.
elements on the page, such as headings of
We have chosen 10 European countries different levels, ordered lists and unordered
randomly to ensure the sample’s
lists, navigation, etcetera.
representativeness: Denmark, Switzerland, Tenon testing was carried out according to the
United Kingdom of Great Britain and Northern following criteria: Total Issues, Error Density.
113
Table 1
Test results according to the relevant criteria
Siteimprove
2021 Digital Quality of
Browser WAVE Browser Extensions Tenon
Extensions
Life Index
Error Density
Total Issues
№ Country
Structural
Elements
Contrast
Warning
Review
Errors
Errors
Errors
Alerts
1 Denmark 1 14 3 20 11 2 7 56 2 7
2 Switzerland 8 16 8 23 19 16 114 139 10 21
3 Great Britain 10 5 4 23 0 1 2 40 2 3
4 Luxembourg 15 8 4 21 20 21 5 45 22 8
5 Poland 25 9 4 21 15 11 9 34 9 11
6 Italy 27 12 2 16 7 0 8 37 18 8
7 Greece 38 7 2 20 0 0 53 5 21 6
8 Ukraine 47 27 4 20 11 9 22 128 14 19
9 Albania 64 22 6 21 16 40 31 66 37 5
10 Montenegro 78 11 4 20 0 0 43 54 2 11
Fig. 3 shows an infographic of the research that have been checked, k is the number of errors
results compared to the DOL Index. for each criterion.
To determine the accessibility index of the Fig. 4 shows an infographic that demonstrates
website, we use the formula: the correspondence between the DQL Index and
𝑛∙𝐸
the accessibility index of the home pages of the
𝐼 = ∑𝑛 ∙ 100%, (1) Government website.
𝑖=1 𝑘𝑖
According to the results of this study, it can be
Where n is the number of evaluation criteria of concluded that there is no correspondence
the web application, E is the number of elements between these indices.
90
78
80
70 64 Siteimprove
60 Browser
Extensions
50 47
2021 Digital
38 37 Quality of Life
40
Index
30 25 27 27
19 2022 18
21 22 WAVE Browser
20 14 16 15 15 14 16 Extensions
11 12 11 11
8 10 10 8 9 9 7 7
10 5
1 2 02 0 02 Total Issues
0
Figure 2. Correspondence diagram of the number of errors according to the relevant criteria and the
DQL Index
114
Montenegro 0.06
0.4668
Albania 0.04
0.5015
Ukraine 0.12
0.564
Greece 0.02
0.5879
Italy 0.06
0.6352
Poland 0.06
0.636
Luxembourg 0.05
0.6867
Great Britain 0.16
0.7065
Switzerland 0.11
0.7093
Denmark 0.07
0.8313
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Index Web Accessibility 2021 Digital Quality of Life Index
Figure 4. The correspondence between the DQL Index and the accessibility index of the home pages
of the Government website.
and Technology, 2019, pp. 39–44. doi:
4. Conclusion 10.1109/picst47496.2019.9061376.
[3] I. Bogachuk, V. Sokolov, V. Buriachok,
Monitoring Subsystem for Wireless Systems
The DQL index is calculated by analyzing the based on Miniature Spectrum Analyzers, in
impact of five main components: Internet V International Scientific and Practical
accessibility, Internet quality, e-infrastructure, e-
Conference Problems of Infocom-
security, and e-government. It does not consider munications. Science and Technology, 2018,
the accessibility requirements of digital products, pp. 581–585. doi: 10.1109/infocommst.
in particular web pages. It should be noted that EU 2018.8632151.
countries and their partner states are constantly [4] Y. Sadykov, V. Sokolov, P. Skladannyi,
working to improve conditions for people with Technology of Location Hiding by Spoofing
disabilities, but the DQL Index does not indicate
the Mobile Operator IP Address, in IEEE
a high level of accessibility of state websites. This
International Conference on Information and
is evidenced by the results of the conducted Telecommunication Technologies and Radio
research. For the first time, an analysis of the Electronics, 2021, pp. 22–25. doi:
correspondence between Index DQL indicators 10.1109/UkrMiCo52950.2021.9716700.
and the web application accessibility index was
[5] 2021 Digital Quality of Life Index,
carried out. Even though the EU countries and
https://surfshark.com/dql2021.
their partner states are constantly working to [6] EU Web Accessibility Compliance and
improve conditions for people with disabilities, a Legislation https://www.deque.com/blog/eu-
high DQL Index does not indicate a high level of web-accessibility-compliance-and-
accessibility of state websites, which was
legislation/.
confirmed by the study. [7] Web Accessibility https://digital-
strategy.ec.europa.eu/en/policies/web-
5. References accessibility.
[8] M. Goldberg, et al., Are Accessible Software
[1] Spending on Digital Transformation Accountable: A Commentary, Assistive
Technologies and Services Worldwide from Technology, 34:1, 2022, pp. 61–63, doi:
2017 to 2025, https://www.statista.com/ 10.1080/10400435.2021.2024627.
statistics/870924/worldwide-digital- [9] M. Campoverde-Molina, S. Luján-Mora,
transformation-market-size/. L. Valverde, Process Model for Continuous
[2] M. Vladymyrenko, et al., Analysis of Testing of Web Accessibility, in IEEE
Implementation Results of the Distributed Access, vol. 9, 2021, pp. 139576–139593,
Access Control System, in VI In-ternational doi: 10.1109/ACCESS.2021.3116100.
Scientific and Practical Conference [10] K. Ordonez, J. Hilera, S. Cueva, Model-
Problems of Infocommunications. Science Driven Development of Accessible
115
Software: A Systematic Literature Review. Economic Annals-XXI, 172(7–8), 2018,
Univ Access Inf Soc 21, 2022, pp. 295–324. pp. 38-43. doi: 10.21003/ea.V172-07.
doi: 10.1007/s10209-020-00751-6. [21] Countries that Have Adopted WCAG
[11] G. E. Constain, et al., Framework for the Standards, https://www.3playmedia.com/
Design of Accessible Software to Support blog/countries-that-have-adopted-wcag-
Users With Autism: A Proposal Oriented standards-map/.
from the HCI, 2022 17th Iberian Conference [22] Directive (EU) 2016/2102 Of The European
on Information Systems and Technologies Parliament And Of The Council of 26
(CISTI), 2022, pp. 1–5. doi: 10.23919/ October 2016, https://eur-lex.europa.eu/
CISTI54924.2022.9820360. legal-content/en/txt/html/?uri=celex:
[12] J. S. Silva, R. Gonçalves, F. Branco, 32016L2102&from=EN.
Accessible Software Development: A [23] T. Zhyrova, et al., Testing the Accessibility
Conceptual Model Proposal, Universal of Web-Applications, Computer Systems
Access in the Information Society 18, 2019, and Information Technologies, no. 3, 2021,
pp. 703–716. doi: 10.1007/s10209-019- pp. 89–95. doi: 10.31891/CSIT.
00688-5 [24] Web Content Accessibility Guidelines,
[13] K. Brown, et al., Students with Disabilities as https://www.w3.org/.
Partners: A Case Study on User Testing an [25] Accessibility Tool Audit, https://alphagov.
Accessibility Website, International Journal github.io/accessibility-tool-audit/.
for Students as Partners, 4(2), 2020, pp. 97– [26] Browser Market Share in Europe, August
109. doi: 10.15173/ijsap.v4i2.4051. 2022, https://gs.statcounter.com/browser-
[14] M. Sashnova, et al., Detection of market-share/all/europe.
Accessibility and Quality of Websites of the [27] Denmark Government Official Website,
Leading Universities of the World, Journal of https://denmark.dk/.
Theoretical and Applied Information [28] Switzerland Government Official Website,
Technology, 99(12), 2021, pp. 2845–2857. https://www.admin.ch/gov/en/start.html.
[15] Nielsen Norman Group, [29] The United Kingdom of Great Britain and
https://www.nngroup.com/. Northern Ireland Government Official
[16] A. Savidis, C. Stephanidis, Inclusive Website, https://www.gov.uk/.
Development: Software Engineering [30] Luxembourg Government Official Website,
Requirements for Universally Accessible https://gouvernement.lu/en.html.
Interactions, in Interacting with Computers, [31] Poland Government Official Website,
vol. 18, no. 1, 2006, pp. 71–116. doi: https://www.gov.pl/.
10.1016/j.intcom.2005.06.005. [32] Italy Government Official Website
[17] J. Ohene-Djan, R. Shipsey, Principles for https://www.governo.it/en.
Inclusive Software Design of Learning [33] Greece Government Official Website
Technologies, in 2018 Eighth IEEE https://www.gov.gr/en/sdg/.
International Conference on Advanced [34] Ukraine Government Official Website
Learning Technologies, 2008, pp. 989–990, https://www.rada.gov.ua/.
doi: 10.1109/ICALT.2008.254. [35] Albania Government Official Website
[18] I. Niculescu, et al., Towards Inclusive https://www.kryeministria.al/en/.
Software Engineering Through A/B Testing: [36] Montenegro Government Official Website,
A Case-Study at Windows, in 2021 https://www.gov.me/en/.
IEEE/ACM 43` International Conference on
Software Engineering: Software Engineering
in Practice (ICSE-SEIP), 2021, pp. 180–187.
doi: 10.1109/icse-seip52600.2021.00027.
[19] C. M. Martínez, et al., Responsive Inclusive
Design (RiD): A New Model for Inclusive
Software Development, Univ Access Inf.
Soc., 2022. doi: 10.1007/s10209-022-00893-
9.
[20] V. Boronos, et al., Digital transformation of
Ukraine: challenges of theory and practice in
implementation of digital quality of life.
116