=Paper= {{Paper |id=Vol-2105/10000290 |storemode=property |title=Information and Technological Service for the Accompaniment of the Educational Process of People with Visual Impairments |pdfUrl=https://ceur-ws.org/Vol-2105/10000290.pdf |volume=Vol-2105 |authors=Nataliia Veretennikova,Oleksandr Lozytskyi,Nataliia Kunanets,Volodymyr Pasichnyk |dblpUrl=https://dblp.org/rec/conf/icteri/VeretennikovaLK18 }} ==Information and Technological Service for the Accompaniment of the Educational Process of People with Visual Impairments== https://ceur-ws.org/Vol-2105/10000290.pdf
     Information and Technological Service for the
  Accompaniment of the Educational Process of People
               with Visual Impairments

   Nataliia Veretennikova, Oleksandr Lozytskyi, Nataliia Kunanets, Volodymyr
                                   Pasichnyk

Information Systems and Networks Department, Lviv Polytechnic National University, Lviv,
                                       Ukraine

          nataver19@gmail.com, Oleksandr.A.Lozytskyy@lpnu.ua,
               nek.lviv@gmail.com, vpasichnyk@gmail.com



      Abstract. The authors propose innovative approaches to the improvement of in-
      formation technologies, methods and tools, which implement the formation of
      electronic library funds with specialized information products, integrating vari-
      ous types of information to provide information needs of people with disabili-
      ties. It is offered the new approaches to the formation of information resources
      and electronic libraries for this category of people promoting the processes of
      inclusive education. The processes of content preparation for providing conven-
      ient information and technological interaction of the blind with the information
      resources are analyzed, as well as the choice of the DAISY format (Digital Ac-
      cessible Information System) for the creation of information products (audio
      books) with the use of flexible navigation mechanisms is substantiated. Moreo-
      ver, it is proposed the classification of documents, which allows you to choose
      the levels of their structuring for effective transformation into DAISY format.
      The developed technology by authors is introduced and allows you to convert
      mathematical formulas of different complexity into a form that can be pro-
      nounced effectively by a synthesizer of the Ukrainian language.
      The developed software and algorithmic means of sounding mathematical for-
      mulas in Ukrainian have given an opportunity to dub the Ukrainian technical
      texts with the help of a language synthesizer, fill DAISY books with technical
      content and convert the mathematical formulas into a text description. To con-
      vert the mathematical formulas presented in various versions of the MathML
      (presentation and semantic) records into an Ukrainian text, a special system of
      rules has been developed, which consists of rules for writing mathematical
      symbols, operators, general and specified expressions.

      Keywords: DAISY format, electronic library, information resources for people
      with special needs.
1      Іntroduction

In the modern information society, a rethinking of traditional forms of information
presentation is needed to the effective information support of the educational process
of people with disabilities. The need of information resource formation in electronic
libraries for this category of people has become especially relevant in connection with
military actions taking place in Ukraine. Military men, who have been injured, after
recovering, need to obtain “peaceful” occupations. But some of them got disabilities.
This category of citizens needs special attention from the side of society. And learn-
ing processes require special approaches and specific information support, especially
for visually impaired people.
    The purpose of this paper is to present innovative methods and means of infor-
mation and technological support of the educational process of people with visual
impairment.


1.1    Analysis of the Research State
Increasingly, the need to expand the ability to provide information conveniently for
people with special needs in different perceptual formats becomes relevant. The in-
formation technologies, methods and means that implement the resource formation of
electronic libraries are developed and improved by specialized information products
integrating different information to provide information needs of people with various
physical disabilities. The research results on the information availability for blind
users are reflected in the works of researchers such as Jeffrey P. Bigham, Meredith
Ringel Morris, Yu Zhong, Samuel White [1] and others. Problems of creating infor-
mation educational content are analyzed by Krapivenko A.V. [2] and Davydova E.V.
[3] who investigate the problem of formation of effective multimedia information
products. However, a comprehensive study of the peculiarities of the formation of
information content for people with special needs has not been carried out yet.


1.2    Research Methods
The analysis of content preparation processes for the information and technological
support of the educational process of the blind, conducted by the authors of the arti-
cle, confirms the validity of the selection of the electronic book format for the blind,
named DAISY format (Digital Accessible Information System). In this format books
are convenient for the formation of an electronic library in terms of presentation of
multimedia content. For the arrangement of the automated workplace of a blind user
(AWP), modern special tools and technologies are used that provide convenient ac-
cess of a blind user to information resources. To prepare the input data, the main pro-
visions and methods of statistical analysis [4], classification [5, 6], and construction of
decision trees are used, and to prepare the output audio content, it is used the method
of transformation of the syntactic tree and the method of creating DAISY books in the
Ukrainian language.
2      Electronic Libraries for People with Special Needs

Electronic libraries in the information society turn into convenient means for preserv-
ing information resources that provide information support and support for education-
al and social and communicative processes for people with special needs. An elec-
tronic library provides easy access to information resources online [7]. A characteris-
tic set of basic features of an electronic library is its computer equipment, a digital
way of recording information, technology of online access to information resources,
etc. [8]. In addition to electronic documents, databases, user maps, hyperlink subsys-
tems, other electronic information resources are the components of electronic libraries
[9].
    In this context, the information support of the educational process of people with
special needs is the following:
     - the use of specific forms of information representation for people with spe-
          cial needs, which in turn affects the features of the relevant information re-
          source;
     - the implementation of technological processes based on the algorithms of the
          information resource presentation created for each category of such users;
     - the development of methodological recommendations for the information
          support implementation and information and technology support for the edu-
          cational process of people with disabilities.
    The term of electronic library for users with special needs means an information
system designed to accumulate, structure and organize an array of electronic docu-
ments with an appropriate access system to provide a user with special needs with an
integral toolkit for relevant and quick work with information resources in an easy
format. In this case, special attention is paid to users with visual impairments. Modern
technologies are capable of not only converting electronic documents to a convenient
audio format for a blind user or dotted Braille font, but also provide the blind with a
complete computerized workplace with all the possibilities, starting from work in text
editors and ending with an access to the Internet or mastering skills on the computer.
    When developing electronic libraries for people with special needs, specific meth-
odological approaches are used, tested at creation and formation of traditional librar-
ies. They have a reference search engine and a library and information resource.
When forming an informational resource for blind users it is reasonable to accumulate
electronic documents in the DAISY format.


3      DAISY Book in the Context of Information Support of the
       Learning Process

DAISY is an open international information and technology standard for access to
multimedia content. The main developer of this standard is the DAISY Consortium,
which interacted with a number of professional and civic organizations and formed
the conceptual framework of the standard in close cooperation with leading staff of a
number of libraries, scientists and user-practitioners. The science and technology
innovation was directed to the main target group of vision-impaired users and users
with other physical disabilities. The DAISY book is multimedia content with syn-
chronization of text, audio and graphic information as well as advanced features for
flexible navigation in it.
     The core of DAISY technology is the effective tools for synchronizing text,
graphics and audio based on the W3C recommendations, according to the people’s
needs that require providing a high-quality special way to information access.
     The authors’ analysis of the functional capabilities of the technologies that are
fixed by the DAISY standard allows us to confirm that it can be used to provide a
high quality support of a wide range of requirements inherent in database creation
processes that contain multimedia information for the blind as it provides the audio
content creation using mechanisms of flexible navigation. Users can listen to such
book not only linearly; navigation tools provide an opportunity to make transitions
from a section to a section, a subsection, a paragraph, a page. Documents in the
DAISY format allow bookmarking of specific places in the text to re-listen to it and
put voice tags.
     A navigation map is imposed on the audiobook in a certain way and a “reader”
can not only listen to the text, but also work with it: make bookmarks, notes, quick
access to the necessary information. The DAISY books can be structured or unstruc-
tured in general. The decision on the structure of such book is taken when converting
it into DAISY format.
     The DAISY format books have the MP3 file archiving technique, which can hold
up to 90 hours of audio tracks that can be listened to both on special playback devices
and on the computer where the appropriate software is installed.
     The DAISY format specification uses numerous cross-references among XHTML
text files, MP3 audio recordings, SMIL synchronization files, and NCX navigation
control. Extensible Hypertext Markup Language (XHTML) is an expanding hypertext
markup language based on XML and features similar to HTML.
     The DAISY book can consist of audio files, text files and images, or their combi-
nations [10]. All DAISY books use a common set of file types, although some files
are optional.
     Almost all types of files in the DAISY format are based on XML [11]. The most
important types of files that are part of DAISY books are batch, text content, image,
audio file, synchronization, navigation management, resource, style presentation and
transformations.


4      Technologies for Voice Recognition                      of    Mathematical
       Expressions in the DAISY Format Books

For the organization of the educational process in the natural sciences, books with a
formulaic component are required. For the transfer of mathematical expressions in
books created in the DAISY format, the mathematical markup language MathML is
used which is an XML element and is designed for use in XHTML documents.
    Documents using the MathML language are not directly reproduced by the syn-
thesizer. For the correct reproduction of such documents, various means are used to
create different mathematical notations and text descriptions in different languages
(except of Ukrainian). Most DAISY format playback players use the notation
LAMBDA, LaTeX and Nemeth (Table 1).

              Table 1. Encoding mathematical formulas in different notations

     Formula encoding                                          Example
       (notation)
         Traditional                     x2  y2
                                 1              (x  y)  0
                                          x y
           LaTeX            1  \ sqrt      x **2  y **2 /  x  y   *  x  y    0
            AMS             1      x **2  y **2 /  x  y   *  x  y   / /2  0
          Nemeth             #1  ? X ^ 2" Y ^ 2"/ X  Y #  X  Y ]  K #0

    Depending on the type of a player, it can voice mathematical formulas, alternative
text (verbal description of a formula), work with the structure of a formula, etc. Our
developed technology allows us to convert mathematical formulas of different com-
plexity to the form which can further be expressed by the Ukrainian language synthe-
sizer.
    In order to automate the process of creation of teaching materials in Ukrainian in
the form of DAISY books it was developed an applied programmed system for pro-
cessing Ukrainian technical texts for people with visual impairments. The basis of the
applied system is the modular structure, which makes it possible to implement it as
separate functional modules.
    The system consists of the components such as drivers of special equipment, basic
software and special software. In Fig. 1, in the Special software block, the modules
developed during this study are highlighted in the dotted lines:
    •    a module for processing files of various formats (allows a user to convert the
          format of an input book according to their needs);
    •    a module of keyword search in the text (provides search and text marking
          based on the structural features of a book, as well as mathematical formulas,
          descriptions of figures, etc.);
    •    an overlay navigation module on the book (this module is responsible for
          overlaying the selected navigation scheme on the input document and divid-
          ing it into separate parts);
    •    a module for converting a formula to a text description (provides the trans-
          formation of mathematical formulas and special characters into a text de-
          scription in Ukrainian, in accordance with the rules developed by the trans-
          formation);
   •     a module for content layout and content storage (is responsible for preserving
         the structural parts of the book text in separate files, as well as giving an op-
         portunity to voice them in Ukrainian).




Fig. 1. Structural scheme of the applied system for processing Ukrainian technical texts for
people with visual impairments

    Each module has a finished functionality, working with an appropriate set of input
parameters and output data.
    The main function of the applied system for processing Ukrainian technical texts
is the processing of information, namely collecting (receiving), processing, storing
and displaying. The software interfaces of the system are designed for an inexperi-
enced user, divided into many types of menus with a possibility to dub them with a
reading system from the screen or output to the Braille line.
    The program module of the applied program system for processing Ukrainian
technical texts, which provides the transformation of mathematical formulas into an
audio format, is developed by means of the language C # and available at:
http://www.mathplay.ho.ua/
    In the process of automated recording over information content, there are prob-
lems associated with the transformation and presentation of mathematical formulas.
    Because of significant differences in constructing and reading mathematical for-
mulas in different languages, it is impossible to adapt English-language systems such
as MathPlayer, Dolphin EasyReader, MathSpeak due to differences in the construc-
tion of sentences, besides, the indicated software is licensed rather than in open
source. It causes precisely to develop an applied programmatic system for the sound-
ing of Ukrainian technical texts with a formulaic component.
    We have not found publications in Ukrainian or Russian that would contain rules
for reading mathematical formulas or methods for decomposing mathematical formu-
las, but there are some publications in English [12, 13]. These publications are basis
for the development of rules for decomposition of the mathematical formula. To con-
vert MathML to a text, a transformation model for a syntax tree is used, based on the
rules described in these publications.
    Taking into account the prevalence of the MathML language, its tree structure and
the availability of means for converting formulas in different formats to the MathML
language, we consider appropriately it to use for internal formula recording. The
MathML language is selected as an intermediate one for the development of means
for sounding formulas.
    The formula written in the MathML language has a tree structure, which makes it
possible to describe the semantics of the mathematical expression unambiguously in
the information system. Writing a mathematical formula in the MathML language can
be represented as a tree whose vertices are operations and operands. In this case, each
node in the tree corresponds to a particular layout, and its branches or descendants are
subexpressions. In other words, it is a graphical representation of a mathematical
formula that shows how exactly the MathML tags should be inserted in each other for
the correct representation of a given mathematical expression on the screen (Fig. 2).




                      a)                                               b)
 Fig. 2. Formula a * x ^ 3 + b in the language MathML (a), its image in the form of a tree (b)

    For the transformation of the mathematical formulas presented in various versions
of the MathML (presentation and semantic) record, a special system of rules has been
developed for the text in Ukrainian. The system consists of rules for writing mathe-
matical symbols, operators, general and specified expressions.
    Rules for specified expressions are necessary in cases when the reading result de-
pends not only on the tree node, but also on the value of its descendant. For example,
   should be read the “x square”, and not “xx in power two”. The rules are designed
so that the original text could be read with the synthesizer of the Ukrainian language.
     The developed rules for the conversion of the mathematical formula are divided
into four groups:
     1. Character conversion rules. In accordance with these rules, the MathPlay pro-
gram is designed to convert formula symbols recorded in the MathML language, for
example, “x” – “x”, “a” – “a”, “b” – “b”, “c” – “c”, “dt” – “dt”, “\u221E” – “infinity”,
etc.
     2. Operator conversion rules. With the help of these rules, formulas are trans-
formed, for example, “-“ or “minus” – “minus”, “+” or “plus” – “plus”, “=” or “eq” –
“equal”, “times” – “multiply”, “±” – “plus minus”, “(“ – “open a bracket”, “)” –
“close a bracket”, “\ u222B” – “integral”, etc.
     3. Common expression conversion rules. These rules are used in the following
cases:
      - raise to the power: “msup” – “*number_1* is raised to the power
          *number_2*”;
      - raise to the power “power” – “*number_1* is raised to the power
          *number_2*”;
      - division: “mfrac” – “*number_1* is divided by *number_2*”;
      - extraction of the square root: “msqrt” – “root square from *number_1*”;
      - extraction of the root of n-th power: “root” – “root of power *number_1*
          from *number_2*” etc.
     4. Specified expression conversion rules. These rules are used in the following
cases:
      - “power 2” – “square”;
      - “power 3” – “cube”;
      - “root 3” – “root cubic from *number_1*”;
      - “apply plus” – “*number_1* plus *number_2*” etc.


5      Analysis of the Results of the work of the Applied Program
       System for Processing Ukrainian Technical Texts

To test the results of the applied programmed system for processing Ukrainian tech-
nical texts, five groups of experiments were carried out to convert mathematical for-
mulas presented in various formats into the mathematical markup language MathML,
as well as dubbing and perception of the obtained formulas by blind users by ear (Fig.
3).
    For experiments on the convertion of formulas into MathML language, a random
set of mathematical formulas of various complexity is taken, from simple ones as
            , to complex ones as                                               .
    The test base for the first four groups of experiments contained 100 formulas, with
25 formulas per group. Among them, 20 formulas are simple arithmetic expressions,
another 20 are trigonometric expressions, 20 ones are integrals and derivatives, 20 are
linear algebra formulas and 20 are series, borders, etc.
    During the experiments, free software (UkrVox, Ttm, OpenOffice, etc.) and demo
versions of commercial products (GrindEQ Math Utilities, InftyReader, MathType,
etc.) were used.




                        Fig. 3. Conducting experimental research

    The group of experiments № 1 (*.DOC  «GrindEQ Math»  TeX  Teacode
Latex  MathML).
    The first group of experiments was to convert the formula set into the MathML
language written as a Microsoft Word document format.
    The group of experiments № 2 (*.PDF  Infty  MathML).
    The second group of experiments was to convert the formula set into the MathML
language written in the Adobe Reader, * .PDF format.
    The group of experiments № 3 (*.HTML  selecting graphic objects  JPG,
PNG, GIF  Infty  MathML).
    The third group of experiments was to convert a formula set into a MathML lan-
guage written as a * .HTML web page.
    The group of experiments № 4 (MathType  MathML).
    The fourth group of experiments was to convert a formula set written with Math-
Type into the MathML language and save the copied formula in a text editor with *
.MML extension.
    According to the results of the conducted experiment, the greatest errors and in-
correctness were when the mathematical formulas were recognized by means of the
Infty program and converted to the mathematical markup language MathML. Only 21
formulas were correctly recognized from the 50 formulas of different complexity (25
formulas from the experiment group № 2 and 25 formulas from the experiment group
№ 3). Thus, Infty software, due to the imperfection of the recognition means, provides
only 42% of the accuracy of recognition processes of complicated formulas. For ex-
ample, the formula of medium complexity                       was recognized wrongly,
with two errors: 3√ (b + c) = Ω.
    In parallel, the processes of converting the formulas to the MathML language by
technologies selected for the first and fourth groups of experiments gave the correct
result on 100%, and non-significant inaccuracies were recorded for the second and the
third groups.
    The group of experiments № 5 (dubbing the formulas in Ukrainian transformed
into MathML language).
    For this group of experiments, 150 different formulas were taken during the re-
search. All formulas are divided into groups: equations (rational, square, linear), ine-
qualities, identical expressions, fractional expressions and functions. Each group in-
cluded formulas of various complexity.
    Experiments were consisted of the correct perception by ear of the mathematical
formula dubbing by the developed MathPlay program and recorded it in the notebook
during the listening.
    Performing experiments on simple formulas (250 formulas), five respondents
made two errors during playback, making up 0.8% of the total number of records.
According to the calculations of experiments, respondents made 11 mistakes on for-
mulas of average complexity with the same number of formulas, that is, 4.4% of the
total number of records. The third part of the experiments over complex formulas
showed the following result: 44 errors, or 17.2% of the total number of records.
    Consequently, the MathPlay program of dubbing mathematical formulas and sym-
bols has produced a good result. The program correctly reproduces 693 out of 750
records (92.5%) for formulas of various complexity.
    According to the participants of the experiments, the main difficulty during the
experiments was the understanding of the synthesized voice and high speed of repro-
duction. In addition, the overall result was influenced by the formula complexity.


6      Interface of a Blind User Interaction with an Electronic
       Library

One of the most important tasks faced by the developers of the library information
and technology service for people with visual impairments is a solution of the issues
of the multimedia structured content formation for a convenient interaction of the
blind with library services by means of computer.
    As the main purpose of developing such system is to maximize the automation of
the interaction between a blind reader and an electronic library, the proposed model of
the system will provide such interaction by means of an automated workplace through
the program interface with special way of input and output infor-
mation.




            Fig. 4. Interaction process of the blind user and the electronic library
    In the Fig. 4 it is shown a block diagram of the interaction process of a blind user
with an electronic library. With the help of a specially equipped automated workplace
and software interface, the blind user can access to library resources and find a neces-
sary book for reading.
    The general scheme of the equipped automated workplace for a blind user consists
of basic and special technical means, software, various organizational measures. In
addition to working on a computer, a person with visual impairment can receive in-
formation from additional technical devices such as a TV, a mobile phone, audio
players and magnifying devices, etc. This automated workplace allows synchronizing
these devices with a computer for convenient and fast operation.
    The system of interaction of a blind user with a computer can be supplemented by
additional software and hardware, in accordance with the needs of the blind, in partic-
ular, for working with math, graphic objects and musical instruments.


7      Conclusion

The developed software and algorithmic means of sounding mathematical formulas in
the Ukrainian language gave an opportunity to voice the technical texts of the Ukrain-
ian language with the help of a language synthesizer, fill DAISY books with technical
content and convert mathematical formulas into a text description. An integral part of
the soundtrack is a special system of rules, which ensures the correct conversion of
the mathematical formulas presented in various versions of the MathML record. The
system of rules consists of algorithms for writing mathematical symbols, operators as
well as general and specified expressions.
    The developed applied programmed system for processing Ukrainian technical
texts provides automation of processes for the creation of adapted teaching materials
containing mathematical formulas for the needs of a blind user.
    Experiments on the correctness of automatic sounding of mathematical formulas
carried out during the testing of the applied programmed system for processing
Ukrainian technical texts allow us to assert that the developed software product pro-
vides a clear dubbing in Ukrainian of 92.5% of the formulas of various complexity.
There are no computer technological and methodological analogues of this software
product in Ukraine. The presented workings are technologically commensurate with
the world's achievements in the field of cutting-edge information technology focused
on people with special needs. The applied program system for processing Ukrainian
technical texts is used to accompany the educational processes of blind pupils and
students.


References
 1. Brady, E., Morris, M. R., Zhong, Yu, White, S., Bigham, J. P.: Visual Challenges in the
    Everyday Lives of Blind People. CHI: Changing Perspectives, Paris, France (2013)
 2. Krapivenko, A.V.: Multimedia technology and the perception of feelings: Textbook.
    BINOM. Laboratory of Knowledge, Moscow (2009). (in Russian)
 3. Davydova, E. V.: Creating graphics for Web pages. Informatics and education 6, 72-80
    (2001). (in Ukrainian)
 4. Rashkevych, Y., Peleshko, D., Vynokurova, O., Izonin, I., Lotoshynska, N.: Single-frame
    image super-resolution based on singular square matrix operator. In: 2017 IEEE First
    Ukraine Conference on Electrical and Computer Engineering (UKRCON), Kiev, pp. 944-
    948. (2017) doi: 10.1109/UKRCON.2017.8100390.
 5. Tkachenko, R., Doroshenko, A., Izonin, I.,Tsymbal Y., Havrysh, B.: Imbalance Data Clas-
    sification via Neural-like Structures of Geometric Transformations Model: Local and
    Global Approaches. In: Hu, Z.B., Petoukhov, S., (eds) Advances in Computer Science for
    Engineering and Education. ICCSEEA2018. Advances in Intelligent Systems and Compu-
    ting. Springer, Cham (2018).
 6. Tkachenko, R., Izonin, I.: Model and Principles for the Implementation of Neural-Like
    Structures based on Geometric Data Transformations. In: Hu, Z.B., Petoukhov, S., (eds)
    Advances in Computer Science for Engineering and Education. ICCSEEA2018. Advances
    in Intelligent Systems and Computing. Springer, Cham (2018).
 7. Schrajberg, Ya.L.: Modern Trends in the Development of Library and Information Tech-
    nologies. Annual plenary report of the international conferences "Crimea", p. 44 (2001)
 8. Sukiasian E.R.: Discussion Club "Termin". Scientific and Technical Libraries 6, 113-119
    (2000). (in Russian)
 9. Proksheva, T.: Libraries, computer technologies and information society: new trends, new
    perspectives, http://www.chl.kiev.ua
10. Ladner, R.E., Ivory, M.Y., Rao, R., Burgstahler, S., Comden, D., Hahn, S., Renzelmann,
    M., Krisnandi, S., Ramasamy, M., Slabosky, B., Martin, A., Lacenski, A., Olsen, S., Cro-
    ce, D.: Automating Tactile Graphics Translation. In: Computers and Accessibility 7th Int.
    ACM Sigaccess Conf., pp. 50–57, New York, (2005).
11. National Information Standards Organization. Specifications for the Digital Talking Book,
    http://www.niso.org
12. Rozenberg G.: Handbook of Graph Grammars and Computing by Graph Transformation.
    Foundations, Volume I, p. 112 (1997).
13. Ehrig H., Engels G., Kreowski H.–J., Rozenberg G.: Handbook of Graph Grammars and
    Computing by Graph Transformations. Applications, Languages and Tools World, Vol-
    ume 2, p. 132 (1999).