The dynamics of modern inclusive society development emphasize the need to integrate people with visual impairments into active social life. The problem of socializing individuals with visual impairments involves various aspects that complicate their education, training, and full participation in society [1]. Specifically, people with visual impairments have limited access to information, as many materials are produced only in the usual printed or digital formats. This issue is further exacerbated by the increasing prevalence of information in graphic form, designed for more effective perception by readers. The aforementioned problems hinder the ability of individuals with visual impairments to receive quality education and professional development [2,3,4,5,7].
An analysis [6] of the activities of publishing and printing industry enterprises that produce educational and methodological literature (textbooks, manuals, etc.) for people with visual impairments revealed problems related to the creation or adaptation of images and illustrative materials, which are particularly crucial for this type of publication. When creating or adapting graphic materials, enterprises encounter the following issues: an insufficient number of trained specialists with specific competencies related to the technical implementation of tactile graphics; additional time and financial costs for training specialists; and the high labor intensity and cost of the process of creating or adapting tactile graphics. Consequently, the production issues surrounding tactile graphics remain one of the primary factors contributing to the low level of access to graphical information for people with visual impairments.
Scientists are working to solve the problem of producing tactile graphics by developing models for the automatic generation of tactile images [8,9,10,11,12,13]. The task of most existing models is to transform the content of a photo image into a tactile one.
Models [8,9,10,11] that attempt to directly convert the content of an image into a tactile format usually utilize computer vision and have the following disadvantages: they violate the requirements for tactile graphics [14,15]; they display redundant elements of the image that are difficult to read and interfere with the overall interpretation of the graphic material.
In models [12,13] whose principle of operation involves the detection and subsequent recognition of individual image elements, replacing these elements with their tactile representations from a limited sample, there is no variability in the synthesized image samples (it is impossible to synthesize new samples, and the attractiveness of the synthesized image for people with visual impairments decreases). Despite the mentioned drawback, it should be noted that the method effectively conveys the content of the original photo at a high level in compliance with the requirements for tactile graphics.
Additionally, such methods require supplementary source graphic information (e.g., photographs), the search for or creation of which slows down the process of preparing material for the production of tactile images.
The development of information technologies, particularly in the field of deep machine learning, has opened new opportunities for addressing the aforementioned problems. Recently, significant advancements have been demonstrated by information technologies based on artificial intelligence [16,17,18], which enable the generation of images based on user text prompts. However, according to the analysis [19,20], confirmed by a series of experiments, the information technologies built upon these mathematical models have proven ineffective for creating tactile graphics. Despite this, the concept of text-guided image generation was chosen as the foundation for this work.
The text-conditional tactile graphics generation model is built upon the Bidirectional and Auto-Regressive Transformer (BART) [21] and Vector Quantized Variational Auto-Encoder (VQ-VAE) [22]. The subject of its modeling is the process of converting text information into graphic information. To do this, the embedded space of the transformer, which was formed during language modeling on pretraining task, was divided into two independent embedded spaces: text and graphics, instead of a shared one. At the same time, the parameters of the graphic embedded space were adjusted so that the dimension of the embedded space was equal to the size of the "codebook" [22], and the dimensionality of the vectors of the graphic embedded space was equal to the dimensionality of the latent space vectors of the variational image synthesis model. The parameters of the text embedded space remained the same as during language modeling.
Before obtaining text tokens using the BPE [22,23] tokenization model, the original text components are normalized by bringing them to a uniform format (uppercase letters were converted to lowercase letters).
Formally, the process of converting text tokens into graphic tokens using a text-conditional tactile graphics generation model is described in successive stages.
The first step is to generate a bounded sequence of text tokens based on a text prompt:
where is a sequence of text tokens of dimension πππ '(),+ = 64; π is a dictionary of tokens. If the size of the generated sequence of text tokens exceeds the value, its size is reduced to the maximum value, discarding the excess tokens. If the size of the generated sequence of text tokens is smaller than the value, its size is increased to the maximum value by adding utility tokens β©ππ΄π·βͺ that do not affect the simulation result.
In the next step, the text tokens that form the sequence π‘ Μ are mapped to the text embedded space vectors π , , forming a subset of it:
where π‘ Μ is the sequence of text tokens; πΈ + is a text embedded space; π , + are elements of the text embedded space. Elements π + 2 reflect the semantic meaning of text tokens in the embedded space.
Next, the vectors of the text embedded space πΈ + are transformed by the transformer's bidirectional encoder, which is formed from several layers, forming hidden states β + ;;; . The bidirectionality of the encoder means that it analyzes the full context of an individual vector of the embedded space, considering both the previous and the following elements of the sequence:
where β + ;;; is the hidden state of the encoder; πΈπππππ(β’) is the transformer's encoding operation defined within [21].
The hidden state of the encoder β + ;;; is then converted by linear layers and a nonlinear activation function to the hidden state of the decoder (i.e., graphic information), forming a subset of the graphic embedded space πΈ -:
where β + ;;; β πΈ + is the hidden state of the encoder; β - ;;;; β πΈ -is the hidden state of the decoder;
πΏπππππ ! is a linear layer; π ππΏπ β max (0, π₯) is a non-linear layer, activation function. At the next stage, an autoregressive [25,26] transformer decoder is used. This means that the decoder generates one graphics token per iteration, considering the context of the previously generated graphics tokens. Thus, during the decoding process, the model performs calculations based on the hidden state β + ;;; and pre-generated elements of the vector sequence of the graphic embedded space π , -, or β - ;;;; :
where π ! -is the i-th element of the vector sequence of the graphic embedded space πΈ -; π 1 -; π < π are previously generated vectors of the graphic embedded space; β - ;;;; is the hidden state of the decoder; π 0 is the size of the final sequence π - ;;; β πΈ -; π·πππππ(β’) is the transformer's decoding operation defined within [21].
Decoding occurs in an iterative manner until the sequence π - ;;; size is equal to π 0 (i.e., the size of the latent space vector of the VQ-VAE model). Once the decoding is complete, the resulting sequence of vectors of the graphics embedded space π - ;;; is converted by a linear layer and ππππ‘πππ₯ function into a sequence of probability distributions from which the element with the highest probability is selected, determining the selected graphics token:
where πΜ is the generated sequence of graphic tokens of size π 0 ; π ! -β π - ;;; is an element of the vector sequence of the graphic embedded space πΈ -. In the next step, on the basis of graphic tokens (5), a sequence of latent quantized vectors is formed π§ & , which is defined by the formula (6). Each graphic token: π ! 1 β€ π ! β€ πΎ; π = 1. . π 0 , is the positional number of the quantized vector in the "codebook" of the VQ-VAE model: where π is the set of latent quantized vectors, or "codebook"; π§ & β π is a sequence of latent quantized vectors; π ! β πΜ is a graphic token; π 0 is the size of the sequence of latent quantized vectors.
The final step is the synthesis of tactile graphics using a sequence-based variational image synthesis model decoder (6):
where π§ & is the sequence of latent quantized vectors; πΌππ·πππππ(β’) is an image decoding operation based on latent representation defined within [22]; π is a generated tactile image. The diagram of the text-conditional tactile graphics generation model is shown in Figure 1.
In this experiment, the proposed model was trained using the parameters presented in Tables 1 and 2 for the BART and VQ-VAE models, respectively. It is important to note that the size of the decoder's dictionary and the length of the sequence are each increased by one unit compared to the original values. This adjustment is necessary to introduce an additional image service token (i.e., SOS token), which is added at the beginning of the sequence to facilitate autoregressive image generation. The language modeling has been done using the BrUK corpus [27] consisting of Ukrainian texts from different sources. Unlike textual datasets (i.e., corpora), which are widely accessible, tactile graphics samples are much less common. A significant obstacle in modeling tactile graphics generation using machine learning is the insufficient number of publicly available samples, as the tactile graphics production industry is less prevalent compared to the traditional one.
Nevertheless, a collection of plant and animal images stored in the APH Tactile Graphics Library [28] was chosen as the original set of images for the model to learn to reproduce. Additionally, the training dataset was expanded with 41 custom tactile image samples, increasing the total number of samples to 179. The custom samples are formed from simple images of animals and were used at one of the enterprises of Ukraine, which provides preschool education for children with visual impairments. The results of the experiment include samples of generated tactile graphics images based on various types of text prompts, such as monosyllabic prompts, prompts with numerals, and prompts with epithets. These samples are presented in Figure 2.
"a cow" "Top view of butterfly" "a tree" "three daisies" "a spotted cow" "a butterfly (side view)" "a naked tree" "a leaf" "a dog" "a turkey" "a deer (side view)" The model's performance was evaluated separately for each component: BART and VQ-VAE. The results of this evaluation are presented in Table 3. The Cross-Entropy metric reflects how well the model converts text prompts into appropriate graphic tokens, and Perplexity represents the uncertainty in the model's predictions. Lower values indicate better performance, meaning the model is more confident in its generation process. For tactile graphics, FID measures how similar the generated tactile images are to real ones in the latent space of the model. A lower FID score indicates that the generated tactile graphics are closer to real tactile images in terms of visual and tactile features.
Additionally, the overall performance of the model was evaluated using the CLIP Score metric [29], which reflects the model's capability in converting textual information into graphical information. The average CLIP Score of the developed model is 23,7.
The current dataset used for training includes relatively simple images (e.g., animals, plants, basic objects). One limitation of the model is its potential difficulty in scaling to more complex images, such as those with intricate details (e.g., architectural blueprints, detailed scientific diagrams). The model's ability to capture fine details may be limited by the size of the latent space and the number of hidden layers used in the VQ-VAE model. Complex tactile graphics might require a more finegrained representation, which could lead to inefficiencies or inaccuracies in generation if the model architecture remains unchanged.
Besides, while the model performs well on simpler prompts (e.g., "a cow," "a tree"), more complex and nuanced prompts (e.g., "a group of children playing soccer with a spotted ball") might pose challenges. This is because the Transformer's encoding of textual information becomes more demanding as the semantic richness and length of the prompt increase. The model may struggle to disentangle and appropriately represent all components of a complex scene in tactile graphics form, leading to loss of information or oversimplification.
Regarding computational requirements, the training process of the proposed model, which integrates both the BART Transformer and the VQ-VAE, requires significant computational resources. Due to the autoregressive nature of the model and the need to process both textual and graphical latent spaces, training is computationally expensive. It requires powerful GPUs or TPUs, large memory capacity, and extended training time, particularly as the dataset grows. This makes scaling to larger datasets or higher-dimensional image outputs challenging without access to advanced computing infrastructure.
One of the key ethical concerns in the development of tactile graphics is ensuring that the generated images do not misrepresent the information. For visually impaired users, the tactile graphic is a primary means of understanding visual content, and any distortion or inaccuracy could lead to misunderstandings. For example, if a generated tactile graphic oversimplifies or omits important details, users might receive an incomplete or misleading representation of the intended information. To mitigate this risk, it's important to validate the model outputs rigorously against established standards for tactile graphics and seek feedback from visually impaired users to ensure that the tactile representations are both accurate and understandable.
As a result of this research, a text-conditional tactile graphics generation model was developed using BART and VQ-VAE. The model employs a modified organization of the latent space, divided into two independent components: textual and graphic.
The method of creating tactile graphics for publications aimed at individuals with visual impairments has been improved. This enhancement increases the variability, controllability, and quality of synthesized tactile graphics, thereby improving the technical and economic aspects of the production process.
This technology can bridge the gap in access to educational materials, allowing visually impaired individuals to better engage with subjects that rely heavily on visual content, such as science, mathematics, and geography. The availability of automated tactile graphics can facilitate greater independence in learning and enhance participation in inclusive classrooms and professional environments.
An important direction of further research is to increase the size and diversity of the training sample to improve the general ability of the model to generalize and ensure its stable operation in various scenarios.