One of the most pressing concerns in the field of adapted printed publications - that is, publications with additional supporting features to make them easily accessible by a wide variety of audiences is preparing illustrations that can clearly convey visual information to the viewer. These illustrations need to be created while accounting for the needs of a diverse inclusive audience, whose requirements may be affected by disabilities such as visual impairment. Currently, there is a limited number of illustrators who can appropriately produce a large number of illustrations which satisfy these requirements; therefore, illustrating print publications is a time-consuming and expensive process for non-profit organizations which are responsible for their production. This article proposes a method for enhancing illustrations within print publications, given the source file (such as a PDF file) of a print publication. Based on modern text-to-image generators, this method extracts all illustrations from a print publication; converts them into textual prompts for a modern text-to-image generator; and finally, produces a series of adapted alternatives for each of the chosen illustrations based on the textual prompts. This allows publishers to obtain accessible illustrations for their publication in a manner of minutes, speeding up the adaptation process and enhancing its accessibility.
In recent years, assistive technologies have enabled diverse audiences of readers to freely access
written information, allowing them to work, study, and participate in civil society. One of the
most essential tools for this category of readers is accessible print publications. While people
with disabilities, such as visual impairment, often rely on e-readers; these may not always be
available or preferred. A recent study suggests that 65% of surveyed Americans have recently
read a print book, while only 30% have read an e-book, indicating that print books still remain
the most popular format for general readers [
Incorrectly adapted printed books can pose various issues for readers with visual
impairment, such as small text size, poor font styling, insufficient contrast, graphics and tables
as well as other design factors. Unlike e-books, where these parameters can be custom-tailored
to the needs of a particular reader, printed publications are inflexible in their design,
exacerbating these issues [
Existing studies in the field most often focus on e-book accessibility. Since e-book formatting is
not rigid and preset by the publisher, but rather determined by the e-reader, the focus on
ebook accessibility lies primarily in correctly marking important parts of the publication (such
as chapters or specific phrases which link to footnotes), as well as developing robust, flexible
ereader software that allows users to adjust fonts, text sizes, as well as other parameters [
However, print books still have notable advantages, especially in the field of education.
Research suggests that readers tend to understand text slightly better when it’s printed rather
than viewed on-screen [
When it comes to adapting illustrations within print books, existing research focuses on
multimodal illustrations – for instance, tactile illustrations, which combine visual illustrations
with tactile Braille overlays, which is useful for readers with legal blindness [
In order to synthesize new adapted illustrations for a print publication, our method must first
obtain the inputs – often known as “prompts” – for a well-known image generator such as
Stable Diffusion or Midjourney. Although in technical publications, illustrations often come
with captions describing the image, these captions are not always present, and often offer brief
interpretations which do not capture the full nuance of the captioned image. Thus, the images
must be converted to textual prompts using an AI model known as an image captioner. Acting
as the inverse of common image generators, image captioners are a form of feature extraction
models which convert images into text, functioning at the crossroads between computer vision
and natural language processing [
Our chosen image captioner model is the CLIP Interrogator [
Specifically, the model’s captioner is an image-grounded text decoder. Its intent is to generate
synthetic captions given training images collected from web datasets. In the BLIP learning
framework, this is combined with the filter, an image-grounded text encoder which removes
texts that are predicted to not match a given image – this is applied to both synthetic captions
and real captions found inside training datasets. This is combined with a set {( ℎ, ℎ)} of
human-annotated images and texts to produce a robust training dataset for a ML algorithm [
Our method works primarily with this model’s captioner. We opt to use the BLIP-ImageCaptioning-Large sub-model, pre-trained on the COCO dataset to produce human-readable captions for input images. However, human-readable captions are not the most effective way to produce the inputs for an image generator. An image generator’s input text, known as a “prompt”, needs to be detailed and describe multiple keywords pertaining to an image: the subject of the image itself (which is typically produced by an image captioner out of the box), as well as the style, resolution, color, lightning and other details.
To that end, we use the CLIP Interrogator to first generate a baseline caption using BLIP, and then simplify the caption while adding additional keywords which most closely match the target image. These come from a predefined dataset known as “flavors”, and include keywords such as “highly detailed”, “sharp focus”, “intricate”, “digital painting” as well as phrases referring to specific objects and entities located within an image. The Interrogator selects the most appropriate keywords and phrases from “flavors” dataset by measuring the distance between a target image and each separate phrase. (1)
After the images within an adapted print publication are converted to text prompts, the next
step is to transfer them to an AI image generator such as Stable Diffusion to obtain a new set of
images designed with accessibility in mind. Since the CLIP Interrogator’s prompt generator has
been designed with Stable Diffusion in mind, its newest stable version, SDXL v 1.0, has been
selected as the image generator of choice [
This image generator is a latent diffusion model, an improvement on traditional diffusion models. Traditional diffusion models work by first “corrupting” training data, such as images, by adding noise to their inputs in a step-by-step-process. At each time step, Gaussian noise is added to a data distribution 0 ~ ( 0) with variance ∈ (0, 1), resulting in the iterative process over the distribution of the variable: ( 1, … , | ) =
( | −1) ( | −1) = ( ; 1 − −1, )
This process is called forward diffusion, and concludes once the distribution is sufficiently similar to pure Gaussian noise.
Reverse diffusion is the process of recovering the original image from the resulting noise. The overall workflow of diffusion models after they have been trained is to generate new images, given random noise as input. Latent diffusion models perform this process within latent space – a mathematical representation of data where similar items are grouped.
Aside from a shortened version of the CLIP Interrogator’s keyword-based output, we append several keywords designed at simplifying and matching them to a more clear and simplified style, such as “illustration for children”, “monochrome”, “very low detail” and “no shading”. This ensures that, while the objects and entities denoted by the keywords are included within the final image, it remains simplified without obstructing valuable information by noisy elements. Keywords can be modified or adjusted as needed – for instance, “monochrome” may be removed should we require a full-color illustration.
The complete workflow of our adaptation method can be seen in Figure 1.
In Figure 1, we can see the results of this image-to-text generation model applied to one of the illustrations from a selected Ukrainian adapted textbook. Initially, the caption generated by the model is a verbose human-readable representation; the CLIP Interrogator modifies this to an image generator based on matching keywords from a preset list, which is less human-readable, but more effective as an input for a future image generation step.
The number of keywords within the text prompt can be modified at will by restricting the number of phrases returned within the Interrogator, prioritizing those which most closely match the image. While a single image-to-text conversion is shown on Figure 1, our method is a batch process. This means that after all images have been extracted from a printed publication’s source file, they are converted into textual prompts with no additional human interference, significantly speeding up the captioning process as compared to a human captioner.
An example of image generations, which are the final output of our method, can be seen in Figure 3. Within this example, the leftmost image, also shown separately as Figure 4, is particularly appropriate for accessible publications, as it creates an image in a simplified cartoon style with minimal shading that still keeps its core elements (the girls in the foreground and trees in the background) legible. This can be translated directly into a mixed-format illustration which combines visual elements with Braille-like tactile dots.
The software implementation of our method uses the Python-based Jupyter Notebook environment to integrate several steps of the process: the PyMuPDF library is used to extract images from a printed publication’s source file; the CLIP Interrogator (internally based on the PyTorch library & its torchvision extension) is responsible for extracting prompts for images; while the SDXL 1.0 generator is used to create new illustrated images. On a RTX 4090 GPU, our pipeline takes ~22 seconds to convert an original illustrated image into four generated variants ready to be reviewed by a publication’s editor, meaning that the entirety of a print publication’s illustrations can be regenerated within hours.
This article describes a method for adapting illustrations within the source file print publication, with no or minimal human supervision, that can significantly speed up the process of making a publication more accessible to a wide audience of readers, such as people with vision impairment. This method enables both educational and volunteer organizations to produce high-quality illustrations for an adapted print publication.
Potential areas for further research include improvements to the prompts used within our image generation step – for instance, adding support for multi-colored, yet clean and simplified illustrations. The image generation model itself also has potential for improvement; as AI image generation is a rapidly developing field of research, our machine learning pipeline can be periodically revisited to make use of the newest models and techniques.