Many blind and low-vision individuals are unable to access digital graphics visually. Currently, the solution to this accessibility problem is to produce text descriptions of visual graphics, which are then translated via text-to-speech screen reader technology. However, if a text description can accurately convey the meaning intended by an author of a visualization, then why did the author create the visualization in the first place? This essay critically examines this problem by comparing the so-called graphic-linguistic distinction to similar distinctions between the properties of sound and speech. It also presents a provisional model for identifying visual properties of graphics that are not conveyed via text-tospeech translations, with the goal of informing the design of more effective sonic translations of visual graphics.
Consider the experience of a blind or low-vision individual who uses a screen reader to access pictures, diagrams, charts, and graphs. Unlike a user who accesses graphical media through visual perception, the screen reader user usually accesses these graphics via text-to-speech “descriptions,” essentially interpretations of what was deemed most relevant by the person who produced the text descriptions of the author’s intended meaning. For example, Figure 1a presents a financial chart with rising and falling stock prices over time, where time is shown on the horizontal axis and monetary value is shown on the vertical axis. Figure 1d presents a text description of the chart compliant with the Web Content Accessibility Guidelines (WCAG), using text to describe the rising and falling monetary values over time. The next sections compare and contrast how these presentations are experienced.
In a text description of a visual graphic (Figure 1d), all of the information is conveyed via text (or text-to-speech, when conveyed via screen reader technology). But in the original chart (Figure 1a), only some of the information is conveyed via text, predominantly numerical values and labels (Figure 1c); the shape of the shaded contour (Figure 1b) is not conveyed via text: the visually perceived shapes are picked up “more directly” and the features of shapes are translated to text descriptions. However, important properties of visually perceived shape information (Figure 1b) are lost in translation and are instead conveyed via text (Figure 1e). This shape information is needed to provide the unique affordances that are often associated with “visual” representations relative to text.
Many scholars have explored the differences between
graphics and text, often referred to as the so-called
“graphic–linguistic distinction”
The graphic–linguistic distinction has been described in
various ways: analogical versus Fregean; analog versus
propositional; graphical versus sentential; and
diagrammatical versus linguistic
Sonic sentential properties. Text-to-speech (the current standard for WCAG accessibility) would seem to be the obvious candidate for the sonic version of what Larkin and Simon referred to as a sentential structure, where elements are arranged in a linear sequence. In the case of visually processed written sentences composed of word forms printed on a page, the sequential properties result from the linear arrangement of characters and word forms on the printed surface. In the case of sonic sentential structures, the sequential properties are temporal, presented as a sequence of sounds that are perceptually processed as words that refer to intended meanings. Larkin and Simon did not define what the elements (that are arranged in sequence) are composed of. For the purpose of this subsection, let us assume that the elements are some combination of properties that, when sequentially processed as words, refer to intended items.
Sonic diagrammatic properties. To present diagrammatic
properties in a way that can be perceived aurally, designers
would need to exploit properties of sound that can convey
topological and geometric relations. People use stereo, echo,
and the Doppler effect to determine the spatial locations of
sound-producing objects in physical environments
According to
Graphical relations could be conveyed sonically. Consider two tones with different pitches: Tone A and Tone B (Figure 2, right). If Tone A is at a lower frequency than Tone B, then the sonic relation between the two tones is the perceptible difference in pitch between the tones. For example, if Tone A refers to a stock price at an earlier point in time, and Tone B refers to a stock price at a later point in time, then the perceptible difference between the pitches of the tones can convey the difference in price over time. Moving the sonic cursor from left to right woul dPitch correspond to a change (increase) in pitch, conveying the cStehreoange in stock price over time via a sonic relation.
“A is lower than B and B is to the right of A” Pitch
A
B
D
E
C Stereo
Cursor Moves to Right
Pitch Increases
The classic distinction between analog versus digital, where
analog refers to visual properties of a graphic and digital
refers to linguistic properties, is most commonly associated
with
The analog versus digital distinction appears to involve two
interrelated capabilities: lower-level perceptual capabilities
to process geometric and topological properties (e.g., those
shown on the speedometer dial); and higher-level
capabilities to process, filter, and interpret how those
perceptually processed features fall into conceptual
categories (e.g., the numerically represented velocity)
For brevity, the following discussion will use the classic
characterization provided by
Diagrams are physical situations. They must be, since we can see them. As such, they obey their own set of constraints . . . By choosing a representational scheme appropriately, so that the constraints on the diagrams have a good match with the constraints on the described situation, the diagram can generate a lot of information that the user never need infer. Rather, the user can simply read off facts from the diagram as needed. This situation is in stark contrast to sentential inference, where even the most trivial consequence needs to be inferred explicitly.
To illustrate how “diagrams are physical situations,” consider the illustration shown in Figure 2 (left). A text (or text-to-speech) description might go as follows: “A is below B and both A and B are to the left of C.” Another textual description might read: “B is between A and C and is above both A and C.” Each text description conveys a different interpretation of what is shown visually and therefore affords different inferences. In contrast, a diagram can convey many other relationships because of how it conveys topological and geometric information through visual perception: Barwise and Etchemendy referred to this as a diagram’s ability to present “countless facts.”
When
Thus, a hybrid stereo–varying frequency interface (see Figure 3) should enable a user to “hear the shape” of a contour. Indexing text-to-speech labels to contours should allow users to form multiple sentences (countless facts) about the geometric and/or topological relations among the labeled elements.
Let us now extend on the various graphic–linguistic distinctions to consider sonic versions of visual charts and graphs.
1. Extending on the diagrammatic versus sentential distinction, text-to-speech can be considered a sonic version of what Larkin and Simon referred to as a sentential structure and is the current WCAG approach to web accessibility. In contrast, spatial sound can be exploited to convey 2D sonic diagrammatic external representations.
2. Extending on the analog versus digital distinction, textto-speech uses language to convey digital properties sonically. The analog properties of sound, such as tone, timbre, stereo, and echo could afford the communication of spatial, geometric, or topological information.
3. Extending on the distinction between relation symbols and object symbols, the current text-to-speech approach uses words to convey relations. Because relations among elements represented by analog and spatial properties of sound are themselves relations, analog and spatial properties of sound could be recruited to map numerical values to perceptual dimensions.
4. Extending on the distinction between intrinsic and extrinsic constraints, producing sonic versions of visual graphics would require identifying “physical situations” that naturally emerge during human perceptual processing of sound to present “countless facts.”
This section integrates these extensions and proposes how the graphic–linguistic distinction could be extended to sonic external representations. First, let us recruit and expand on the distinction between lower-level perceptually processed topological and geometric features of an environment versus the recognition, categorization, and linguistic communication of those features.
Visual and aural sentential structures and relations are detected and perceptually processed via lower-level sensory receptors and perceptual categories (Figure 3, left). In written text or text-to-speech, what is most relevant is the higher-level conceptual category (Figure 3, right) that a given feature (such as perceptually processed printed text on a page or text-to-speech) is taken to fall under. What is needed is a way to convey topological and geometric relations among elements by exploiting lower-level perceptually processed features of a visual graphic or sonic structure (Figure 3, left). Let us refer to these perceptually processed features as perceptual properties. Let us refer to these perceptually processed relations among elements as perceptual relations. Let us refer to relations that are communicated via text as text-described relations.
We are now ready to build on previous work by
The model is based on the idea that an individual’s
perception–reaction loop
These simulations are constructed from the memory
traces of past perception–reactions (conjunctive neurons), so
simulation involves many of the same neural systems used
during perception
At lower-level association areas, which are more tightly
coupled with sensory receptors, simulated prototypes fall
under perceptual categories. At higher-level association
areas (see Figure 3, right), conjunctive neurons converge in
zones across multiple sensory modes. These “convergence
zones”
Similarly, a child can learn to associate sounds with certain intended meanings (learning a language), or to associate marks with intended meaning (learning to read). The abstract concept of ‘square’ can apply to a shape on a raised surface that is touched but not seen, as well as to a drawing on a piece of paper that is seen and not touched. These “less modally specific” simulations have been described as more “interpreted” or “conceptual,” while more perceptually based simulations are considered to be more “concrete.” The next section applies this interpretation to external graphic representation.
Back to charts and graphs. In a financial chart (and many other kinds of diagrams), relations are conveyed via lower-level perceptual processing of the geometrical and topological properties of the marked physical surface (Table 1). In contrast, in text descriptions (sentential structures), relations are conceptual (and conveyed linguistically; see Table 2); although visual properties of printed text or aural properties of text-to-speech are also picked up by sensory receptors, what is meaningful about them is the conceptual relation that is conveyed linguistically. The idea of “specificity” is central to understanding what is lost in translation, so let us begin by clarifying what is meant by “more or less specific” in this context. Consider the line shown in Figure 4b. Relative to the line of Figure 4c, we have more knowledge about the location of a point in a one-dimensional space, due to the shaded red marker. This means we have more certainty (or more information) about the specified location of the point in Figure 4b than we do about the location of the point in Figure 4c.
Extending the line example to discuss perceptual relations, Figure 4b refers to intentionally configured marks or sounds from an author to cause intended audience percepts (the diagram in Figure 4a). However, the perceptual relations of Figure 4a can be processed, filtered, and interpreted to fall under a range of possible relational categories (that can be text-described), indicated by the highlighted segment of the right line in Figure 4c (as shown in Figure 4d: “A is below B and both A and B are to the left of C” or “B is between A and C and is above both A and C”). In other words, although perceptual specificity is high, conceptual specificity of the intended relation is low because the perceptual relations can fall under numerous conceptual categories. However, the reverse is also true and this reversal exposes the heart of what is lost during the translation process.
Extending the line example to discuss the perceptual ambiguity of text-described (conceptual) relations, the right highlighted line in Figure 5c refers to a specific (sentential) text description authored to convey intended conceptual relations (Figure 5d). However, numerous perceptual relations (Figure 5a) can fall under the text-described conceptual relations, indicated by the highlighted segment of the left line in Figure 5b. In other words, although conceptual specificity is high, perceptual specificity of the intended relations is low, because numerous perceptual relations can fall under the text-described conceptual relations.
Let us now return to the WCAG text description example from Figure 1 in order to demonstrate what is lost in translation and how what is lost could be conveyed via nonlinguistic sound. In the text description (Figure 1d), the problem is that all content is conveyed conceptually (via text-to-speech) whereas the original visual graphic that the text description is based on conveys much of the content (the contour of the shape) perceptually: Perceptual relations are lost and replaced by conceptual relations, generating perceptual ambiguity. If the objective is to present Figure 1a sonically, how can a designer decide which aspects should be conveyed via conceptual properties (text-to-speech) and which aspects should be conveyed via perceptual sonic properties (such as spatial sound)?
Recall the perceptual distinction, where perceptual properties are predicted to afford the communication of concrete structures more effectively compared with conceptual properties, and an aspect of a graphic can be identified as “more concrete” if it produces a perceptual structure that corresponds to what could be picked up and perceptually processed from a physical environment. In this account, the graphically represented shape contour (Figure 1b) is primarily perceptual, and is therefore more appropriate for translation to sonic properties that can use spatial sound to convey geometric and topological relations among conceptually conveyed objects.
To determine which aspects of a graphic should be conveyed via text-to-speech, recall the conceptual distinction: text is predicted to afford the communication of abstract conceptual categories more effectively compared with perceptual properties, and a concept can be identified as more abstract if it is more amodal. In other words, it is less easily mapped back to a structure that could be picked up and perceptually processed from a physical environment. Under this account, the numbers that label increments on the x and y axes (Figure 1a) are more conceptual because they cannot be mapped back to a perceptual structure that could be picked up from a physical environment.
This essay proposes a provisional model to underpin the various accounts of the graphic–linguistic distinction described in the literature as a means to extend the graphic– linguistic distinction into aural domains. The model makes the distinction in terms of lower level perceptual capabilities that enable perceivers to perceptually process concrete structures (e.g., geometric and topological features) on the one hand, and higher level capabilities that enable perceivers to process and interpret how those perceptually processed structures fall under more abstract conceptual categories on the other.
Due to these distinctions, the model predicts that perceptual relations (conveyed via graphics or non-linguistic sonification) afford the communication of concrete relations (conveyed via text or text-to-speech) more effectively compared to conceptual relations conveyed via text or textto-speech. In addition, the model predicts that conceptual relations (conveyed via text or text-to-speech) afford the communication of abstract relations more effectively compared to perceptual relations conveyed via graphics or non-linguistic sonification. This could be tested, for example, by observing whether perceivers can identify visual data sets more accurately using sonification or text descriptions.
In addition, the model streamlines accounts that distinguish diagrammatic from sentential structures to (1) characterize sentential structures as composed of conceptual relations among conceptual objects on the one hand, and (2) diagrammatic structures as perceptually represented relations among conceptual objects on the other. Under this account, (3) a sonic diagram is conceptualized as sonically conveyed relations among linguistically conveyed (via text-to-speech) objects.
This model is useful within a design context because designers lack clear models or guidelines for converting visual graphics into non-visual perceptual modes. This can be seen in the WCAG text description example, which ignores the pictorial properties of graphics.
By reverse engineering the classic graphic–linguistic distinction to more fundamental perceptual principles, this model provides a way to understand how the distinction applies to sonic representations. This approach can also be applied to haptic representations but the focus of this paper was on sound for its ubiquity in the consumer market.
This research was supported in part by grants from the Centre for Innovation in Data-Driven Design and the Graphics Animation and New Media Centre for Excellence. I would like to thank Research Assistant Ambrose Li for his assistance in the preparation of this essay and Dr. David Steinman for the many fruitful conversations that helped inform the ideas explored in the work described here.