Sign languages are natural languages existing in the visual-gestural modality used by the deaf communities. Previous research found that literary texts in sign languages use a number of features that distinguish them from non-literary texts, including phonetic features such as the hand movement amplitude, symmetry and enhanced use of nonmanual markers (facial expressions and body movements). However, this type of research has been largely based on manual annotation and observation. We use Computer Vision (MediaPipe Holistic) to automatically extract measurements from video recordings in order to quantify such phonetic features. The data set comprises of 2D video recordings of five short literary pieces in Lithuanian Sign Language that were recorded by their authors (originals) and then retold by the same authors in a non-literary form (as “prose” retellings). We extract landmark coordinates from the videos and measure the following features: variation in hand movement amplitude; symmetry as a comparative measure of activity of the two hands; sideward body leans; eyebrow movement. We discuss the steps necessary to use the measurements, namely filtering and normalization by body size. We compare the original literary pieces with their retelling according to these features. We find that the original literary pieces clearly have more/larger hand movements, larger sideward body leans, and more use of eyebrow movement. However, we do not find clear symmetry diferences between the originals and the retellings. The study is a proof of concept for the use of Computer Vision in phonetic analysis of sign languages, also in the context of literary analysis.
Sign languages are natural languages existing in the visual-gestural modality used by the deaf
communities. Being natural languages, they are used in all possible domains where human
languages can be used, including for creation of literature [
According to sign language literature researchers R. Sutton-Spence and M. Kaneko [
The first studies in the field did not aim to compare oral/written literature and sign language
literature, but focused on the following questions instead: how do deaf artists convey rhyme
(deaf artists use the repetition of handshape, movement and space) or metaphors and other
means of expression; and what tools exist only in sign languages and cannot be found in
oral/written artistic texts [
In recent years, substantial advances in Computer Vision have happened, and reliable landmark
detection and tracking for human bodies in 2D videos has been achieved. A major advance of
this field was OpenPose [
Since sign languages exist in the visual modality, and the primary data for sign languages is
represented in video recordings (or video streams), it is natural that CV has been extensively
applied to sign language research. More specifically, a large part of work in Automatic Sign
Language Recognition and Translation uses CV solutions as part of the pipeline [
However, a few researchers have also tested the applicability of CV solutions and applied
them to linguistic analysis of signed languages, see a brief overview in [
In the current paper, we combine insights from linguistic research on sign language literature with the advances in using Computer Vision for sign language analysis. We provide a proofof-concept study showing that it is possible to measure phonetic properties in recordings of literary pieces using a Computer Vision tool, and compare literary and non-literary renditions of the same texts based on these measurable properties.
The current dataset contains five literary pieces in Lithuanian Sign Language and their nonliterary (prose) retellings produces by the same signers. Figure 1 illustrates the same event (drinking) being described in the original literary piece and in the retelling. One can see that diferent signs are used, but also that the body posture and facial expressions are diferent.
Lithuanian Sign Language (LSL) is the language of the deaf community in Lithiania. The study into LSL began at the Surdology Centre in 1996, following the declaration by the Lithuanian Government of LSL as the native language of the deaf a year before. The Centre’s team collected LSL lexicon, studied LSL grammar, and embarked on LSL dictionaries (print versions). Currently, the bulk of the work in Lithuania is devoted to the digital LSL dictionary. LSL literature research began in 2018.
The literary pieces analyzed in the current study are the following:
2. “Sign Language” (Gestų kalba)1 by Nijolė Karmazienė 3. “Time is running” (Bėgantis laikas) by Nijolė Karmazienė 4. “Being Yourself” (Būti savimi) by Karolis Pilipavičius 5. ”Sign Language” (Gestų kalba) by Karolis Pilipavičius
All the video recordings were re-scaled to 853x480 resolution. 1In the graphs it is represented by the older title Kurčias žmogus ”Deaf person”.
In order to extract landmark coordinates, we used MediaPipe Holistic [
The individual .csv files from the pose component and from the face component were
combined into single files (with only the relevant landmarks) using the tidyverse package [
The pose component outputs the following features: frame number, video height and width, landmark number, the x, y (on the scale 0 to 1) and z coordinates, and visibility (on the scale 0 to 1). The face component outputs the same features except for visibility.
Exploring the data, we discovered that, despite the fact that the hand coordinates should be between 1 and 0 for both the x and y coordinates, the model actually outputs values beyond this interval, especially so for the y-coordinate (between -0.6 and 1.2 in our data). This correlates with visibility: visibility is somewhat low for out of bound coordinates (below 0.5). Whatever the explanation, it is clear that MediaPipe outputs erroneous coordinates for the hand landmarks sometimes, and they cannot be used. We therefore filter out all the measurements that go outside of the frame boundaries, and also all the measurements with visibility below 0.5. We also discovered that shoulder coordinates are not always detected clearly, with extreme outlier measurements infrequently occurring in the absence of any body movements in some videos. Therefore, to filter out such cases, we remove any measurements that are not within +-3SD from the mean per landmark per video.
The original pose dataset contains 101175 lines (20235 frames * 5 relevant body landmarks), and the face dataset contains 57321 lines (19107 frames * 3 relevant face landmarks).3 After ifltering, the pose dataset contains 98474 lines, so we remove 2.7% of the data, and the face dataset does not change as it does not contain extreme outliers. The small reduction in the size of the pose dataset should not be considered a comment on the quality of MediaPipe output. In fact, most of the data we filter out are simply frames where the hands are above or below the frame boundaries, and so the measurements simply cannot be made.
The next step that we need to take is normalization. Because the coordinates depend on the
size of the body of the person in the video recording, as well as on the distance between the
person and the camera, in order to compare across diferent videos, normalization is necessary.
We use the procedure and the code recently proposed by Börstell [
As discussed in the previous sections, we decided to focus on some phonetic aspects of sign language production that are potentially linked to literary texts, and can be measured using Computer Vision tools at our disposal. Therefore, we conducted measurements of general hand activity, of symmetry, of body leans, and of eyebrow movements. For each of the measures, we compare the individual original pieces and their retellings.
To measure hand activity, we explore separately the x- and y-coordinates of the two hands. Note
that the z-coordinates in the MediaPipe pose output (unlike the ones in the face component) have
not been tested or calibrated by the creators, so they are not useful for phonetic analysis. This
means that we are not measuring actual hand position in the 3D space, but only its projection
to the 2D plane of the video recording. While this is a substantial drawback in general because
all information along the z-scale is lost, we think that it is acceptable for the purposes of this
paper. First, the signers in the videos all stand facing the camera in each video and so the
loss of information at least is expected to be similar across the videos.4 Second, because we
compare individual recordings to each other, the x- and y-coordinates already provide very
large numbers of data points that can be analyzed. Still, clearly, if it happens that literary pieces
and prose retellings are diferent in how far the hands move along the z-axis specifically, we
would not be able to detect this diference using the current method (see also [
In Figure 2, the x-coordinates (normalized and centered) are plotted for each piece and its retelling as boxplots. We can see that, for each piece, the original has a wider distribution for the coordinates. This means that the hands move further to the right and to the left in the original than in the retelling. In other words, the horizontal amplitude of the hand movements is larger in the original.
Because the diferences are not in the mean position of the hand, but in the distribution, in Figure 3 we plot the Standard Deviations of the x-coordinates for each piece for the two hands. Here we can also clearly see that the SDs are consistently larger in the original than in the retelling. The only case where there is almost no diference is left hand for the last piece. The left hand does not in fact move much in either the original or the retelling.
We have tested whether the diference in SDs between the original and the retelling is significant using Levene’s test for equality of variances, and found that it is significant overall and for each individual piece for both hands, except for the single case described above. Note, however, that significance testing is not very meaningful because the datasets contain very 4Of course, the signers use body and head turns for linguistic and literary purposes, so they are not always facing the camera directly. many points, so even very small diferences easily reach significance. We suggest that graphical exploration is more meaningful here.
In Figure 4, the y-coordinates (normalized and centered) are plotted for each piece and its retelling. We can again see that, for each piece, the original has a wider distribution for the coordinates. This means that the hands move further up and down in the original than in the retelling. In other words, the vertical amplitude of the hand movements is larger in the original.
Figure 5 plots the SDs of the y-coordinates across the pieces. Again, we can clearly see that in each case the SD is higher in the original than in the retelling, even though in some pieces the diferences are pretty small (albeit statistically significant).
To sum up, it is very clear that, in our dataset, the hands move across a larger space in the originals, both horizontally and vertically. In the retellings, the hand coordinates show less variation and smaller extreme values.
As we discussed above, symmetry can be used in literary texts in sign languages as a literary
device. In order to quantify symmetry, we follow the insights (and the specific measure)
proposed by Börstell [
For our purposes, we calculate this measure for each frame, and compare the averages and distributions between the original and retelling of each piece. The results (mean symmetry measures) are represented on Figure 6.
We can see that symmetry is not used consistently to contrast originals and retellings. In one piece, the symmetry measure is much higher in the original, which means that there is much less symmetry in the original than in the retelling. Of the other four pieces, two have higher symmetry and two have lower symmetry in the originals, but the diferences are small.
The results thus indicate that symmetry is not used as a literary device in the pieces that we analyzed. This agrees with our qualitative of the data: we were not able to see marked symmetry used for literary purposes in these pieces.
Not only the hands, but also the body can move more in literary texts (recall also Figure 1). We decided to specifically measure sideward body leans by looking at the distributions of the x-coordinates (normalized and centered) of the two shoulders. As with the hand coordinates, the caveat is that we do not have measurements along the z-scale. In this case we also do not consider the y-coordinates because we conceptualize body leans as primarily sideward movement of the shoulders. While this is reasonable for our dataset where the signers face the camera, it is clearly not applicable to videos filmed at a diferent angle, and so other measures should be used.
The resulting distributions of the x-coordinates of the shoulders can be seen in Figure 7. It is clear that, in four of the five pieces, both the left and the right shoulder move to much more extreme points in the original as compared to the retelling. In one piece (the last piece on the Figure), the diference is small, and it is in the opposite directions: the shoulders move more in the retelling. This agrees with our observations of that recording: the signer moves their body more in the retelling than in the original, where the body is very still. Figure 8 represents the same data by comparing SDs of the horizontal shoudler coordinates between the original and the retelling.
To sum up, in four of the five pieces we indeed find a much more pronounced use of sideward body leans, as represented by the more extreme and varied coordinates of the two shoulders. However, even in our small dataset, in one of the literary pieces the signer barely moves their body, so large body movements are not a universal feature of literary texts.
We also hypothesize that more pronounced facial expressions can be used as literary devices. Because facial expressions are complex, for this pilot study we only focus on eyebrow movement. We operationalize eyebrow position as a measure of Euclidean distance between an eyebrow landmark and the landmark on the top of the nose. This time we use the z-coordinates as well because they are a stable output of the face component of MediaPipe Holistic.
The distance between an eyebrow point and the top of the nose is a rough measure because,
as has been shown in much previous research also for signed languages [
Looking at the results, we can see in Figure 9 that, for the most part, the mean eyebrow
distance is larger in the original, and also that the variation in the original is clearly larger. The
diferences are less pronounced in the left eyebrow compared to the right eyebrow, which is not
surprising as eyebrows often move asymmetrically [
As for the hands and body movements, we think that the relevant feature to focus on is variation and not the mean positions because variation reflects how much the eyebrows move, not where they are on average. We can see in Figure 10 that, in all the pieces, SDs of the eyebrow distance measures are larger in the original compared to the retelling. In one piece, the diferences are tiny, but still in the same direction.
To sum up, we find evidence that, in our dataset, facial expressions as manifested by eyebrow movements are used more actively in the original literary pieces. However, future work should include more detailed measures of the eyebrow position, and also of other facial articulators.
In this paper, we presented a proof of concept for the use of Computer Vision (specifically,
MediaPipe Holistic) for the analysis of literary texts in a signed language. We have demonstrated
that some phonetic features that are used in literary devices can be measured using CV outputs,
and these measurements can be meaningfully compared across genres. Specifically, using a
dataset of five literary pieces in Lithuanian Sign Language and their prose retellings by the
same authors, we found that the originals contain more hand activity (larger and more varied
hand movements), larger and more varied lateral movements of the body (as measured by the
shoulder position), and more eyebrow movement. At the same time, we did not find clear
diferences in symmetry between the originals and the retellings, even though symmetry can
sometimes be used as a literary device [
There is a number of limitations to the use of Computer Vision for analysis of sign language
data. First, MediaPipe Holistic and other CV solutions have errors in their outputs (see also [
Second, at least for the pose component, MediPipe Holistic does not provide reliable 3D
reconstruction. Because sign languages employ the 3D space for many linguistic purposes
[
Third, the measures that we developed to compare the literary texts and their retellings
are quire rough measures, and there further improvements are clearly possible. One direction
for future improvement is segmenting the videos into smaller pieces, such as utterances or
individual signs. Segmentation into individual signs can potentially be done automatically
[
Despite these limitations, our study clearly shows that CV solutions such as MediaPipe can clearly be used to measure and quantify certain phonetic features in sign languages, and these features can be used further to answer a wide variety of research question in linguistic, and even, as in this case, literary analysis.
Funded by the European Union (ERC, NONMANUAL, project number 101039378). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
The authors would like to thank Carl Börstell for a helpful discussion of the analysis. A. Content summary of the literary pieces 1. “Black Orchid” by Paulius Jurjonas (2018). The poetry is about love. The piece is about a man having a flashback upon seeing a motorcycle. In the flashback, the protagonist gives a ride to a long-haired heart-breaker (femme fatal) woman with thick pufy lips. The flashback ends; in the present, the heart of the protagonist is portrayed as black and dead.
2. “Sign language” by Nijolė Karmazienė (2015). The poet tells the story of a powerful tree that has grown from a small seed. The tree is swinging in the wind. In the distance, on the top of a rock, there is a nest with a hawk watching the tree. The hawk is strong and powerful. The bird flies down toward the tree, circles around it and finally alights on a branch. The hawk admires the tree and distinguishes it from among all other trees. Finally, the bird returns to the nest. The work shows that both the tree and the bird feel to have gained strength, bliss and joy from the encounter. The poet employs metaphors: the tree stands for the deaf communities and sign language. The hawk is a metaphor of highly qualified sign language interpreters.
3. Nijolė Karmazienė in poem “Time is running” tells a story about a watch. At first the watch works great, then it breaks. The poet metaphorically represents the eyelid of a human eye, which opens, and tears are visible in it, which soon pour down and wash the watch‘s gears. After that the watch is ticking again. And since the text is metaphorical, it talks about human health through the prism of a watch, the poet talks about deteriorating well-being, illnesses, and death. And vice versa, when the mechanism starts to work, the hands start to go (up) – the poet talks about improving well-being, health, and life.
4. Poet Karolis Pilipavičius in “Being Yourself” portrays a man tightly holding the folds of his jacket against his chest. People go past the man one by one, touching him as if wishing to see what is behind the jacket. The poet creates a metaphor: the jacket is a man’s identity, his inner world. At a certain point, a passer-by touches the protagonist and his body is revealed. Ashamed, the man wants to hide, but he decides to come out. A multi-coloured rainbow is seen on his chest. This is also a metaphor. Of course, one of the interpretations is that the rainbow is multi-coloured, so it is a sign of diversity and otherness. However, another interpretation is that the rainbow is a symbol of the LGBT+ community. Ultimately, the male protagonist in the work sets an example for others by encouraging them to come out and be open with the world, themselves and other people. It is a sign of community, freedom and calmness.
5. “Sign Language” by Karolis Pilipavičius. The poet tells the story about a deaf tourist who, in a happy mood to get to know a new city, faces challenges: when he asks for directions in sign language, he gets no help, when he asks what time it is, he only sees people scrambling. The protagonist is confused by this, he starts to wonder why he is so diferent from other tourists? Is it because he speaks sign language? It breaks his heart. Another person approaches the protagonist and speaks with him in sign language. Soon, many sign language speakers gather around the deaf tourist to help him. He feels happy, understood, accepted and safe.