=Paper=
{{Paper
|id=Vol-2744/paper83
|storemode=property
|title=Development of a Universal Pictographic Language Keyboard
|pdfUrl=https://ceur-ws.org/Vol-2744/paper83.pdf
|volume=Vol-2744
|authors=Polina Belimova,Saltanat Zhalimova,Alena Dzhumagulova
}}
==Development of a Universal Pictographic Language Keyboard==
https://ceur-ws.org/Vol-2744/paper83.pdf
Development of a Universal Pictographic Language
Keyboard*
Polina Belimova [0000-0001-8581-4924], Saltanat Zhalimova [0000-0002-2208-0529], Alena
Dzhumagulova [0000-0003-2389-2729]
ITMO University, Saint-Petersburg, Russian Federation
belimova.polina95@gmail.com, zhalimovasaltanat@gmail.com,
aledjuna@gmail.com
Abstract. The paradigm of communication between people is changing along
with global changes in the modern world. This process is related to the
integration of technology into everyday life and represents an expansion of the
signs and symbolic environment in the field of communication: more and more
new images are appearing designed to mark the attributes of the modern world.
Pictorial form of communication is a simplified, cost-effective means of
communication. That is why it is widely used in education, rehabilitation,
navigation systems of public spaces, web and human interaction with
cyberphysical systems. However, modern pictographic systems are fragmented
and poorly systematized in a constant race for compliance with the
requirements of the modern world. They are almost not investigated in their
practical application and are limited in their expressive possibilities. This article
is devoted to the description of the initial stage of empirical research of
pictographic language LoCoS and the stages of development of an alternative
keyboard for communication based on it. The conclusion is made about the
possibility of the existence of a universal pictographic language on the basis of
its semantic and technological improvement.
Keywords: Human-computer Interaction, Usability, Communication,
Semantics, Linguistics, Icons, Keyboard, Cyber-physical Systems.
1 Introduction
The idea of a universal language as a means of barrier-free communication has
occurred occasionally throughout almost all of human history. These attempts
concerned not only language grammar and vocabulary (Esperanto, Blissymbolics),
but also logical operations (the mathematical universal language of Leibniz), and even
music (Solresol). None of these cases can be called successful, because these
languages have not gained a sufficient number of followers, and therefore they do not
deserve the status of universality. In other words, they could not compete with
Copyright © 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
*
Supported by the Government of Russian Federation (Grant 08-08)
2 P. Belimova, S. Zhalimova, A.Dzhumagulova
established sign systems that have a great deal of expressive capacity and diversity, as
well as because they required a deep and complex study.
The notion that a universal language should be pictographic (consisting of
schematic subject images) is also not new. On the one hand, such features had ancient
hieroglyphic languages, such as Egyptian and Chinese, as well as cuneiform writing,
runic writing, Mayan writing, etc.. They have complementarity in their basis, and can
be combined with each other, forming new concepts, reflecting the attributes of the
material and spiritual world.
On the other hand, the modern technological world demands universality and
velocity of information transmission, and the pictographic representation is ideally
suited for this purpose. In the XX century pictograms were widely used in the fields
of mechanical engineering, major international events (Olympiads), graphic computer
interfaces. This cultural phenomenon of industrial sign environment was the impulse
for further development of local symbolic systems [1]. The problem of multimodality,
linguistics and social semiotics of sign systems in the XXI century is being developed
in the context of various image-centric genres studies: magazines, advertising, social
media platforms and others [2]. Most frequently the virtual dialogue of social
networks as a cultural and psychological phenomenon is analyzed, in particular emoji
as a means of emotional expression in the Internet environment [3,4]. At the same
time, such a characteristic of virtual dialogue as occazionality, i.e. the use of
individual-author's means of expression in a specific context, is emphasized [5]. Of
great interest in this area are studies on a Pictograph-to-Text translation system for
people with Intellectual or Developmental Disabilities. For example, WordNet links
and an n-gram language model are proposed [6].
Thus, the usage of non-verbal symbols in everyday activities and in the process of
communication is a characteristic feature of the information society culture. The
number of symbolic semantic groups is increasing, but they are isolated and have no
common grammatical basis, which does not allow them to form a unified linguistic
structure.
Standardization of the sign environment is a priority if we need to have a consistent
idea of a certain phenomenon. This concerns both public spaces and public languages.
The step towards such universalization is made by Yukio Ota©, who suggests the
visual communication language LoCoS in addition to creating universal signs of
public spaces (Fig. 1) [7].
Fig. 1. Emergency exit sign accepted by the International Organization for
Standardization (ISO) in 1987.
Development of a Universal Pictographic Language Keyboard 3
Fig. 2. Examples of LoCoS language symbols.
The language LoCoS was inspired by Blissymbolics (Charles K. Bliss, 1949),
which is quite popular in education and rehabilitation of children with disabilities.
LoCoS is a complimentary language as well as Blissymbolics, i.e. its symbols can be
combined to form other concepts. However, according to the author, LoCoS is more
intuitive, authentic to the modern world and flexible language (Fig. 2).
In 2007, Aaron Markus, a specialist in the field of human-computer interaction,
published a prototype keyboard based on LoCoS language [8]. The prototype
considered the pixel dimensions of symbols, provided the possibility of symbol
combining and composing sentences, and offered a specific information architecture.
However, this prototype was created for a keypad phone and became outdated in a
few years due to the development of mobile technologies and the emergence of
smartphones with different configurations and greater capabilities.
Thus, it seems relevant to find out whether communication using the LoCoS
pictographic language is possible. The conclusion can be drawn from empirical
research conducted in the context of psychological and linguistic paradigms. In
addition, a necessary condition for the study is the development of a LoCoS keyboard
that meets modern technical requirements, since it is possible to study the
communicative behavior of a modern user.
2 LoCoS Communication Capacity Research
The main objective of the pilot empirical study was to understand whether people,
native speakers of Russian, could communicate in LoCoS (to receive and transmit
information using images). It was important to find out how comfortable it was for the
respondents to express their thoughts and to understand others.
The study was attended by 22 students of the ITMO University Faculty of
Software Engineering and Computer Systems, St. Petersburg, Russia. The average
age of the respondents is 21 years, the ratio of men to women was 50:50. None of the
study participants had ever heard of LoCoS before participating in the experiment.
4 P. Belimova, S. Zhalimova, A.Dzhumagulova
After a 15-minute instruction on how to use the pictographic language, each person
was given a detailed description of the language and a set of 252 symbols. The set
included all currently existing LoCoS symbols, both basic and specific.
The respondents were divided into 2 variants and received the handouts according
to their variant. In the first phase, students were offered to translate 5 sentences from
Russian into LoCoS, which were different by syntax and number of symbols. The
level of abstractness and complexity of sentences increased from the first to the last.
In the second phase, students exchanged the results of their translation in pairs, i.e.
variant 1 was changed with variant 2. The respondents had to translate from LoCoS
into Russian. During the process they were allowed to use all the handouts without
any restrictions. The total time of experiment was on average 60 minutes.
After completion of the tasks, the respondents were asked to evaluate writing and
reading comprehensibility on a 10-point scale. They were also asked if they would
use this language in life, and if so, in which cases.
Subsequently, both subjective evaluations and the accuracy of student’s responses
were taken into account when translating from LoCoS into Russian. The degree of
understanding was transformed into points: the meaning is completely translated and
comprehensible (3 points), the meaning is comprehensible but the nuances are not
noticed (2 points), the meaning is partially comprehensible (1 point), the meaning is
not comprehensible (0 points).
The results demonstrate significant difficulties in translating complex abstract
sentences from LoCoS. Sentences consisting of two words, one of which the
respondents were asked to design themselves, also caused difficulties in translation
into Russian (Fig. 3).
Fig. 3. The comprehensibility level of sentences of different complexity written on
LoCoS.
The two-component common sentences with one concept, which respondents were
asked to design using the symbols from the dictionary, were easier to translate than
the previous two sentences (simple with pronouns, exclamations, past or future
simple) and allowed all the respondents to understand at least some part of what was
Development of a Universal Pictographic Language Keyboard 5
written. In some cases, simple sentences with pronouns and expressions (?,!) were
translated incorrectly with a partial translation of meaning. Simple sentences with a
time indication, consisting of terms present in the dictionary, were totally expressed
and understood in 100% of cases.
The following conclusions can be drawn from these results:
1. There is a lack of symbols in the LoCoS language, including frequently used
symbols. Further development of the semantic core of this language is required;
2. Complex abstract concepts are almost impossible to translate exactly
(meaning of life, etc.), as well as unions, pronouns, status categories;
3. Sentences from simple concepts available in the dictionary can be translated
with high precision;
4. The minimum number of symbols in a sentence may prevent the accurate
translation of meaning;
5. A free symbols design and combination reduces the probability of accurate
translation and understanding.
The analysis of subjective assessments of writing on LoCoS and understanding of
what is written has shown that writing is easier for the respondents than reading and
translation into Russian. Most of the respondents rated reading and writing
convenience slightly above average (Fig. 4).
Fig. 4. Subjective evaluations of writing and reading convenience in LoCoS (Mode
shows the value that appears most often).
The respondents suggested areas in which they would use LoCoS. 27% indicated
that it is possible to communicate effectively with foreigners if the participants do not
speak a common language. 23% see the usage of LoCoS for entertainment purposes:
quests, riddles. 23% would communicate with LoCoS in everyday life:
correspondence on the Internet, communication with friends, etc. 27% thought the
language was ineffective and did not specify possible areas of usage.
6 P. Belimova, S. Zhalimova, A.Dzhumagulova
The results of the pilot experiment allow us to conclude that communication in
LoCoS language is possible with minimal training, which suggests the thesis about its
intuitive comprehensibility. However, the semantic core is clearly limited and
requires further development, which means the generation of new symbols that reflect
the everyday concepts used by modern people in ordinary life. In addition, the
complementary effect of language makes it more difficult to understand if there is no
concept in the dictionary. Further research with a larger number of participants may
bring resolution of these problems closer.
3 LoCoS Keyboard Prototype Development
After confirming that communication is possible with the LoCoS language, we
started a keyboard developing. One step towards implementing LoCoS
communication capabilities is to create a keyboard. Technical implementation of this
tool is automatization of some necessary translation processes: incorporating symbols
dictionary, translation into user's native language or any other language, reasonable
information architecture for searching symbols, predictive set of symbols, set of user's
most frequently used symbols. Creating a LoCoS keyboard will make further research
possible on communication capabilities in close to natural user conditions.
There are a number of challenges associated with adapting the LoCoS language to
modern devices. The structure of the language is quite complex, and contains about
3000 signs, which are built from basic characters. This leads to the question of
keyboard architecture, input method and keyboard ergonomics. Non-standard
keyboards have been studied for designing LoCoS keyboard and solving mentioned
problems: keyboards of hieroglyphic languages because LoCoS is similar to Chinese
language in structure, atypical keyboards of natural alphabetic-sound languages, non-
qwerty keyboards and keyboards of alternative languages. As a result, 3 prototypes of
keyboards for modern mobile devices were proposed.
These three prototypes have the same working process but different architecture.
The interface of Prototype 1 includes a keyboard, a line for proposed icons and an
input field. The keypad consists of 34 keys: 32 of them - basic, contain basic symbols
LoCoS, two of them - function keys: backspace, space (Fig. 5).
The interface of the Prototype 2 consists of a keyboard, a line for proposed icons and
a three-line input field: the first line for adverbs, the second contains the base sentence
and the third - adjectives. The keyboard consists of 13 keys, 10 of them are basic,
contain basic LoCoS symbols, three are function keys: backspace, shift, space. Eight
main keys offer a choice of certain symbols with a long press. This number of keys
allows you to reduce the size of the keyboard and place a three-line input field instead
of a single-line (Fig. 6).
In 3 lines field the middle line is the main and text cursor is on the main line by
default. The upper and lower lines are auxiliary. To move the text cursor to the top or
bottom line, you need to "swipe" the shift key up or down respectively. When the
character set is complete, the text cursor automatically returns to the main line.
The interface of Prototype 3 includes a keyboard, a line for proposed icons and a
three-line input field. The keyboard consists of 23 keys, 20 main keys and 3 function
keys: backspace, shift, space. The main keys contain basic symbols of the LoCoS
Development of a Universal Pictographic Language Keyboard 7
language, 10 of them can change the direction when "pile" up, down, right, left and
diagonally. Due to the small number of keys in this interface it is possible to
implement a three-line input field (Fig. 7, 8). The three-line input field works on the
same principle as in Prototype 2.
Fig. 5. Prototype 1. Fig. 6. Prototype2.
Fig. 7. Prototype 3. Selected keys change the direction.
8 P. Belimova, S. Zhalimova, A.Dzhumagulova
Fig. 8. The selection of characters changes direction when the keys are pressed for
a long time.
All three keyboards have the same operating principle. The pictograms are
constructed by a sequence of characters and are separated by a space. All entered
characters appear in the input field, and when you press the spacebar, the character
set is converted into a pictogram that contains the character set. At the same time,
the "proposed icons" field offers users "iconic words" consisting of characters
similar to those they have entered in the input field. When a pictogram is selected
from the offered ones, the pictogram in the input field will be replaced by the
selected one.
During the development of these keyboards numerous questions were raised,
such as technical implementation and convenience of using by people with
disabilities: limited vision, motor disabilities, etc. The solution of these problems
could be found in software implementation of the interface and further user
research.
Also, an experiment was conducted to prove the usage of three-line or one-line
method of entering.
The main goal of the experiment is to check how much the three-line way of
writing sentences simplifies or complicates its reading. The goal is to prove the
usage of three-line or one-line input field in the keyboard interface.
The study involved 8 students from the Faculty of Software Engineering and
Computer Engineering at ITMO University. The average age of respondents is 23
years, the ratio of men to women is 50:50. The experiment participants were given
the guide to using the language and a LoCoS language symbol library. The
respondents were divided into two groups: control and experimental.
After reading the manual on using the LoCoS language, the respondents were
given 2 sentences of varying complexity in the LoCoS language. The first sentence
consisted of 9 characters, and contained the main content and 3 descriptive words
Development of a Universal Pictographic Language Keyboard 9
(adverb of the time, adjective, possessive pronoun). The second sentence consisted
of 7 characters: the main content and 2 descriptive words (adverb of the time,
definitive pronoun). One-line sentences were given to the control group of
respondents, three-line sentences to the experimental group. One-line and three-
line sentences were the same for different groups of respondents.
Participants were asked to translate sentences from LoCoS into Russian. The
time taken to translate and the result of the execution were taken into account: the
meaning of the proposal was conveyed correctly or incorrectly.
The results of the experiment demonstrate a clear advantage in favor of writing
three-line sentences. Separation of characters into main content and descriptive
words made it possible to translate all sentences into Russian faster and without
errors, while in single-line sentences the meaning of what was written was
translated incorrectly into 3 out of 8 sentences. In addition, the respondents spent
20% less time translating three-line sentences, compared with single-line
sentences.
In the future, the main experiment with a larger number of participants is
planned, which will confirm or refute the conclusions made in the pilot experiment
and experiment made on the LoCoS keyboard.
4 Conclusion
The communicative ability of a pictographic language could be realized only
through the joint work of specialists from different fields: psychologists, linguists,
special educators, engineers, programmers, designers, usability researchers. Thus,
further research means interdisciplinary interaction and will be focused on the
development of the semantic core of LoCoS language. This includes specialized
terms, design and creation of the ergonomic LoCoS keyboard, testing of the user
experience. The possibility of adapting LoCoS language to the needs of people with
disabilities in the context of enhancing their communication, learning and
rehabilitation skills is also of great interest.
The results of work on Universal Pictorial Language can be in demand both
in interpersonal communication, and in communication between a person and a
machine. For example, in two-way information transfer between a user and
cyberphysical systems of different properties. In a world of constantly changing and
improving technical conditions, communication is essential to the wellbeing of the
modern individual.
References
1. Korpan L.M.: Utilitarnye sistemy graficheskikh simvolov XX veka i graficheskie
komp’yuternye interfeisy [Utilitarian Systems of Graphic Symbols of the XXth Century
and Graphic Computer Interface]. Kul'tura i tsivilizatsiya [Culture and Сivilization], 7
(2А), pp. 435-448 (2017).
2. Stöckl H., Caple H., Pflaeging J.: Shifts toward Image-Centricity in Contemporary
Multimodal Practices. Routledge. (2019).
10 P. Belimova, S. Zhalimova, A.Dzhumagulova
3. Kerslake L. and Wegerif R.: The Semiotics of Emoji: The Rise of Visual Language in the
Age of the Internet. Media and Communication (ISSN: 2183–2439), vol. 5, Issue 4, pp.
75–78 (2017).
4. Swartz M., Crooks A., Kennedy W.: Diversity from Emojis and Keywords in Social
Media. 92-100 (2020).
5. Krylov, Y.V.: Semantika emodzi v virtual'nom dialoge [Semantics of emoji in virtual
dialogue]. Bulletin of Omsk State Pedagogical University. Humanitarian researches. №2
(15) (2017).
6. Sevens L., Vandeghinste V., Schuurman I., Eynde F.: Natural Language Generation from
Pictographs. 71-75 (2015).
7. Ota Y.: The Societal Role and Design of Pictograms as "Kansei Language" (Perceptual
Language). Biometrics and Kansei Engineering, International Conference on,
Takamatsu, Kagawa Japan, 2011 pp. 1-13 (2011).
8. Marcus A.: m-LoCoS UI: A Universal Visible Language for Global Mobile
Communication. In: Jacko J.A. (eds) Human-Computer Interaction. HCI Intelligent
Multimodal Interaction Environments. HCI 2007. Lecture Notes in Computer Science,
vol. 4552. Springer, Berlin, Heidelberg. (2007).