Increasing information saturation in all spheres of life entails accumulation of large amounts of data that must be perceived, processed and decisions must be made on their basis. Therefore, the issues of interaction with data and of visual analytics require solutions at a new level. The authors investigate communication between the user and the data, highlight a number of negative trends that prevent effective use of visualization in solving practical problems, formulate aspects of subjective influence on the decision-making procedure, namely those leading to significant decrease in the efficiency of visual analytics systems. The paper proposes an approach to the problem of reasonable use of existing and potential visualization capabilities for solving data analysis problems and making control decisions. A significant factor hindering the development of visual analytics is a lack of a model for coordinated use of computational and subjective resources corresponding to the technical level of computer visualization capable of ensuring close communication between the researcher and the available data. The paper describes an approach based on the concept of visual communication, the properties of which are determined on the basis of a number of key concepts of semiotics and linguistics.
The continuously increasing saturation of information environment of our time is a factor influencing the effectiveness of use of new data and knowledge both in everyday life and in scientific research. The availability of up-to-date digital data as well as the possibility of their (practically) unlimited preservation, at a certain stage, made
The reported study was funded by RFBR, project number 19-07-00844. vital the task of preliminary analysis, data formalization, aimed at extracting new knowledge and reducing the amount of information involved in decision-making procedures for solving control problems in real scientific and production processes.
In practice, many existing approaches to the formalized data presentation provoke one of the possible negative situations. On the one hand, a simplified representation of data used to increase the speed of decision making leads to distancing of the decision maker from the initial data and makes the resulting solution dependent on subjective experience and knowledge, as well as on the correctness of the formalized representation. In the opposite case, tools are used to ensure the availability of both the initial data and the results of their subsequent analysis. For situations involving generation of many alternative decision hypotheses and their subsequent verification, the use of such tools can significantly increase the amount of analyzed data and, as a result, complicate the final decision.
Additional circumstances, that often make a significant contribution to the efficiency of data analysis and making control decisions based on it, are simultaneous participation of more than one subject in this process and the necessity for selecting the most attractive decision hypothesis on the basis of joint analysis of data related to different subject areas and accumulated knowledge. In this case, the conflict of opinions becomes an obstacle, overcoming of which depends on circumstances that go beyond the boundaries of the current problem, for example, on coordination of participants’ competencies or the priority of their opinions. 2
Technologies for solving applied problems associated with the use of machine
learning capabilities as a basis for making a final decision are deliberately left out of the
discussion [
Upon close examination, communication between the user and the data appears to
be a poorly researched process, despite the fact that it is mandatory for analytical tools
of any type. The increasing complexity of analysis problems and tools proposed for
their solution leads to the emergence of significant requirements for methods of such
communication including their implementation in the form of HCI (human-computer
interface) tools. Proliferation of computer systems for data processing and analysis is
gradually forming an expanding community of users who employ a specialized
communication system to interact with information environment [
A distinctive feature of the development of such a system is borrowing ways of presenting information that are characteristic of other types of human activity. The variety of acceptable representation methods is based on the use of all possible senso
Semiotic Approach in the Development of Interactive Visual Analytics Systems 3 ry types and their combinations. This enables transfer of various types of data that satisfies constraints of a specific task including achieving a desired transfer rate, interpretation time, the need to employ user experience and any others. Visualization tools are still the most common variant of technical solutions in the development of data analysis tools, because visual perception is rightly believed to have a number of necessary advantages. However, further progress in this direction depends on the answers to several questions. 3
Review of existing and emerging data visualization tools, especially those used in
science, social process management or business analytics [4; 5], allow recognizing a
number of negative tendencies that impede the effective use of visualization in
solving practical problems.
1. Application of many modern technical developments in the field of computer
visualization is more caused by the need for their commercial promotion than the
achievable effectiveness.
2. The developed HCI tools use conceptual solutions proposed many years ago and
probably do not correspond to the increased complexity level of information
communication tasks [
Thus, the development of visual data representation tools intended for the user to
obtain a solution to a certain practical analysis problem requires a formalized
description of communication between the user and the visualized data (hereinafter, visual
communication) [
Special attention should be paid to the subjective side of visual communication with the analyzed data, since it determines the features of the achieved result, which distinguish the use of visual analytics from the use of machine learning tools. There are several aspects of subjective influence on the final decision-making procedure, ignoring or mistaken use of which leads to a significant decrease in the effectiveness of visual analytics systems: • Uncertainty in the results of interpretation experienced by the user largely depends on the consistency of his own data representation (internal image) and the proposed visualization metaphor. Therefore, the use of expressive means known or understood by the user earlier should help to overcome one of the communication barriers. • The procedure for verifying the information obtained as a result of visual communication, taking into account the peculiarities of visual perception, occurs as a multiple repetition of the search for an answer to the formulated question in an accessible visual data image, with the consideration of new information accumulated by the user at each step of communication. A need arises for a systematized and operational, from the user's point of view, access to the changing volume of unverified data. • Doubt. The repeated procedure of comprehension (interpretation), depending on the culture of thinking inherent in the user, can last as long as desired until the image formed under the influence of new information is accepted as an element of the user's knowledge system.
There arises a problem of coordinating several ways of representing information: visualization tools used for the initial data, temporary fixation of unverified versions as well as images that are remembered and used in the future by the user as his own knowledge. 5
Based on definitions of a number of terms used in linguistics, developers of visualization systems face several independent tasks that currently do not have substantiated and complete solutions (Figure 1).
Visualization sigmatics can indicate problems related to the selection or development of visually perceptible objects designed to convey the required meaning to the user. Visualization semantics, obviously, should determine principles of coordinated use of available visualization technologies to form a consistent system for displaying initial data and interpretation results. Visualization syntactics should focus on solving problems of effective combination of visual objects integrated in data images. Finally, visualization pragmatics corresponds to the study of peculiarities of understanding by an observer of a visual image or their sequence, including such complex situations as
Semiotic Approach in the Development of Interactive Visual Analytics Systems 5 visualization of abstract data or ideas that do not have perceptually-based representations known to the user. A certain paradox is obvious in the development of visualization tools and their practical application associated with insufficient attention paid by developers to the pragmatics of visualization. Modern technical and software solutions used by computer visualization systems allow achieving a high level of complexity and detail of images reproduced in various ways. However, assessment of information content and its interpretability necessary for reasonable application of such systems cannot be obtained on the basis of the widespread concept of visual communication as a process similar to the functioning of communication systems.
As a consequence of peculiarities of user perception and thinking implemented at
the time of visual communication (Figure 2), ambiguity of data visual interpretation
arises which predetermines several approaches to the use of visualization tools that
differ in tasks solved, visualization requirements and end user characteristics [
The established hypothetical correspondence between the concepts of linguistics and some processes inherent in visualization functioning suggests the existence of similarities between speech and visual communication. An important consequence of this resemblance is preparedness of the user's thinking for the appearance in the samples of synthesized computer visualization of expressive techniques similar to the usual verbal communication: emotional statements, rhetorical questions, metaphors, ambiguous images, etc. Moreover, visualization, based on the modern understanding of its technical capabilities, can be considered as a communication option, similar to language communication, but more capacious, multifaceted and preferential for many practical situations.
Focusing on the similarity between speech and visual communication, developers of information visualization tools, used among other things in control decision support systems, get a promising direction for the development of information visualization tools. Its feature is structured sequential transmission to the consumer (recipient) of information volumes that exceed the capabilities of both language messages and traditional computer graphics. However, the achievement of this result faces at least several problems that require special research.
Firstly, the developer of visualization tools must get systematic and deep understanding of the features of user communication with visualized information. This will allow to move from the extensive complication of visualization tools to communication focused on the user's perception and thinking.
Secondly, the question of how to overcome the communicative barrier associated with different levels of preliminary informing users, their cultural (including linguistic) and physiological characteristics remains practically unexplored. Existing visualization methods circumvent these issues by using a standardized (for historical and economic reasons) set of expressive techniques, which is currently becoming a negative factor that hinders the development of visual analytics.
Thirdly, the need naturally arises to develop techniques for visual data representation both borrowing effective means of verbal communication (imagery, comparisons, metaphors, attention retention, emotional coloring) and using unique visualization capabilities (parallel presentation; memorizing and operating not with attributes but images; interpretation of movement, etc.). Experimental evaluation of the effectiveness of new visualization techniques is still another independent and laborious task.
Based on the statements made, benefits arising from independent improvement of
some characteristics of visualization tools corresponding to various aspects of the
semiotic approach have been assessed. The most significant results obtained when
solving a series of test problems for multidimensional data analysis (Table 1) are
associated with improvement in visualization pragmatics as a result of organizing
interactive communication between the user and visual analytics (Figure 4).
In addition, transition to the use of visualization systems with high information
saturation makes it more important to solve a number of problems related to perceptual
psychology. The most common of them include passivity of thinking, leading to
resistance to the new or unexpected, subjective fears and prejudices, changing
emotional background, fatigue, loss of concentration or motivation [