The processing of information not uttered in spoken interaction -subjective, perceived, context-related information, and its conversion into "visible" information in knowledge graphs and subsequent use in vectors and other forms of training data contributes to registering and monitoring fairness in spoken interaction and to the enrichment of NLP models and refinement of HCI/HRI applications.
The present approach focuses on the processing of information not uttered in spoken interaction and its conversion into "visible" and processable information in the form of knowledge graphs for its subsequent use in vectors and other forms of training data (Wang et al., 2021, Mountantonakis and Tzitzikas, 2019, Tran, and Takashu, 2019, Mittal et al., 2017). The knowledge graphs are intended, at least in the present stage, as a dataset for training a neural network.
Here, we describe the modelling of information not uttered into knowledge graphs for their subsequent conversion into neural networks, which, in turn, are targeted to learn this particular type of data.
This subjective, perceived, context-related information is directly linked to Cognitive Bias and to the monitoring of (true) fairness in spoken interaction. Here, fairness is referred to the sense that all voices-aspects-opinions are heard clearly -that all participants are given a fair chance in the interview or discussion and are not purposefully or unconsciously repressed, oppressed, offended or even bullied. In other words, the proposed graphs depict "sensitive" information -"Sensitivity" of the speakers-participants.
A crucial element in achieving "visibility" of information not uttered is causality, namely the registration and processing of reactions triggered by that very information not uttered -the multiple facets of the "Sense" of the words and/or transcribed video and speech segments.
The detection and registration information not uttered and its conversion into knowledge graphs is based on previous research presented. Previous research involves an interactive application allowing the monitoring of fairness in interviews and discussions in spoken political and journalistic texts, especially in respect to Cognitive Bias, namely detecting Lexical Bias and avoiding Confidence Bias.
In our previous research (Alexandris et al., 2021, Alexandris et al., 2020, Alexandris 2019, Alexandris, 2018), a processing and evaluation framework was proposed for the generation of graphic representations and tags corresponding to values and benchmarks depicting the degree of information not uttered and non-neutral elements in Speaker behavior in spoken text segments. The implemented processing and evaluation framework allows the graphic representation to be presented in conjunction with the parallel depiction of speech signals and transcribed texts. Specifically, the alignment of the generated graphic representation with the respective segments of the spoken text enables a possible integration in existing transcription tools.
Although the concept of the generated graphic representations originates from the Discourse Tree prototype (Marcu, 1999), the characteristics of spontaneous turn-taking (Wilson and Wilson, 2005) and short spoken speech segments did not facilitate the implementation of typical strategies based on Rhetorical Structure Theory (RST) (Stede,et al., 2017, Zeldes, 2016, Carlson et al., 2001).
In particular, strategies typically employed in the construction of most Spoken Dialog Systems (such as keyword processing in the form of topic detection (Jurafsky andMartin, 2008, Nass andBrave 2005) from which approaches involving neural networks are developed (Jurafsky and Martin 2020, Williams, et al., 2017)) were adapted in an interactive annotation tool designed to operate with most commercial transcription tools (Alexandris et al., 2021, Alexandris et al., 2020, Mourouzidis et al., 2019). The output provides the User-Journalist with (i) the tracked indications of the topics handled in the interview or discussion and (ii) the graphic pattern of the discourse structure of the interview or discussion. The output (i) and (ii) also included functions and respective values reflecting the degree in which the speakersparticipants address or avoid the topics in the dialog structure ("RELEVANCE" Module) as well as the degree of tension in their interaction ("TENSION" Module).
The implemented "RELEVANCE" Module (Mourouzidis et al., 2019), intended for the evaluation of short speech segments, generates a visual representation from the user's interaction, tracking the corresponding sequence of topics (topic-keywords) chosen by the user and the perceived relations between them in the dialog flow. The generated visual representations (not presented here) depict topics avoided, introduced or repeatedly referred to by each Speaker-Participant, and, in specific types of cases, may indicate the existence of additional, "hidden" (Mourouzidis et al., 2019) Illocutionary Acts (Austin , 1962, Searle, 1969) other than "Obtaining Information Asked" or "Providing Information Asked" in a discussion or interview. In the "RELEVANCE" Module (Mourouzidis, et al., 2019), a high frequency of Repetitions (value 1), Generalizations (value 3) and Topic Switches (value -1) in comparison to the duration of the spoken interaction is connected to the "(Topic) Relevance" benchmarks with a value of "Relevance (X)" (Alexandris, 2020, Alexandris, 2018). These values were converted into generated visual representations and were registered as tuples or as triple tuples (Fig. 1).
(chemical weapons, military confrontation, 2) (chemical weapons, military confrontation, 3) chemical weapons -> ASSOC-> military confrontation chemical weapons -> GEN-> military confrontation Fig. 1. Analysis of triple tuples: Alternative perceived "Association" (value 2) and "Generalization" (value 3) relations between topics (Alexandris, 2020, Alexandris, 2018) Thus, the evaluation of Speaker-Participant behavior targets to by-pass Cognitive Bias, specifically, Confidence Bias (Hilbert, 2012) of the user-evaluator, especially if multiple users-evaluators may produce different forms of generated visual representations for the same conversation and interaction. The generated visual representations for the same conversation and interaction may be compared to each other and be integrated in a database currently under development. In this case, chosen relations between topics may describe Lexical Bias (Trofimova, 2014) and may differ according to political, socio-cultural and linguistic characteristics of the user-evaluator, especially if international speakers/users are concerned (Du et al, 2017, Paltridge 2012, Ma, 2010, Yu et al., 2010, Pan, 2000) due to lack of world knowledge of the language community involved (Hatim, 1997, Wardhaugh, 1992).
The detecting and processing of information not uttered but perceived-sensed by speakers-participants allows the integration of additional information content -meanings/senses-in training data. This allows the enrichment of data for understanding speaker-participant psychologymentality and sensitivities and the possible impact or consequences of a spoken journalistic/political text or interview. This also allows an additional approach to registering of cause-result relations on a discourse basis.
The way sensitive topics and speakers-participant sensitivity are purposefully or unconsciously treated and managed contributes to registering and monitoring fairness in spoken interaction, especially if non-native speakers and/or an international community is concerned.
The registration and integration of "invisible" information in training data contributes to enriching models and to refining various Natural Language Processing (NLP) tasks such as Sentiment Analysis and Opinion Mining -especially when videos and multimodal data are processed (Poria et al., 2017). This approach may serve as (initial) training and test sets or for Speaker (User) behavior and expectations in Human-Computer Interaction and even in Human-Robot Interaction systems.
The complexity of the above-described type of spoken interaction can be accurately depicted in knowledge graphs. Knowledge graphs allow the multidimensional presentation of information and the relations-links between information (word -entities) within a dataset. The very nature and structure of knowledge graphs allows the representation of multiple facets of information -the multiple facets of the "Sense" of the words and/or transcribed video speech segments -although it is considered that there may exist some types of information and/or some cases where there may not be a 100% coverage by a knowledge graph.
The possibility of converting knowledge graphs into vectors and other types of data, (Mittal et al., 2017) for training neural networks (or other types of approaches and models) is presented in recent research, with Wang et al., 2021, Mountantonakis and Tzitzikas, 2019, Tran, and Takashu, 2019 as characteristic examples applying to the approach presented here.
The conversion of knowledge graphs into training data contributes to the integration and processing of complex information and information not uttered in Natural Language Processing (NLP) tasks, thus, contributing to the creation of even more sophisticated systems. This possibility would not be considered if the above-stated characteristic research work were not accomplished. Thus, the triple tuples presented in the example illustrated in Fig. 1, may be converted into the following form (Fig. 2): ASSOC GEN Fig. 2. Fragments of knowledge graphs for alternative perceived "Association" and "Generalization" relations between topics
The knowledge graphs, generated by an interactive application presented in related/previous research (Alexandris et al., 2022, Alexandris et al., 2021, Mourouzidis et al., 2019), involve the depiction of two main categories of information not uttered in spoken interaction.
The first category (I) concerns additional perceived information content and dimensions of -notably-very common words -information not registered in language resources. This additional information may concern context-specific socio-cultural associations and Cognitive Bias. These words may also constitute the perceived topic of a spoken utterance or they may be perceived to play a crucial role in the content of the spoken utterance. The perceived information is language-and socio-culturally specific and is purposefully or subconsciously conveyed or perceived-understood by speakers-participants in the same language community.
The second category (II) concerns perceived paralinguistic elements influencing the information content of spoken utterances.
Both types of information not uttered are context-specific and rely on whether they are perceived by the communicating parties and on socio-cultural factors.
The knowledge graphs can, subsequently, be converted into vectors and other forms of training data which is targeted to contain (a) "visible" and processable information not uttered in spoken interaction and (b) multiple versions and varieties of training data with perceived information generated by the interactive application.
Evaluation is based on the comparison of the (interactively annotated) information in the original sequences of tuples and triplets with the information depicted in the created knowledge graphs. Therefore, there should be a 100% compatibility between the information of the original sequences and the knowledge graphs.
In the context of the spoken interaction concerned, namely interviews and discussions-debates in spoken political and journalistic texts, Cognitive Bias concerns association relations and argumentation related to inherent yet subtle socioculturally determined linguistic features in (notably) commonly occurring words presented in previous research (examples from the international community: (the) "people", (our) "sea"). These word types are detectable from the registered reactions (Alexandris, 2021) they trigger in the processed dialog segment with two (or multiple) speakers-participants.
Since these words are very common and do not contain descriptive features, the subtlety of their content is often unconsciously used or is perceived (mostly) by native speakers and may contribute to the degree of formality or intensity of conveyed information in a spoken utterance. Here, these words concerning Cognitive Bias -Lexical Bias are referred to as "Gravity" words (Alexandris, 2021, Alexandris, 2020).
In other cases, these word types, although common words, may contribute to a descriptive or emotional tone in an utterance and they may play a remarkable role in interactions involving persuasion and negotiations. Specifically, it is considered that, according to Rockledge et al, 2018, "the more extremely positive the word, the greater the probability individuals were to associate that word with persuasion". Here, these words concerning Cognitive Bias -Lexical Bias are referred to as "Evocative" words (Alexandris, 2021, Alexandris, 2020).
The subtle impact of words is one of the tools typically used in persuasion and negotiations (Skonk, 2020, Evans andPark, 2015).
In other words, information that is not uttered and information that is perceived plays an essential role in understanding the above-described types of spoken interaction. The modeling and processing of information not uttered and information perceived does not only allow access to the complete content of spoken utterances and to registering and monitoring fairness in spoken interaction, but also to predict user-speaker behavior and reactions.
In the knowledge graphs, this additional information of the above-described categories (I) and (II) is linked as an additional node to the spoken word with the proposed "Context" The context of additional information perceived and implied by the speaker or perceived by the recipient influences the information content of the spoken utterance and its impact in the spoken interaction and dialogue structure.
The "Context" relation signalizes the perceived "Gravity" or "Evocative" word and links it to the word-topic of the utterance. In other words, both words in the utterance -perceived word-topic and/or perceived "Gravity" or "Evocative" word may contribute to the type of response generated by an/the other speaker-participant, possibly also to tension. This case may be compared to multiple factors contributing to a creation of a particular state or situation.
The existence of a "Gravity" or an "Evocative" word is signalized by the "Context" relation itself, however, the word's additional dimension and content and/or interpretation (for example, "important" -for a "Gravity" word or "heartfelt" for an "Evocative" word) is not signalized and generated, at least not in the current stage of the present research. This is because any additional content is may not be limited to a singular interpretation summarized by a particular expression-keyword.
We focus on the signalization and (cause-) effect of these words during spoken interaction, as an additional factor in the context. Generated graphical representations of perceived wordtopic relations and registered "Gravity" and "Evocative" words (concerning Cognitive Bias -Lexical Bias) can be converted into sequences for their subsequent conversion into knowledge graphs or other forms of data for neural networks and Machine Learning applications (Wang et al., 2021, Mountantonakis and Tzitzikas, 2019, Tran, and Takashu, 2019, Mittal et al., 2017).
As described in previous research (Alexandris et al, 2020), registered "Gravity" and "Evocative" words are appended as marked values with "&" in the respective tuples or triple tuples. In the sequences with the respective tuples or triple tuples, the "&" indication is converted into a "CON-TEXT" relation.
For example, a "No" answer (-2) preceded by "sanctions" as a perceived word topic accompanied with a perceived "Gravity" word "dignity" (sanctions, -2, &dignity), is converted into the following sequences (Fig. 3):
(sanctions, -2, &dignity): sanctions ->NO -> SWITCH -> [...] sanctions -> CONTEXT -> dignity Fig. 3. Conversion of triple tuples and tuples for the generation of knowledge graphs: Integration of "Gravity" word "dignity" contributing to "No" answer and subsequent topic switch (SWITCH).
If the perceived word-topic also constitutes a perceived "Gravity" or "Evocative" word, the "&" indication is converted into a "CONTEXT" relation with the same word.
Furthermore, perceived word-topics and "Gravity" and "Evocative" words may also trigger tension or other reactions and can be depicted as sequences for their subsequent modelling into knowledge graphs (Fig. 5, Fig. 6) or other forms of data. Figure 4 depicts a speech segment with two occurrences of a registered tension trigger from a speech segment with detected "Tension" (the "TENSION" Module implemented in previous research, Alexandris et al., 2020, Alexandris, 2019). The first occurrence is the "Gravity" word "dignity" co-occurring within the same utterance with the word-topic "sanctions" to which there is a negative response ("No"). In other words, within the detected "Tension" context, the negative response is linked to the utterance with the perceived wordtopic "sanctions", containing the "Gravity" word "dignity".
The second occurrence of a registered tension trigger is the "Gravity" word "justice" co-occurring with the word-topic "chemical weapons" and linked to the word-topic "military confrontation" with a perceived "Association" (ASSOC) relation. Fragments of knowledge graphs for the perceived and registered relations between topics of the speech segment in Fig. 4 are depicted in Fig. 5 and Fig. 6.
NO SWITCH CONTEXT Fig. 5. Fragment of knowledge graph for perceived "Gravity" word ("dignity"), co-occurring with topic "sanctions" sanctions dignity [.] in utterance segment with detected tension between speakers. ASSOC CONTEXT Fig. 6. Fragment of knowledge graph for perceived "Gravity" word ("justice"), co-occurring with "Association" (AS-SOC) linked topics in utterance segment with detected tension between speakers.
As presented in previous research (Alexandris et. al, 2020, Alexandris, 2019), multiple points of tension ("hot spots"consisting of a question-answer pair or a statement-response pair (or any other type of relation) between speaker turns) indicate a more argumentative than a collaborative interaction, even if speakers-participants display a calm and composed behavior (Alexandris et. al, 2020, Alexandris, 2019).
These points of tension ("hot spots") involving, among others, the registration of words and word-topics and the reactions they provoke ("tension-triggers" -Alexandris et. al, 2020, Alexandris, 2019), can contribute to the detection and identification of more subtle emotions, in the middle and outer zones of the Plutchik Wheel of Emotions (Plutchik, 1982). For example, for subtle negative reactions in the Plutchik Wheel of Emotions, namely "Apprehension", "Annoyance", "Disapproval", "Contempt", "Aggressiveness" (Plutchik, 1982). These emotions are usually too subtle to be easily extracted by sensor and/or speech signal data. However, such subtle emotions may play a crucial role in spoken interactions involving persuasion and negotiations, although they are not always easily detectable or "visible".
Points of possible tension and/or conflict between speakers-participants ("hot-spots") are identified by a set of criteria based on the Gricean Cooperative Principle (Grice, 1989, Grice, 1975) (including paralinguistic elements, as presented in the following section) and signalized in generated graphic representations of registered negotiations (or other type of spoken interaction concerning persuasion), with special emphasis on words and topics triggering tension and non-collaborative speaker-participant behavior (Alexandris et. al, 2020, Alexandris, 2019). The detection of "hot spots" -points of tension implemented in previous research and integrated in knowledge graphs facilitates the detection of words and word-topics associated with Persuasion and/or Tension, according to the factor of perception, subjectivity, socio-cultural factors and the current state-of-affairs.
Paralinguistic features constituting information that is not uttered may often contribute to the correct detection and identification of subtle emotions, complementing or intensifying the information content of the word or utterance. There are also cases where the semantic content of a spoken utterance may be contradicted by a gesture, facial expression or movement. However, as described in previous research (Alexandris et. al, 2020, Alexandris, 2019), the use of linguistic information with or without a link to paralinguistic features is proposed as a more reliable source of a speaker's attitude, behavior and intentions than stand-alone paralinguistic features, especially if international speakers and/or an international public are concerned.
The Gricean Cooperative Principle is violated if the information conveyed is perceived as not complete (Violation of Quantity or Manner) or even contradicted by paralinguistic features (Violation of Quality) (Grice, 1989, Grice, 1975).
Paralinguistic features may often contribute to the correct detection and identification of subtle emotions, complementing or intensifying the information content of the word or word-topic, however, they are not always reliable, especially if international speakers and/or an international public are involved.
Paralinguistic features constituting information that is not uttered is also problematic in Data Mining and Sentiment Analysis-Opinion Mining applications. These applications mostly rely on word groups, word sequences and/or sentiment lexica (Liu, 2012), including recent approaches with the use of neural networks (Hedderich and Klakow, 2018, Shah et al., 2018, Arockiaraj, 2013), especially if Sentiment Analysis from videos (text, audio and video) is concerned. However, even if context dependent multimodal utterance features are extracted, as proposed in relatively recent research (Poria, 2017), the semantic content of a spoken utterance may be either complemented or contradicted by a gesture, facial expression or movement.
As in the above-presented cases of "Gravity" and "Evocative" words, for paralinguistic features, the additional information in the form of a linked node and respective wordentity with the "Context" relation allow the "visibility" and, subsequently, the processing of information not uttered.
As in the case of perceived "Gravity" and "Evocative" words, paralinguistic elements can be similarly annotated as appended messages and processed with a "CONTEXT" relation for their subsequent modelling into knowledge graphs or other forms of data. As described above, the "CON-TEXT" relation enables the conversion of knowledge graphs and into vectors or other forms of data for neural networks and Machine Learning applications (Wang et al., justice chemical weapons military confrontation 2021, Mountantonakis and Tzitzikas, 2019, Tran, and Takashu, 2019, Mittal et al., 2017).
In the case of paralinguistic elements, the "Context" relation links an additional expression -a word-entity, to the word uttered, for example, a modifier (Alexandris, 2010), completing its perceived content. This practice is typical of professional translators and interpreters when correctness and precision is targeted (Koller, 2000), as research and reports demonstrate.
Therefore, expert knowledge, concerning a finite set of expressions-keywords, is integrated into the knowledge graphs (with the interactive application presented in related research, Alexandris et al., 2022). The additional information in the form of a linked node and respective wordentity allows the "visibility" and, subsequently, the processing of information not uttered.
As described in previous research (Alexandris, 2020), the interactive annotation of paralinguistic features is proposed, depicting information complementing the information content of the spoken utterance (for example, "[+ facial-expr: eyebrow-raise]" and "[+ gesture: low-hand-raise]") or constituting "stand-alone" information (Alexandris, 2021, Alexandris, 2020). In the latter case, information was interactively annotated with the insertion of a separate message or response [Message/Response].
For example, the raising of eyebrows with the interpretation "I am surprised" [and / but this surprises me] (Alexandris, 2021, Alexandris, 2020) was indicated as [I am surprised] (a), either as a pointer to information content or as or as a substitute of spoken information, a "stand-alone" paralinguistic feature [Message /Response: I am surprised] (Alexandris, 2020).
The alternative interpretations of the paralinguistic feature (namely, "I am listening very carefully" (b), "What I am saying is important"(c) or "I have no intention of doing otherwise" (d) Alexandris, 2021, Alexandris, 2020) The insertion of the respective type of annotation for the paralinguistic features was according to the parameters of the language(s) and the speaker(s) concerned (Alexandris, 2021, Alexandris, 2020).
The "CONTEXT" relation connects the chosen wordtopic from the speech segment with a word-expression emphasizing / complementing the spoken content such as "indeed" or respective word summarizing the message. For example, for the paralinguistic element [eyebrow-raise], possible options are: word-topic -> CONTEXT -> indeed, word-topic -> CONTEXT -> surprised, word-topic -> CONTEXT -> important, or word-topic -> CONTEXT -> No.
We note that the "CONTEXT" relation may link both a "Gravity"/ "Evocative" word and a paralinguistic element to the word-topic of a spoken utterance.
Figure 7 and Figure 8 depict examples of registered paralinguistic elements and their respective messages from speech segments. CONTEXT Fig. 7. Fragment of knowledge graph for perceived meaning of eyebrow-raise ("indeed") co-occurring with topic "sanctions" in utterance CONTEXT CONTEXT Fig. 8. Fragment of knowledge graph for perceived meaning of eyebrow-raise ("important") co-occurring with topic "sanctions" and perceived "Gravity" word ("dignity") in utterance.
For paralinguistic features depicting contradictory information to the information content of the spoken utterance, the additional signalization of "!" is proposed in previous research (Alexandris, 2021, Alexandris, 2020), for example, "[! facial-expr: eye-roll]" and "[! gesture: clenched-fist]" (Alexandris, 2021, Alexandris, 2020) or even a smile. In this case, the "CONTEXT" relation connects the chosen wordtopic from the speech segment with a word-expression contradicting the spoken content with the expression "not really" as a special indication (Fig. 9 and Fig. 10).
The processing of (subjective) perceived information, information concerning Cognitive Bias and information not uttered and its integration in training data contributes to a better understanding of spoken interaction, registration of cause-result relations on a discourse basis and a fair evaluation of all parties concerned, especially if non-native speakers and an international community are taken into account. Furthermore, apart from contributing to enriching models and refining NLP tasks such as Sentiment Analysis and Opinion Mining, the integration of "invisible" information in training data may serve as training and test sets for Human-Computer Interaction and Human-Robot Interaction applications.
Expert knowledge and world knowledge is, therefore, integrated in training data using knowledge graphs. This possibility contributes to the enrichment of models for NLP, HCI and HRI applications, allowing the processing of information not uttered as well as multiple varieties and versions of socio-linguistically related and user/speaker -specific implied and perceived information.
The next stages of research concern the application of the training and test sets converted from the proposed knowledge graphs into the Human-Computer Interaction and/or Human-Robot Interaction systems for evaluating the effectiveness of the proposed knowledge graphs and for their further upgrading and improvement. This includes evaluating the behavior and output of the neural networks and the data learnt, especially if multiple datasets of different registered versions of the (subjective) perceived information are concerned. Further research is geared towards the extensive implementation, evaluation and improvement of the training data created by the knowledge graphs, especially in respect to a wider range of languages and speakers -and possibly, to other types of information not uttered related to Cognitive Bias and affecting Fairness.