=Paper=
{{Paper
|id=Vol-3276/SSS-22_FinalPaper_14
|storemode=property
|title=Sense and Sensitivity: Knowledge Graphs as Training
Data for Processing Cognitive Bias, Context and Information Not Uttered
in Spoken Interaction
|pdfUrl=https://ceur-ws.org/Vol-3276/SSS-22_FinalPaper_14.pdf
|volume=Vol-3276
|authors=Christine Alexandris
|dblpUrl=https://dblp.org/rec/conf/aaaiss/Alexandris22
}}
==Sense and Sensitivity: Knowledge Graphs as Training
Data for Processing Cognitive Bias, Context and Information Not Uttered
in Spoken Interaction==
https://ceur-ws.org/Vol-3276/SSS-22_FinalPaper_14.pdf
Sense and Sensitivity: Knowledge Graphs as Training Data for
Processing Cognitive Bias, Context and Information Not Uttered
in Spoken Interaction
Christina Alexandris
National and Kapodistrian University of Athens
calexandris@gs.uoa.gr
Abstract The detection and registration information not uttered and
The processing of information not uttered in spoken interac- its conversion into knowledge graphs is based on previous
tion - subjective, perceived, context-related information, and research presented. Previous research involves an interac-
its conversion into “visible” information in knowledge tive application allowing the monitoring of fairness in inter-
graphs and subsequent use in vectors and other forms of train-
views and discussions in spoken political and journalistic
ing data contributes to registering and monitoring fairness in
spoken interaction and to the enrichment of NLP models and texts, especially in respect to Cognitive Bias, namely detect-
refinement of HCI/HRI applications. ing Lexical Bias and avoiding Confidence Bias.
Introduction Registering and Monitoring Fairness in Spo-
The present approach focuses on the processing of infor- ken Political and Journalistic Texts
mation not uttered in spoken interaction and its conversion In our previous research (Alexandris et al., 2021, Alexandris
into “visible” and processable information in the form of et al., 2020, Alexandris 2019, Alexandris, 2018), a pro-
knowledge graphs for its subsequent use in vectors and other cessing and evaluation framework was proposed for the gen-
forms of training data (Wang et al., 2021, Mountantonakis eration of graphic representations and tags corresponding to
and Tzitzikas, 2019, Tran, and Takashu, 2019, Mittal et al., values and benchmarks depicting the degree of information
2017). The knowledge graphs are intended, at least in the not uttered and non-neutral elements in Speaker behavior in
present stage, as a dataset for training a neural network. spoken text segments. The implemented processing and
Here, we describe the modelling of information not ut- evaluation framework allows the graphic representation to
tered into knowledge graphs for their subsequent conversion be presented in conjunction with the parallel depiction of
into neural networks, which, in turn, are targeted to learn speech signals and transcribed texts. Specifically, the align-
this particular type of data. ment of the generated graphic representation with the re-
This subjective, perceived, context-related information is spective segments of the spoken text enables a possible in-
directly linked to Cognitive Bias and to the monitoring of tegration in existing transcription tools.
(true) fairness in spoken interaction. Here, fairness is re- Although the concept of the generated graphic represen-
ferred to the sense that all voices-aspects-opinions are heard tations originates from the Discourse Tree prototype
clearly –that all participants are given a fair chance in the (Marcu, 1999), the characteristics of spontaneous turn-tak-
interview or discussion and are not purposefully or uncon- ing (Wilson and Wilson, 2005) and short spoken speech seg-
sciously repressed, oppressed, offended or even bullied. In ments did not facilitate the implementation of typical strate-
other words, the proposed graphs depict “sensitive” infor- gies based on Rhetorical Structure Theory (RST) (Stede,et
mation – “Sensitivity” of the speakers-participants. al., 2017, Zeldes, 2016, Carlson et al., 2001).
A crucial element in achieving “visibility” of information In particular, strategies typically employed in the con-
not uttered is causality, namely the registration and pro- struction of most Spoken Dialog Systems (such as keyword
cessing of reactions triggered by that very information not processing in the form of topic detection (Jurafsky and Mar-
uttered - the multiple facets of the “Sense” of the words tin, 2008, Nass and Brave 2005) from which approaches in-
and/or transcribed video and speech segments. volving neural networks are developed (Jurafsky and Martin
___________________________________
In T. Kido, K. Takadama (Eds.), Proceedings of the AAAI 2022 Spring Symposium
“How Fair is Fair? Achieving Wellbeing AI”, Stanford University, Palo Alto, California,
USA, March 21–23, 2022. Copyright © 2022 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY 4.0).
47
2020, Williams, et al., 2017)) were adapted in an interactive conversation and interaction may be compared to each other
annotation tool designed to operate with most commercial and be integrated in a database currently under develop-
transcription tools (Alexandris et al., 2021, Alexandris et al., ment. In this case, chosen relations between topics may de-
2020, Mourouzidis et al., 2019). The output provides the scribe Lexical Bias (Trofimova, 2014) and may differ ac-
User-Journalist with (i) the tracked indications of the topics cording to political, socio-cultural and linguistic character-
handled in the interview or discussion and (ii) the graphic istics of the user-evaluator, especially if international speak-
pattern of the discourse structure of the interview or discus- ers/users are concerned (Du et al, 2017, Paltridge 2012, Ma,
sion. The output (i) and (ii) also included functions and re- 2010, Yu et al., 2010, Pan, 2000) due to lack of world
spective values reflecting the degree in which the speakers- knowledge of the language community involved (Hatim,
participants address or avoid the topics in the dialog struc- 1997, Wardhaugh, 1992).
ture (“RELEVANCE” Module) as well as the degree of ten- The detecting and processing of information not uttered
sion in their interaction (“TENSION” Module). but perceived-sensed by speakers-participants allows the in-
tegration of additional information content – mean-
Sensitive Topics, Sensitive Participants: Previous ings/senses- in training data. This allows the enrichment of
Research data for understanding speaker-participant psychology-
mentality and sensitivities and the possible impact or conse-
The implemented “RELEVANCE” Module (Mourouzidis et
quences of a spoken journalistic/political text or interview.
al., 2019), intended for the evaluation of short speech seg-
This also allows an additional approach to registering of
ments, generates a visual representation from the user’s in-
cause-result relations on a discourse basis.
teraction, tracking the corresponding sequence of topics
The way sensitive topics and speakers-participant sensi-
(topic-keywords) chosen by the user and the perceived rela-
tivity are purposefully or unconsciously treated and man-
tions between them in the dialog flow. The generated visual
aged contributes to registering and monitoring fairness in
representations (not presented here) depict topics avoided,
spoken interaction, especially if non-native speakers and/or
introduced or repeatedly referred to by each Speaker-Partic-
an international community is concerned.
ipant, and, in specific types of cases, may indicate the exist-
The registration and integration of “invisible” infor-
ence of additional, “hidden”(Mourouzidis et al., 2019) Illo-
mation in training data contributes to enriching models and
cutionary Acts (Austin , 1962, Searle, 1969) other than “Ob-
to refining various Natural Language Processing (NLP)
taining Information Asked” or “Providing Information
tasks such as Sentiment Analysis and Opinion Mining – es-
Asked” in a discussion or interview. In the “RELEVANCE”
pecially when videos and multimodal data are processed
Module (Mourouzidis, et al., 2019), a high frequency of
(Poria et al., 2017). This approach may serve as (initial)
Repetitions (value 1), Generalizations (value 3) and Topic
training and test sets or for Speaker (User) behavior and ex-
Switches (value -1) in comparison to the duration of the spo-
pectations in Human-Computer Interaction and even in Hu-
ken interaction is connected to the “(Topic) Relevance”
man-Robot Interaction systems.
benchmarks with a value of “Relevance (X)” (Alexandris,
2020, Alexandris, 2018). These values were converted into
generated visual representations and were registered as tu- Creating Knowledge Graphs
ples or as triple tuples (Fig.1).
The complexity of the above-described type of spoken in-
(chemical weapons, military confrontation, 2) teraction can be accurately depicted in knowledge graphs.
(chemical weapons, military confrontation, 3) Knowledge graphs allow the multidimensional presentation
of information and the relations-links between information
chemical weapons -> ASSOC-> military confrontation (word –entities) within a dataset. The very nature and struc-
chemical weapons -> GEN-> military confrontation ture of knowledge graphs allows the representation of mul-
tiple facets of information – the multiple facets of the
Fig. 1. Analysis of triple tuples: Alternative perceived “As- “Sense” of the words and/or transcribed video speech seg-
sociation” (value 2) and “Generalization” (value 3) relations ments – although it is considered that there may exist some
between topics (Alexandris, 2020, Alexandris, 2018) types of information and/or some cases where there may not
be a 100% coverage by a knowledge graph.
Thus, the evaluation of Speaker-Participant behavior targets The possibility of converting knowledge graphs into vec-
to by-pass Cognitive Bias, specifically, Confidence Bias tors and other types of data, (Mittal et al., 2017) for training
(Hilbert, 2012) of the user-evaluator, especially if multiple neural networks (or other types of approaches and models)
users-evaluators may produce different forms of generated is presented in recent research, with Wang et al., 2021,
visual representations for the same conversation and inter- Mountantonakis and Tzitzikas, 2019, Tran, and Takashu,
action. The generated visual representations for the same
48
2019 as characteristic examples applying to the approach tuples and triplets with the information depicted in the cre-
presented here. ated knowledge graphs. Therefore, there should be a 100%
The conversion of knowledge graphs into training data compatibility between the information of the original se-
contributes to the integration and processing of complex in- quences and the knowledge graphs.
formation and information not uttered in Natural Language
Processing (NLP) tasks, thus, contributing to the creation of
even more sophisticated systems. This possibility would not Integrating Cognitive Bias in Knowledge
be considered if the above-stated characteristic research Graphs
work were not accomplished. Thus, the triple tuples pre-
In the context of the spoken interaction concerned, namely
sented in the example illustrated in Fig. 1, may be converted
interviews and discussions-debates in spoken political and
into the following form (Fig. 2):
journalistic texts, Cognitive Bias concerns association rela-
tions and argumentation related to inherent yet subtle socio-
ASSOC culturally determined linguistic features in (notably) com-
chemical military
weapons confrontation monly occurring words presented in previous research (ex-
amples from the international community: (the) “people”,
(our) “sea”).
GEN
chemical military These word types are detectable from the registered re-
weapons confrontation actions (Alexandris, 2021) they trigger in the processed di-
alog segment with two (or multiple) speakers-participants.
Since these words are very common and do not contain
Fig. 2. Fragments of knowledge graphs for alternative per-
descriptive features, the subtlety of their content is often un-
ceived “Association” and “Generalization” relations be-
tween topics consciously used or is perceived (mostly) by native speakers
and may contribute to the degree of formality or intensity of
The knowledge graphs, generated by an interactive applica- conveyed information in a spoken utterance. Here, these
tion presented in related/previous research (Alexandris et words concerning Cognitive Bias – Lexical Bias are referred
al., 2022, Alexandris et al., 2021, Mourouzidis et al., 2019), to as “Gravity” words (Alexandris, 2021, Alexandris, 2020).
involve the depiction of two main categories of information In other cases, these word types, although common
not uttered in spoken interaction. words, may contribute to a descriptive or emotional tone in
The first category (I) concerns additional perceived infor- an utterance and they may play a remarkable role in interac-
mation content and dimensions of –notably- very common tions involving persuasion and negotiations. Specifically, it
words – information not registered in language resources. is considered that, according to Rockledge et al, 2018, “the
This additional information may concern context-specific more extremely positive the word, the greater the probabil-
socio-cultural associations and Cognitive Bias. These words ity individuals were to associate that word with persuasion”.
may also constitute the perceived topic of a spoken utterance Here, these words concerning Cognitive Bias – Lexical Bias
or they may be perceived to play a crucial role in the content are referred to as “Evocative” words (Alexandris, 2021, Al-
of the spoken utterance. The perceived information is lan- exandris, 2020).
guage- and socio-culturally specific and is purposefully or The subtle impact of words is one of the tools typically
subconsciously conveyed or perceived-understood by used in persuasion and negotiations (Skonk, 2020, Evans
speakers-participants in the same language community. and Park, 2015).
The second category (II) concerns perceived paralinguis- In other words, information that is not uttered and infor-
tic elements influencing the information content of spoken mation that is perceived plays an essential role in under-
utterances. standing the above-described types of spoken interaction.
Both types of information not uttered are context-specific The modeling and processing of information not uttered and
and rely on whether they are perceived by the communi- information perceived does not only allow access to the
cating parties and on socio-cultural factors. complete content of spoken utterances and to registering and
The knowledge graphs can, subsequently, be converted monitoring fairness in spoken interaction, but also to predict
into vectors and other forms of training data which is tar- user-speaker behavior and reactions.
geted to contain (a) “visible” and processable information
not uttered in spoken interaction and (b) multiple versions The “Context” Relation: Visualizing and Linking
and varieties of training data with perceived information Perception and Sensitivity
generated by the interactive application. In the knowledge graphs, this additional information of the
Evaluation is based on the comparison of the (interac- above-described categories (I) and (II) is linked as an addi-
tively annotated) information in the original sequences of tional node to the spoken word with the proposed “Context”
49
relation. The term “Context” is chosen to signalize the per- “dignity” contributing to “No” answer and subsequent topic
ceived context of additional information in the form of co- switch (SWITCH).
occurring linguistic and/or paralinguistic features. If the perceived word-topic also constitutes a perceived
The context of additional information perceived and im- “Gravity” or “Evocative” word, the “&” indication is con-
plied by the speaker or perceived by the recipient influences verted into a “CONTEXT” relation with the same word.
the information content of the spoken utterance and its im- Furthermore, perceived word-topics and “Gravity” and
pact in the spoken interaction and dialogue structure. “Evocative” words may also trigger tension or other reac-
The “Context” relation signalizes the perceived “Gravity” tions and can be depicted as sequences for their subsequent
or “Evocative” word and links it to the word-topic of the modelling into knowledge graphs (Fig. 5, Fig. 6) or other
utterance. In other words, both words in the utterance –per- forms of data. Figure 4 depicts a speech segment with two
ceived word-topic and/or perceived “Gravity” or “Evoca- occurrences of a registered tension trigger from a speech
segment with detected “Tension” (the “TENSION” Module
tive” word may contribute to the type of response generated
implemented in previous research, Alexandris et al., 2020,
by an/the other speaker-participant, possibly also to tension.
Alexandris, 2019).
This case may be compared to multiple factors contributing
to a creation of a particular state or situation.
(sanctions, -2, &dignity),
The existence of a “Gravity” or an “Evocative” word is
(chemical weapons, military confrontation 2, &justice):
signalized by the “Context” relation itself, however, the
word’s additional dimension and content and/or interpreta-
TENSION {
tion (for example, “important” – for a “Gravity” word or sanctions ->NO -> SWITCH->[...]
“heartfelt” for an “Evocative” word) is not signalized and sanctions -> CONTEXT -> dignity
generated, at least not in the current stage of the present re-
search. This is because any additional content is may not be chemical weapons -> ASSOC-> military confrontation
limited to a singular interpretation summarized by a partic- chemical weapons ->CONTEXT -> justice
ular expression-keyword. } TENSION
We focus on the signalization and (cause-) effect of these
words during spoken interaction, as an additional factor in
the context. Fig. 4. Conversion of triple tuples and tuples for the gener-
Generated graphical representations of perceived word- ation of knowledge graphs from a speech segment with de-
topic relations and registered “Gravity” and “Evocative” tected “Tension”.
words (concerning Cognitive Bias – Lexical Bias) can be
The first occurrence is the “Gravity” word “dignity” co-oc-
converted into sequences for their subsequent conversion
curring within the same utterance with the word-topic “sanc-
into knowledge graphs or other forms of data for neural net-
tions” to which there is a negative response (“No”). In other
works and Machine Learning applications (Wang et al.,
words, within the detected “Tension” context, the negative
2021, Mountantonakis and Tzitzikas, 2019, Tran, and Ta- response is linked to the utterance with the perceived word-
kashu, 2019, Mittal et al., 2017). topic “sanctions”, containing the “Gravity” word “dignity”.
As described in previous research (Alexandris et al, The second occurrence of a registered tension trigger is the
2020), registered “Gravity” and “Evocative” words are ap- “Gravity” word “justice” co-occurring with the word-topic
pended as marked values with “&” in the respective tuples “chemical weapons” and linked to the word-topic “military
or triple tuples. In the sequences with the respective tuples confrontation” with a perceived “Association” (ASSOC) re-
or triple tuples, the “&” indication is converted into a “CON- lation. Fragments of knowledge graphs for the perceived
TEXT” relation. and registered relations between topics of the speech seg-
For example, a “No” answer (-2) preceded by “sanctions” ment in Fig.4 are depicted in Fig. 5 and Fig. 6 .
as a perceived word topic accompanied with a perceived
“Gravity” word “dignity” (sanctions, -2, &dignity), is con-
verted into the following sequences (Fig. 3): [.]
sanctions
NO SWITCH
(sanctions, -2, &dignity):
CONTEXT
sanctions ->NO -> SWITCH -> [...]
sanctions -> CONTEXT -> dignity dignity
Fig. 3. Conversion of triple tuples and tuples for the genera- Fig. 5. Fragment of knowledge graph for perceived “Grav-
tion of knowledge graphs: Integration of “Gravity” word ity” word (“dignity”), co-occurring with topic “sanctions”
50
in utterance segment with detected tension between speak- Paralinguistic Features: Sense and Sensitivity
ers.
Paralinguistic features constituting information that is not
uttered may often contribute to the correct detection and
ASSOC identification of subtle emotions, complementing or intensi-
chemical military
weapons confrontation fying the information content of the word or utterance.
There are also cases where the semantic content of a spoken
utterance may be contradicted by a gesture, facial expres-
CONTEXT sion or movement. However, as described in previous re-
search (Alexandris et. al, 2020, Alexandris, 2019), the use
justice of linguistic information with or without a link to paralin-
guistic features is proposed as a more reliable source of a
speaker’s attitude, behavior and intentions than stand-alone
Fig. 6. Fragment of knowledge graph for perceived “Grav-
paralinguistic features, especially if international speakers
ity” word (“justice”), co-occurring with “Association” (AS-
and/or an international public are concerned.
SOC) linked topics in utterance segment with detected ten-
The Gricean Cooperative Principle is violated if the infor-
sion between speakers. mation conveyed is perceived as not complete (Violation of
Quantity or Manner) or even contradicted by paralinguistic
On Registering Tension features (Violation of Quality) (Grice, 1989, Grice, 1975).
As presented in previous research (Alexandris et. al, 2020, Paralinguistic features may often contribute to the correct
Alexandris, 2019), multiple points of tension (“hot spots”- detection and identification of subtle emotions, comple-
consisting of a question-answer pair or a statement-response menting or intensifying the information content of the word
pair (or any other type of relation) between speaker turns) or word-topic, however, they are not always reliable, espe-
indicate a more argumentative than a collaborative interac- cially if international speakers and/or an international public
tion, even if speakers-participants display a calm and com- are involved.
posed behavior (Alexandris et. al, 2020, Alexandris, 2019). Paralinguistic features constituting information that is not
These points of tension (“hot spots”) involving, among uttered is also problematic in Data Mining and Sentiment
others, the registration of words and word-topics and the re- Analysis-Opinion Mining applications. These applications
actions they provoke (“tension-triggers” -Alexandris et. al, mostly rely on word groups, word sequences and/or senti-
2020, Alexandris, 2019), can contribute to the detection and ment lexica (Liu, 2012), including recent approaches with
identification of more subtle emotions, in the middle and the use of neural networks (Hedderich and Klakow, 2018,
outer zones of the Plutchik Wheel of Emotions (Plutchik, Shah et al., 2018, Arockiaraj, 2013), especially if Sentiment
1982). For example, for subtle negative reactions in the Analysis from videos (text, audio and video) is concerned.
Plutchik Wheel of Emotions, namely “Apprehension”, “An- However, even if context dependent multimodal utterance
noyance”, “Disapproval”, “Contempt”, “Aggressiveness” features are extracted, as proposed in relatively recent re-
(Plutchik, 1982). These emotions are usually too subtle to search (Poria, 2017), the semantic content of a spoken utter-
be easily extracted by sensor and/or speech signal data. ance may be either complemented or contradicted by a ges-
However, such subtle emotions may play a crucial role in ture, facial expression or movement.
spoken interactions involving persuasion and negotiations, As in the above-presented cases of “Gravity” and “Evoc-
although they are not always easily detectable or “visible”. ative” words, for paralinguistic features, the additional in-
Points of possible tension and/or conflict between speak- formation in the form of a linked node and respective word-
ers-participants (“hot-spots”) are identified by a set of crite- entity with the “Context” relation allow the “visibility” and,
ria based on the Gricean Cooperative Principle (Grice, 1989, subsequently, the processing of information not uttered.
Grice, 1975) (including paralinguistic elements, as pre-
sented in the following section) and signalized in generated The “Context” Relation: Visualizing and Linking
graphic representations of registered negotiations (or other Information Not Uttered
type of spoken interaction concerning persuasion), with spe- As in the case of perceived “Gravity” and “Evocative”
cial emphasis on words and topics triggering tension and words, paralinguistic elements can be similarly annotated as
non-collaborative speaker-participant behavior (Alexandris appended messages and processed with a “CONTEXT” re-
et. al, 2020, Alexandris, 2019). The detection of “hot spots” lation for their subsequent modelling into knowledge graphs
- points of tension implemented in previous research and in- or other forms of data. As described above, the “CON-
tegrated in knowledge graphs facilitates the detection of TEXT” relation enables the conversion of knowledge
words and word-topics associated with Persuasion and/or graphs and into vectors or other forms of data for neural net-
Tension, according to the factor of perception, subjectivity, works and Machine Learning applications (Wang et al.,
socio-cultural factors and the current state-of-affairs.
51
2021, Mountantonakis and Tzitzikas, 2019, Tran, and Ta- We note that the “CONTEXT” relation may link both a
kashu, 2019, Mittal et al., 2017). “Gravity”/ “Evocative” word and a paralinguistic element to
In the case of paralinguistic elements, the “Context” rela- the word-topic of a spoken utterance.
tion links an additional expression – a word-entity, to the Figure 7 and Figure 8 depict examples of registered para-
word uttered, for example, a modifier (Alexandris, 2010), linguistic elements and their respective messages from
completing its perceived content. This practice is typical of speech segments.
professional translators and interpreters when correctness
and precision is targeted (Koller, 2000), as research and re-
ports demonstrate. sanctions […]
Therefore, expert knowledge, concerning a finite set of
expressions-keywords, is integrated into the knowledge
graphs (with the interactive application presented in related CONTEXT
research, Alexandris et al., 2022). The additional infor- indeed
mation in the form of a linked node and respective word-
entity allows the “visibility” and, subsequently, the pro- Fig. 7. Fragment of knowledge graph for perceived mean-
cessing of information not uttered. ing of eyebrow-raise (“indeed”) co-occurring with topic
As described in previous research (Alexandris, 2020), the “sanctions” in utterance
interactive annotation of paralinguistic features is proposed,
depicting information complementing the information con-
tent of the spoken utterance (for example, “[+ facial-expr: sanctions
eyebrow-raise]” and “[+ gesture: low-hand-raise]”) or con- […]
stituting “stand-alone” information (Alexandris, 2021, Al-
exandris, 2020). In the latter case, information was interac- CONTEXT CONTEXT
tively annotated with the insertion of a separate message or
important dignity
response [Message/Response].
For example, the raising of eyebrows with the interpreta-
Fig. 8. Fragment of knowledge graph for perceived mean-
tion “I am surprised” [and / but this surprises me] (Alexan-
ing of eyebrow-raise (“important”) co-occurring with topic
dris, 2021, Alexandris, 2020) was indicated as [I am sur-
“sanctions” and perceived “Gravity” word (“dignity”) in
prised] (a), either as a pointer to information content or as or
utterance.
as a substitute of spoken information, a “stand-alone” para-
linguistic feature [Message /Response: I am surprised] (Al- For paralinguistic features depicting contradictory infor-
exandris, 2020). mation to the information content of the spoken utterance,
The alternative interpretations of the paralinguistic fea- the additional signalization of “!” is proposed in previous
ture (namely, “I am listening very carefully” (b), “What I research (Alexandris, 2021, Alexandris, 2020), for example,
am saying is important”(c) or “I have no intention of doing “[! facial-expr: eye-roll]” and “[! gesture: clenched-fist]”
otherwise” (d) Alexandris, 2021, Alexandris, 2020) was in- (Alexandris, 2021, Alexandris, 2020) or even a smile. In this
dicated with the respective annotations “[I am listening], case, the “CONTEXT” relation connects the chosen word-
[Please pay attention], [No] - [Message /Response: I am lis- topic from the speech segment with a word-expression con-
tening], [Message /Response: Please pay attention], [Mes- tradicting the spoken content with the expression “not re-
sage /Response: No]. The insertion of the respective type of ally” as a special indication (Fig. 9 and Fig. 10).
annotation for the paralinguistic features was according to
the parameters of the language(s) and the speaker(s) con- sanctions […]
cerned (Alexandris, 2021, Alexandris, 2020).
The “CONTEXT” relation connects the chosen word-
topic from the speech segment with a word-expression em- CONTEXT
phasizing / complementing the spoken content such as “in-
deed” or respective word summarizing the message. For ex- not really
ample, for the paralinguistic element [eyebrow-raise], pos-
sible options are: word-topic -> CONTEXT -> indeed,
word-topic -> CONTEXT -> surprised, word-topic -> Fig. 9. Fragment of knowledge graph for perceived contra-
CONTEXT -> important, or word-topic -> CONTEXT -> dictory meaning of eye-roll (“not really”) co-occurring
No. with topic “sanctions” in utterance.
52
Journalistic Data: From Graphical Representations to Knowledge
Graphs in an Interactive Application. Lecture Notes in Computer
sanctions […] Science Heidelberg, Germany: Springer.
Alexandris, C., Floros V., Mourouzidis D. 2021. Graphic Repre-
CONTEXT sentations of Spoken Interactions from Journalistic Data: Persua-
sion and Negotiations. In Human-Computer Interaction. Design
not really and User Experience Case Studies, Lecture Notes in Computer Sci-
ence, vol 12764, edited by M. Kurosu, 3-17. Cham, Switzerland:
Springer. doi.org/10.1007/978-3-030-78468-3_1.
Fig. 10. Fragment of knowledge graph for perceived con-
Alexandris, C. 2021. Registering the impact of Words in Spoken
tradictory meaning of smile (“not really”) co-occurring Political and Journalistic Texts. Journal of Human Language,
with topic “sanctions” in utterance. Rights and Security (1): 26-48. doi.org/10.22363/2713-0614-2021-
1-1-26-48
Alexandris, C. 2020. Issues in Multilingual Information Pro-
Conclusions and Further Research cessing of Spoken Political and Journalistic Texts in the Media and
Broadcast News. Newcastle upon Tyne, UK: Cambridge Scholars.
The processing of (subjective) perceived information, infor-
mation concerning Cognitive Bias and information not ut- Alexandris, C, Mourouzidis, D., Floros, V. 2020. Generating
Graphic Representations of Spoken Interactions Revisited: The
tered and its integration in training data contributes to a bet- Tension Factor and Information Not Uttered in Journalistic Data.
ter understanding of spoken interaction, registration of In Human-Computer Interaction. Design and User Experience,
cause-result relations on a discourse basis and a fair evalua- Lecture Notes in Computer Science, vol 12181, edited by M.
tion of all parties concerned, especially if non-native speak- Kurosu, 523–537. Cham, Switzerland: Springer Nature. doi.org/
ers and an international community are taken into account. 10.1007/978-3-030-49059-1_39
Furthermore, apart from contributing to enriching models Alexandris, C. 2019. Evaluating Cognitive Bias in Two-Party and
Multi-Party Spoken Interactions. In Proceedings of Interpretable
and refining NLP tasks such as Sentiment Analysis and
AI for Well-being: Understanding Cognitive Bias and Social Em-
Opinion Mining, the integration of “invisible” information beddedness (IAW 2019) in conjunction with AAAI Spring Sym-
in training data may serve as training and test sets for Hu- posium (SS-19-03), Stanford University, Palo Alto, CA. ceur-
man-Computer Interaction and Human-Robot Interaction ws.org/Vol-2448
applications. Alexandris, C. 2018. Measuring Cognitive Bias in Spoken Interac-
Expert knowledge and world knowledge is, therefore, in- tion and Conversation: Generating Visual Representations. In Be-
tegrated in training data using knowledge graphs. This pos- yond Machine Intelligence: Understanding Cognitive Bias and
Humanity for Well-Being AI: Papers from the 2018 AAAI Spring
sibility contributes to the enrichment of models for NLP, Symposium. Technical Report SS-18-03, Palo Alto, CA: AAAI
HCI and HRI applications, allowing the processing of infor- Press.
mation not uttered as well as multiple varieties and versions Alexandris, C. 2010. English, German and the International
of socio-linguistically related and user/speaker -specific im- “Semi-professional” Translator: A Morphological Approach to
plied and perceived information. Implied Connotative Features. Journal of Language and Transla-
The next stages of research concern the application of the tion, Sejong University, Korea, September 2010, vol. 11 (2): 7- 46.
training and test sets converted from the proposed Arockiaraj, C. M. 2013.Applications of Neural Networks In Data
knowledge graphs into the Human-Computer Interaction Mining. International Journal Of Engineering And Science, vol.3,
Issue 1 (May 2013): 8-11.
and/or Human-Robot Interaction systems for evaluating the
Austin J. L. 1962. How to Do Things with Words. 2nd edition 1976,
effectiveness of the proposed knowledge graphs and for
edited by J.O., Urmson and M. Sbisà. Oxford, UK: Oxford Univer-
their further upgrading and improvement. This includes sity Press, Oxford Paperbacks.
evaluating the behavior and output of the neural networks Carlson, L., Marcu, D., Okurowski, M. E. 2001. Building a Dis-
and the data learnt, especially if multiple datasets of differ- course-Tagged Corpus in the Framework of Rhetorical Structure
ent registered versions of the (subjective) perceived infor- Theory. In Proceedings of the 2nd SIGDIAL Workshop on Dis-
mation are concerned. Further research is geared towards course and Dialogue, Eurospeech 2001, Denmark.
the extensive implementation, evaluation and improvement Du, J., Alexandris, C., Mourouzidis, D., Floros, V.,Iliakis, A. 2017.
of the training data created by the knowledge graphs, espe- Controlling Interaction in Multilingual Conversation Revisited: A
Perspective for Services and Interviews in Mandarin Chinese. In
cially in respect to a wider range of languages and speakers
Lecture Notes in Computer Science, vol 10271, edited by M. Ku-
–and possibly, to other types of information not uttered re- rosu, 573–583. Heidelberg, Germany: Springer.
lated to Cognitive Bias and affecting Fairness. Evans, N. J., Park, D. 2015. Rethinking the Persuasion Knowledge
Model: Schematic Antecedents and Associative Outcomes of Per-
suasion Knowledge Activation for Covert Advertising, Journal of
References Current Issues & Research in Advertising, 36(2): 157-176.
doi.org/10.1080/10641734.2015.1023873
Alexandris, C., Du, J., Floros V. 2022. Forthcoming. Visualizing
and Processing Information Not Uttered in Spoken Political and
53
Grice, H. P. 1989. Studies in the Way of Words. Cambridge, MA: Plutchik, R. 1982. A psychoevolutionary theory of emotions. So-
Harvard University Press. cial Science Information. (21): 529-553.
Grice, H.P. 1975. Logic and conversation. In Syntax and Seman- doi.org/10.1177/053901882021004003
tics, vol. 3, edited by P. Cole and J. Morgan. New York: Academic Poria, S., Cambria, E., Hazarika, D., Mazumder, N., Zadeh, A.,
Press. Morency, L-P. 2017. Context-Dependent Sentiment Analysis in
Hatim, B. 1997.Communication Across Cultures: Translation The- User-Generated Videos. In Proceedings of the 55th Annual Meet-
ory and Contrastive Text Linguistics. Exeter, UK: University of ing of the Association for Computational Linguistics, Vancouver,
Exeter Press. Canada, July 30 - August 4, 2017, 873–88. Association for Com-
putational Linguistics - ACL. doi.org/10.18653/v1/P17-1081.
Hedderich, M. A., Klakow, D. 2018.Training a Neural Network in
a Low-Resource Setting on Automatically Annotated Noisy Data. Rocklage, M.D, Rucker D.D., Nordgren, L.F. 2018. Psychological
In Proceedings of the Workshop on Deep Learning Approaches for Science 2018, vol. 29(5): 749–760.
Low-Resource NLP, Melbourne, Australia, 12–18. Association for doi/org/10.1177/0956797617744797.
Computational Linguistics-ACL. Searle, J. R. 1969. Speech Acts: An Essay in the Philosophy of Lan-
Hilbert, M. 2012. Toward a Synthesis of Cognitive Biases: How guage. Cambridge, MA: Cambridge University Press.
Noisy Information Processing Can Bias Human Decision Making. Shah, K., Kopru, S., Ruvini, J-D. 2018. Neural Network based Ex-
Psychological Bulletin 138(2): 211-237. treme Classification and Similarity Models for Product Matching.
Jurafsky, D. and Martin, J.H. 2008. Speech and Language Pro- In Proceedings of NAACL-HLT 2018, New Orleans, Louisiana,
cessing, an Introduction to Natural Language Processing, Compu- June 1 - 6, 2018, 8–15. Association for Computational Linguistics-
tational Linguistics and Speech Recognition. 2nd ed. Prentice Hall ACL.
series in Artificial Intelligence. Upper Saddle River, NJ: Pearson Skonk, K. 2020. 5 Types of Negotiation Skills, Program on Nego-
Education. tiation Daily Blog, Harvard Law School, May the 14th 2020.
Jurafsky, D. and Martin, J.H. Forthcoming. Speech and Language https://www.pon.harvard.edu/daily/negotiation-skills-daily/types-
Processing, an Introduction to Natural Language Processing, of-negotiation-skills/ Accessed: 2021-12-30.
Computational Linguistics and Speech Recognition. 3rd edition. Stede, M., Taboada, M., Das, D. 2017. Annotation Guidelines for
(2020) Draft: https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf Rhetorical Structure. Manuscript. University of Potsdam and Si-
Koller, W. 2000. Der Begriff der Äquivalenz in der Übersetzung- mon Fraser University, March 2017.
swissenschaft. In Übertragung, Annäherung, Angleichung, edited Tran, H. N., Takashu, A. 2019. Analyzing Knowledge Graph Em-
by C. Fabricius-Hansen, C. and J. Østbø, 11-30. Frankfurt, Ger- bedding Methods from a Multi-Embedding Interaction Perspec-
many: Peter Lang. tive. In Proceedings of the 1st International Workshop on Data Sci-
Liu, B. 2012. Sentiment Analysis and Opinion Mining. San Rafael, ence for Industry 4.0 (DSI4) at EDBT/ICDT 2019 Joint Confer-
CA: Morgan & Claypool. ence. arxiv.org/abs/1903.11406 [cs.AI]. Ithaca, NY: Cornell Uni-
versity Library.
Ma J. 2010. A comparative analysis of the ambiguity resolution of
two English-Chinese MT approaches: RBMT and SMT. Dalian Trofimova I. 2014. Observer Bias: An Interaction of Temperament
University of Technology Journal 31(3): 114-119. Traits with Biases in the Semantic Perception of Lexical Material.
PLoSONE 9(1): e85677.
Marcu, D. 1999. Discourse trees are good indicators of importance
in text. In Advances in Automatic Text Summarization, 123-136, Wang, M., Qiu , L. 2021. A Survey on Knowledge Graph Embed-
edited by I. Mani and M. Maybury. Cambridge, MA: The MIT dings for Link Prediction. Symmetry 2021, 13: 485.
Press. doi.org/10.3390/sym13030485
Mittal, S., Joshi, A., Finin, T. 2017. Thinking, Fast and Slow: Com- Wardhaugh, R. 1992. An Introduction to Sociolinguistics, 2nd edi-
bining Vector Spaces and Knowledge Graphs. tion. Oxford, UK: Blackwell.
arXiv:1708.03310v2 [cs.AI]. Ithaca, NY: Cornell University Li- Williams, J.D., Asadi, K., Zweig, G. 2017. Hybrid Code Networks:
brary. practical and efficient end-to-end dialog control with supervised
Mountantonakis, M., Tzitzikas, Y. 2019. Knowledge Graph Em- and reinforcement learning. In Proceedings of the 55th Annual
beddings over Hundreds of Linked Datasets. In Metadata and Se- Meeting of the Association for Computational Linguistics, Van-
mantic Research, Communications in Computer and Information couver, Canada, July 30 - August 4, 2017, 665–677, Association
Science, vol 1057. (2019), edited by E. Garoufallou, F. Fallucchi for Computational Linguistics-ACL.
and E.W. De Luca. Cham, Switzerland: Springer. Wilson, M., Wilson, T.P. 2005. An oscillator model of the timing
doi.org/10.1007/978-3-030-36599-8_13 of turn taking. Psychonomic Bulletin and Review 2005:12 (6):,
Mourouzidis, D., Floros, V., Alexandris, C. 2019. Generating 957-968.
Graphic Representations of Spoken Interactions from Journalistic Yu, Z., Yu, Z., Aoyama, H., Ozeki, M., Nakamura, Y. 2010. Cap-
Data. In Lecture Notes in Computer Science, vol. 11566, edited by ture, Recognition, and Visualization of Human Semantic Interac-
M. Kurosu, 559–570. Basel, Switzerland: Springer. tions in Meetings. In Proceedings of PerCom, Mannheim, Ger-
Nass, C., Brave, S. 2005. Wired for Speech: How Voice Activates many, 2010.
and Advances the Human-Computer Relationship. Cambridge, Zeldes, A. 2016. rstWeb - A Browser-based Annotation Interface
MA: The ΜΙΤ Press. for Rhetorical Structure Theory and Discourse Relations. In Pro-
Paltridge B. 2012. Discourse Analysis: An Introduction. London: ceedings of NAACL-HLT 2016 System Demonstrations. San Di-
Bloomsbury Publishing. ego, CA, 1-5. aclweb.org/anthology/N/N16/N16-3001.pdf
Pan, Y. 2000. Politeness in Chinese Face-to-Face Interaction. Ad-
vances in Discourse Processes Series vol. 67. Stamford, CT, USA:
Ablex Publishing Corporation.
54