Affective Computing (AC) systems have been well developed in the past decades as an effective solution for recognizing, interpreting, processing, and simulating human affects. Additionally, context-aware Affective Computing systems [2] emerged as a novel way of shifting from context-aware Affective Computing systems to affective aware intelligent Human Computer Interaction systems to overcome the fact that contextual information can not be discounted in doing automatic analysis of human affective behavior. Thus, the fundamental assumption of context-aware Affective Computing systems is that context is able of shaping how people interpret the high and complex expressions of people and machines. The variation of these expressions in human behavior arise not only from a subject's internal psychological or cognitive state but also from other subjects or the environment. For instance, frowning behavior may be an indicator of anger or it may be due to concentration depending on the contextual interactional setting. Context-based affect aware analysis needs to clarify the preliminary selection of contextual variables in order to further assist intepretation of contextual effects in Affective Computing systems. As a result, how to incorporate contexts into Affective Computing systems is always a research question in this domain. However, prior to that, which variables should be considered as contexts is still under question.
Currently, several context-aware Affective Computing systems [5] have been developed but very few research went back to discuss the definition of contexts. And researchers simply blend location, identities of people around the user, environment, social interaction etc. and other variables together to consider them as contexts, which further creates the confusion on which should be the first decisions to be made prior to creating context-aware automatic affect analysis systems. The definition and exploration of context are not only related to the selection of contextual variables in Affective Computing systems, but are also relevant to the interpretation of contextual effects based on the outcomes or findings in the experiments. It is obvious that the academic area focuses more on the development of effective context-aware Affective Computing systems, but ignores the identification of contexts and interpretation of contextual effects in recent years.
This paper is organized as follows: Section 2 provides an overview of how the term context has been studied so far in various disciplines. Section 3 presents an overview of my research and the preliminary findings in my previous work. Section 4 discusses my plan for achieving my overall objective for the remainder of my doctoral work and finally, Section 5 concludes by summarizing the impact and relevance of the proposed approaches to the field of context-aware Affective Computing.
Actually contexts have been studied in various disciplines, such as ubiquitous computing, contextual advertising, social signal processing, HCI, gaming, mental health etc., where the definition differs resulting in a different understanding of contexts among those areas. In context-aware Affective Computing systems, the earliest research papers [5] may bring us to look back upon almost ten years ago; however, the field has yet to agree on the definition of context. Several researchers simply blend verbal content (semantics), knowledge of the general interactional setting, discourse and social situations and other variables together and consider all of them as contexts, which further creates confusion on which should be the most appropriate contextual parameter w.r.t context-aware Affective Computing systems (Figure 1).
The most commonly used definition is the one given by Abowd et al. in 1999 [1], "context is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves." This definition hardly limits the set of variables that can be taken into account, and it is still ambiguous without clear guidelines to select appropriate variables in AC systems. Apparently, the definition and selection of contexts is a domain-specific problem, where classification of contextual variables is a typical way to put different variables in categories but it is still not general enough and not flexible in interpreting the contextual effects. The debate or discussion may be finally ended by the idea proposed by Duric et al. [3], which is known as W5+ formalization and incorporates context corresponing to the answering of the following important aspects of context: Who you are with (e.g. dyadic/multiparty interactions), What is communicated (e.g., (non)-linguistic message), How the information is communicated (the persons affective cues), Why, i.e., in which context the information is passed on, Where the user is, What his current task is, How he/she feels (has his mood been polarized changing from negative to positive?) and which (re)action should be taken to satisfy humans needs).
Unfortunately, so far the efforts on human affective behavior understanding are usually context independent [6]. In light of these observations, understanding the process of a natural progression of context-related questions when people interact in a social environment could provide new insights into the mechanisms of their interaction context and affectivity. The Who, What, Where context-related questions have been mainly answered either separately or in groups of two or three using the information extracted from multimodal input streams [6]. Thus, as of date, no general W5+ formalization exist, due to the fact that current systems which answer to most of the W questions are founded on different psychological theories of emotion. Recent research on progressing to the questions of "Why" and "How" has led to the emerging field of sentiment analysis, through mining opinions and sentiments from natural language, which involves a deep understanding of semantic rules proper of a language.
The key objective of my PhD research is to computationally identify, automatically extract and incorporate contextual information into affect aware recognition frameworks, with the aim of identifying context-aware emotional-specific patterns.
To fulfill the need of understanding whether and how context is incorporated in automatic analysis of human affective behavior, we propose a novel contextaware incorporation framework (Fig. 2) which (i): includes detection and extraction of semantic context concepts, (ii): better enriches a number of Psychological Foundations with sentiment values and (iii): enhances emotional models with context information and context concept representation in appraisal estimation [11].
As a first step, our preliminary results are focused on bridging the gap at concept level by exploiting semantics, cognitive and affective information, associated with the image verbal content (semantics), which for the needs of our research is the contextual interactional information between the user and the operator of the SEMAINE database [8], keeping fixed the "Where" context-related question. This context concept-based annotation method, that we are examining, allows the system to go beyond a mere syntactic analysis of the semantics associated with fixed window sizes 1 . In most of traditional annotation methods, emotions and contextual information are not always inferred by appraisals and thus contextual information about the causes of that emotion is not taken into account [6].
In this section, our preliminary results of our proposed semantic context concept extractor, are described in details in [11], and their application to indicative examples validate our proposed approach are presented: A. Data Corpus: The model here is confronted with the SEMAINE corpus [8]. This audiovisual corpus comprises manually-transcribed sessions with natural emotional displays. These sessions are recordered from two individuals, an operator and a user, interacting through teleprompter screens from two different rooms. The emotions were elicited with the sensitive artificial listener (SAL) framework, where the operator assumes four personalities aiming to elicit positive and negative emotional reactions from the user. Agent's utterances are constrained by a script, however, some deviations to the script occur in the database. B. Pre-Processing: The pre-processing submodule firstly interprets all the affective valence indicators usually contained in the verbal content of transcriptions, such as special punctuation, complete upper-case words, exclamation words and negations. Handling negation is an important concern in such scenario, as it can reverse the meaning of the examined sentence. Secondly, it converts text to lower-case and, after lemmatizing it, splits the sentence into single clauses according to grammatical conjunctions and punctuation.
These n-grams are not used blindly as fixed word patterns but are exploited as reference for the module, in order to extract multiple-word concepts from information-rich sentences. So, differently from other shallow parsers, the module can recognize complex concepts also when irregular verbs are used or when these are interspersed with adjectives and adverbs, for example, the concept "buy easter present" in the sentence "I bought a lot of very nice Easter presents". C. Semantic context concept parser: The aim of the semantic parser is to break sentences into clauses and, hence, deconstruct such clauses into concepts. This deconstruction uses lexicons which are based on sequences of lexemes that represent multiple-word concepts extracted from ConceptNet, WordNet [9] and other linguistic resources.
Under this view, the Stanford Parser 2 has been used according to Python NLTK 3 ; a general assumption during clause separation is that, if a piece of text contains a preposition or subordinating conjunction, the words preceding this function or are interpreted not as events but as objects. Secondly, dependency structure elements are processed by means of Stanford Lemmatizer for each sentence. Each potential noun chunk associated with individual verb chunks is paired with the stemmed verb in order to detect multi-word expressions of the form "verb plus object". The pos-based bigram algorithm extracts concepts, but in order to capture event concepts, matches between the object concepts and the normalized verb chunks are searched. It is important to build the dependency tree before lemmatization as swapping the two steps result in several imprecisions caused by the lower grammatical accuracy of lemmatized sentences. Each verb and its associated noun phrase are considered in turn, and of more concepts is extracted from these. D. Opinion and Sentiment Lexicon: Current approaches to concept-level sentiment analysis mainly leverage on existing affective knowledge bases such as ANEW, WordNet-Affect and SentiWordNet [4]. However, for the needs of our current work, we use the SentiWordNet, which is a concept-level opinion lexicon and contains multi-word expressions labeled by their polarity scores. 4
My future work will focus on the integration of previous preliminary findings and insights on context affect aware methodologies w.r.t. the production, intepretation and analysis level respectively. To achieve this goal, we have to deal with the following specific challenges: 1) Extraction and Representation: How can we extract and computationally measure contextually rich features w.r.t. the verbal content? 2) Learning: What are the proper learning methods needed to build models? 3) Incorporation: At what time unit should we make an emotion inference (e.g. at the frame or utterance level) and how should we measure the performance of our system?
In Affective Computing, all of the state-of-the-art algorithms perform well on individual sentences without considering any context information, but their accuracy is dramatically lowered because they fail to consider context and the syntactic structure of the verbal content (transcriptions) at the same time. Based on the assumption that the context around a sentence or pair of sentences also plays an important role in determining sentiment, I plan to employ in my experiments a conditional random field (CRF) model [7] to capture syntactic, structural and contextual features of sentences. To this aim, I propose to utilize the SEMAINE and RECOLA datasets [8,10] for both human-agent and human-human social and situational interactions5 , as they provide continuous-time dimensional labels (valence-activation dimensional space).
In the learning stage, I plan to use discriminative learning methods and learn patterns of contextual emotion-specific segments in a supervised manner. My hypotesis is that training contextual rich classification systems using segments lacking clarity may lead to lower analysis rates, and using only segments with high clarity will lead to improved performance as well as more efficient training due to the decrease in training data. I will first extract contextual rich audiovisual descriptors of the above mentioned emotional corpora [8,10] and then I will utilize these "socially contextually rich" segments with consistent emotional cues and their emotional labels to train a discriminative model. In the case of the RECOLA dataset, in which only the first 5 minutes have been annotated, I will use the active learning method to build classifiers using less tarining data for expanding labeled data pool.
Finally, the goal at the interpretation stage is to show that incorporating context can both improve the system's performance and disambiguate multimodal behaviors. To this aim, I propose to first identify "socially contextually rich" segments of the test data and incorporate emotion at the segment level using the learned weights of discriminative models. I plan to conduct both classification and regression tasks. I hypothesize that the classification of the "socially contextually rich" segments can increase the regression of all the data, since the captured and longer-range information may be more useful. I will predict continuouas-valued labels in dimensional space using regression models such as Support Vector Regression. Also, I will cluster and define new emotion classes from the dimensional labels and identify classes using Support Vector Machines (SVM). The SVM outputs will be combined to infer dimensional space outputs.
In this paper, we point out the motivation and importance of identifying or defining the incorporation of context in Affective Computing systems, especially when the disambiguation of a multimodal behavior depends on the contextual interactional setting. Afterwards, we propose the context incorporation framework based on semantic concept detection, extraction enrichment and representation in cognitive estimation. And finally, we provide relevant analysis and conclude to future work that needs to be developed. In future work, we would like to explore the interpretation of contextual effects on affect production, interpretation and analysis respectively.
This research has been co-financed by the European Union (European Social Fund -EFS) and Greek national funds through the Operational Program "Education and Lifelong Learning" of the National Strategic Reference Framework (NSRF) -Research Funding Program: Thales. Investing in knowledge society through the European Social Fund.