The research and development of a fully functioning socially intelligent service is still at a very early stage. However, various components that will ultimately enable such services are under intensive development for several decades. We brie y present them grouped according to the following subsections. 2.1

There are many de nitions of social intelligence applicable in this context [ 23 ]. The wider de nition used here is by Vernon [ 44 ], who de nes social intelligence as the person's "ability to get along with people in general, social technique or ease in society, knowledge of social matters, susceptibility to stimuli from other members of a group, as well as insight into the temporary moods or underlying personality traits of strangers". Furthermore, social intelligence is demonstrated as the ability to express and recognize social cues and behaviors [ 2 ], [ 6 ], including various non-verbal cues (such as gestures, postures and face expressions) exchanged during social interaction [ 47 ].

Social signals are extensively being analyzed in the eld of human to computer interaction [ 47 ], [ 46 ], often under di erent terminology. For example, [ 33 ] use the term 'social signals' to de ne a continuously available information required to estimate emotions, mood, personality, and other traits that are used in human communication. Others [ 31 ] de ne such information as 'honest signals' as they allow to accurately predict the non-verbal cues and, on the other hand, one is not able to control the non-verbal cues to the extent one can control the verbal form. Here, we will use the term social signal. 2.2

Several services exist that support some level of social intelligence, ranging from emotion-aware to meta-cognitive. One of the more relevant examples is the intelligent tutoring system AutoTutor/A ective AutoTutor [ 15 ]. AutoTutor/A ective AutoTutor employs both a ective and cognitive modelling to support learning and engagement, tailored to the individual user [ 15 ]. Some other examples include: Cognitive Tutor [ 7 ] { an instruction based system for mathematics and computer science, Help Tutor [ 3 ] { a meta-cognitive variation of AutoTutor that aims to develop better general help-seeking strategies for students, MetaTutor [ 9 ] { which aims to model the complex nature of selfregulated learning, and various constraint-based intelligent tutoring systems that model instructional domains at an abstract level [ 28 ], among many others. Studies on a ective learning indicate the superiority of emotion-aware over nonemotion-aware services, with the former o ering signi cant performance increase in learning [ 37 ], [ 22 ], [ 43 ]. 2.3

One of the core requirements for socially intelligent service is the ability to detect and recognize emotions, and exhibit the capacity for expressing and eliciting basic a ective (emotional) states. Most of the literature in this area is dedicated to the a ective computing and computational models of emotion [ 26 ], [ 25 ], [ 34 ], which are mainly based on the appraisal theory of emotions [ 48 ]. Several challenges remain, most notably the design, training and evaluation of computational models of emotion [ 20 ], their critical analysis and comparison, and their relevancy for other research elds (e.g., cognitive science, human emotion psychology), as most computational models of emotion are overly simplistic [ 12 ]. 2.4

The development of wearable sensors enabled the acquisition of user data in near-real-time, as well as the research and estimation of user's internal states (such as emotion and stress level estimation) that started more than a decade ago [ 5 ], [ 4 ]. Notable advances can also be found in the elds of psychological computing and HCI, with the development of several novel measurement related procedures and techniques. For example, psychophysiological measurements are being employed to extend the communication bandwidth and develop smart technologies [ 18 ], along with the design guidelines for conversational intelligence based on the environmental sensors [ 14 ]. Several studies deal with human stress estimation [ 36 ], workload estimation [ 30 ], cognitive load estimation [ 27 ], [ 8 ], among others, and speci c learning tasks related to physiological measurements [ 49 ], [ 21 ]. 2.5

The eld of a ective computing has developed several approaches to modeling, analysis and interpretation of human emotions [ 19 ]. The most known and widely used emotion annotation and representation model is the Valence-ArousalDominance (VAD) emotion space, an extension of Russell's valence-arousal model of a ect [ 35 ]. The VAD space is used in many human to machine interaction settings [ 50 ], [ 40 ], [ 32 ], and was also adopted in the socially intelligent typing tutor (see section 3.3.2). There are other attempts to de ne models of human emotions, such as speci c emotion spaces for human computer interaction [ 16 ], or more recently, models for the automatic and continuous analysis of human emotional behaviour [ 19 ]. Recent research on emotion perception argues that traditional emotion models might be overly simplistic, pointing out the notion of emotion is multi-componential, and includes "appraisals, psychophysiological activation, action tendencies, and motor expressions" [ 38 ]. Consequently, and relevant to the interpretations of valence in the existing models, some researchers argue there is a need for the "multifaceted conceptualization of valence" that can be linked to "qualitatively di erent types of evaluations" used in the appraisal theories [ 39 ].

Research of emotion elicitation via graphical user interface is far less common. Whereas several studies on emotion elicitation use di erent stimuli (e.g., pictures, movies, music) [ 41 ] and behavior cues [ 13 ], none to our knowledge tackle the challenges of graphical user interface design for the purpose of emotion elicitation.

In the intelligent typing tutor, user emotions are elicited by the graphical emoticons (smileys) via the dynamic graphical user interface of the service. The choice of emoticons was due to their semantic simplicity, unobtrusiveness, and ease of continuous measurement { using pictures as a stimuli would add additional cognitive load and likely evoke multiple emotions. This approach also builds upon the results of previous research, which showed that human face-like graphics increase user engagement, that the recognition of emotions represented by emoticons is intuitive for humans, and that emotion elicitation based on emoticons is strong enough to be applicable [ 17 ]. The latter assumption is veri ed in this paper. 3.

The following sections discuss the role of emotion elicitation in socially intelligent services and its importance for e cient HMC. General requirements and the role of emotion elicitation are discussed in the context of our study case { the intelligent typing tutor. Later sections present the design of the intelligent typing tutor and its emotion elicitation model. 3.1

A given service is socially intelligent if it is capable of performing the following elements of social intelligence: 1. Read relevant user behavior cues: human emotions are conveyed via behaviour and non-verbal communication cues such as face expression, gestures, body posture, color of the voice, etc. 2. Analyze, estimate and model user emotions and nonverbal (social) communication cues via computational model: behavior cues are used to estimate user's temporary emotion state. Selected physiological measurements (pupil size, acceleration of the wrist, etc.) are believed to be correlated with user's emotion state and other non-verbal communication cues. These are used as an input to the computational model of user emotions and other non-verbal communication cues. 3. Integrate and model machine generated emotion expressions and other non-verbal communication cues: for example, the notion of positive reinforcement could be integrated into a service to improve user engagement, taking into account user's temporary emotion state and other non-verbal communication cues. 4. Generate emotion elicitation to improve user engagement: continuous feedback loop between user emotion state and machine generated emotion expressions for purpose of emotion elicitation. 5. Context and task-dependent adaptation: adapt the service according to the design goals. For example, in the intelligent typing tutor case study, the intended goal is to improve learner's engagement and progress. The touch-typing lessons are carefully designed and adapt in terms of typing speed and di culty to meet individual's capabilities, temporary emotion state and other non-verbal communication cues.

Such service is capable of sustaining e cient, continuous and engaging HMC. It also minimizes user-service adaptation procedures. An early-stage example of socially intelligent service is provided below. 3.2

The overall goal of the socially intelligent typing tutor is to improve the process of learning touch-typing. For this purpose, emotion elicitation is integrated into HMC together with the notion of positive reinforcement, to amplify the attention, motivation, and engagement of the individual learner. In its current form, the rudimentary model of emotion elicitation utilizes emoticon-like graphics via the graphical user interface of the service, presented to the learner in real-time (see section 3.3). The tutor uses state-of-the-art technology (3.2.1) and is able to model, measure and analyze emotion elicitation throughout the tutoring process. 3.2.1

The role of emotion elicitation in the intelligent typing tutor is that of e cient HMC and reward system. The positive reinforcement assumption [ 29 ] is used in the design of the emotion elicitation model. Positive reinforcement argues that learning is best motivated by a positive emotional responses from the service when learners ratio of attention over fatigue goes up, and vice versa. Here, machine generated positive emotion expressions act as rewards, with the aim to improve learner's attention, motivation and engagement during the touch-typing practice. The learner is rewarded by a positive emotional response from the service when she invest more e ort into practice (the service does not support negative reinforcement). According to the positive reinforcement assumption, the rewarded behaviors will appear more frequently in the future. Negative reinforcement is not used for two reasons: there is no clear indication how negative reinforcement would contribute to the learning experience, and it would require an introduction of additional dimension, making the research topic of the experiment even more complex. 3.3.1

The following sections give an overview of the user experiment and results on emotion elicitation in the intelligent typing tutor. 4.1

The experiment consisted of 32 subjects invited to practice touch-typing in the intelligent typing tutor (see 3.2), with the average duration of the typing session approx. 17 minutes (1020 seconds). The same set of carefully designed touch-typing lessons was given to all test subjects. User data was acquired in real-time using sensors (as described in section 3.2), and used as an input to the computational model of machine generated emotion expressions, and recorded for later analysis. For the preliminary analysis presented here, ve randomly selected subjects were analysed on the segment of the overall duration of the experiment.3 The test segment spans from 6 to 11:5 mins (330 seconds) of the experiment.

The test segment used for the analysis is composed of the two primary dimensions for measuring emotion elicitation. 3To simplify the presentation of the experiment results. Note that similar results were found for the remaining subjects. following steps:4 1. Instructions are given to the test users: users are personally informed about the goal and the procedure of the experiment (by the experiment personnel); 2. Setting up sensory equipment, start of the experiment: a wrist accelerometer is put on, the video camera is set on, and the experimental session time recording is started (at 00 seconds); 3. At 60 seconds: machine generated sound disruption of the primary task: "Name the rst and the last letter of the word: mouse, letter, backpack, clock"; 4. At 240 seconds: machine generated sound disruption of the primary task, "Name the color of the smallest circle", in the gure (Fig 2). This cognitive task is expected to signi cantly disrupt learner's attention away from the typing exercise; 5. The test segment ends at 330 seconds.

During the experiment, users' emotion expressions are analyzed using Noldus Observer video analysis software http: //www.noldus.com. The recordings are in sync with the machine generated emoticons, readily available for analysis (see next section 4.2). 4.2

The analysis of the experimental data was conducted to measure the e ectiveness of emotion elicitation. The x-axis times for all graphs presented below are relative in seconds [s], for the whole duration of the test segment (330 seconds). The estimation is based on the emotion elicitation model (1) tting. To detect the time when the emotion elicitation is present, we conducted the null hypothesis testing H0 = [R2 = 0] at risk level = 0:05. The emotion elicitation is determined as present where the null hypotheses is rejected, and not present otherwise.

An example of valence and arousal ratings for a randomly selected subject is shown in Fig. 3.

The model (1) is tted using linear regression on the measured data for the duration of the test segment. The data is 4Due to limited space, the two disruption parts of the experiment (Steps 3. and 4.) are not further discussed. sampled in a non-uniform manner due to the technical properties of the sensors (internal clocks of sensors are not sufciently accurate, etc.). The data is approximated by continuous smooth B-splines of order 3, according to the upper frequency limit of measured phenomena, and uniformly sampled to time-align data (we skip re-sampling details here).

To t the regression models the 40 past samples from the current (evaluation) time representing 4 seconds of realtime were used. These two value were selected as an optimum according to competitive arguments for more statistical power (requires more samples) and for enabling to detect time-dynamic changes in the e ectiveness of emotion elicitation (requiring shorter time interval leading to less samples). Note that changing this interval from 3 to 5 seconds did not signi cantly a ect the tting results. Results are given in terms of RV2 , RA2 representing the part of explained variance of valence and arousal when the elicitation is known, and in terms of a pV , pA-values testing the null hypothesis regression models H0V = [RV2 = 0], H0A = [RA2 = 0], respectively. The time dynamics of emotion elicitation is represented by p-values pA and pV on Fig. 4.

In order to estimate the e ect of emotion elicitation, the percentages were computed on the number of times the elicitation was signi cant. The analyzed time intervals were uniformly sampled every 2 seconds. The results are shown in Table 1. It turned out that the test interval sampling had no signi cant impact on the results.

P-values: arousal 50 100 150 200 250 300 350

We also analyzed the reduced percentages. These are 5% lower than the measured ones, since the signi cance testing was performed at a risk level = 0:05 and approximately 5% detections are false (type I. errors). Note that Bonferroni correction does not apply here. However, we nevertheless computed the above given percentages using Bonferroni correction and it turned out the percentages drop approximately to one half of the reported values.

The strength of emotion elicitation is shown in the linear regression model R2 as a function of time (Fig. 5). 50 100 250 300

350

The strength of emotion elicitation e ect is signi cant, but also varies highly (Fig. 5). Similar results were detected among all test subjects. However, it is too early to draw any meaningful conclusions on the reasons for high variability at this stage, as many of the potential factors in uencing emotion elicitation need further analysis.

To estimate the average strength of emotion elicitation, the average values of R2 were computed for the ve selected subjects (as in Table 1) { these values are part of the explained variance for learner emotions when the machine generated emotion is known. The average value of R2 varies across test subjects from 18:3% to 24:5% for valence and 19:7% to 31:4% for arousal, for all time intervals (when signi cant or non-signi cant elicitation is present). If we average only over the time intervals when the elicitation is signi cant, the average value of R2 varies across test subjects from 32:5% to 39:3% for valence and 36:3% to 44:9% for arousal (see Table 2).

Observe that there is considerably less variability among the subjects in terms of elicitation strength (average R2), compared to the proportions of time the elicitation is significant (see Table 1).

The paper discussed the e ciency of emotion elicitation in socially intelligent services. The experiment was conducted using the socially intelligent typing tutor. The overall aim of the intelligent typing tutor is to elicit emotions and thus improve learning and engagement in the touch-typing training. Emotion elicitation is utilized together with the notion of positive reinforcement. The tutor is able to model and analyze learner's expressed emotions and measure the e ciency of emotion elicitation in the process. Experimental results show that the e ciency of emotion elicitation is signi cant, but at times also varies highly for the individual learner and moderately among learners.

Future work will focus on reasons for variations in emotion elicitation by analyzing potential factors, such as the e ects of machine generated emotion expressions on emotion elicitation, learner's emotional state, cognitive load, attention, and engagement, among others.