A crowdsourcing toolbox for a user-perception based design of social virtual actors

Abstract

One of the key challenges in the development of social virtual actors is to give them the capability to display socio-emotional states through their non-verbal behavior. Based on studies in human and social sciences or on annotated corpora of human expressions, di↵ erent models to synthesize virtual agent's non-verbal behavior have been developed. One of the major issues in the synthesis of behavior using a corpus-based approach is collecting datasets, which can be di cult, time consuming and expensive to collect and annotate. A growing interest in using crowdsourcing to collect and annotate datasets has been observed in recent years. In this paper, we have implemented a toolbox to easily develop online crowdsourcing tools to build a corpus of virtual agent's non-verbal behaviors directly rated by users. We present two developed online crowdsourcing tools that have been used to construct a repertoire of virtual smiles and to define virtual agents' non-verbal behaviors associated to social attitudes.

-

Virtual agents are increasingly used in roles that are typically fulfilled by humans, such as tutors in virtual learning class, assistants for virtual task realization, or play-mate in video game (e.g. [ 1, 2 ]). To embody successfully these social roles, virtual agents have to be able to express socio-emotional behavior during humanmachine interaction. Indeed, several researches have shown that the expressions of emotion may enhance not only the believability of the virtual agent [ 3 ] but also the satisfaction of the user and his performance in task achievement [ 4, 5 ].

One of the key challenges in the development of embodied virtual agents is to give them the capability to display socio-emotional states through their nonverbal behavior. Several virtual agents are already able to express emotions or social attitudes through di↵ erent modalities such as facial expressions, gestures or postures [ 6–9 ]. We can distinguish two main approaches to define the virtual agent’s non-verbal behaviors associated to socio-emotional states: a theoreticalbased approach and a corpus-based approach.

The theoretical-based approach consists in exploiting the studies in human and social sciences that have highlighted the characteristics of human’s nonverbal behavior conveying socio-emotional state. For the expressions of emotion, most of the computational models are based on the categorical approach proposed by Ekman [ 10 ] describing the human facial expressions of the “big six” basic emotions (joy, fear, anger, surprise, disgust, and sadness) [ 11 ]. Other psychological theories have been explored to define the emotional facial expressions of virtual agents, such as the dimensional theory [ 12 ] or the appraisal theory [ 13 ].

To gather more subtle and natural expressions, another approach is based on the analysis of annotated corpora of human expressing socio-emotional states. Based on an annotated corpus of humans expressions, di↵ erent methods to synthesize virtual agent’s non-verbal behavior have been explored. Using a motion capture system, the non-verbal behavior can be synthesized at a very low-level by re-targeting the points tracked on a human face and body to a virtual mesh (e.g. in [ 14 ]). Another method consists in applying machine learning technique on the collected data to automatically generate the non-verbal behavior associated to particular socio-emotional state (e.g. in [ 15 ]). Finally, the corpus may also be exploited by analyzing in detail the correspondences between the expressed socio-emotional states and the characteristics of the displayed non-verbal behaviors. Rules are then extracted and integrated in virtual agents (e.g. in [ 16 ]). Most of the corpus-based models of virtual agent’s non-verbal behavior is based on corpus of real humans.

Some researchers have proposed to create corpus of virtual agent’s non-verbal behaviors. For instance, in [ 17 ], a large amount of expressive virtual faces has randomly been generated. They have then been rated with emotional labels by numerous participants. This method has several advantages. First, it considers directly the user’s perception of the virtual agent instead of replicating findings of human’s non-verbal behavior on virtual agents. Moreover, a corpus of virtual agent’s non-verbal behaviors avoids the problematic of acted human’s expressions or the di culty to collect spontaneous expressions. Secondly, this method may generate one-to-many correspondences between socio-emotional states and non-verbal behaviors. Thus, in [ 17 ], several facial expressions for each emotion type have been identified. The main problem with this method is the number of required participants (more than 400) for a repetitive and time-consuming task of rating each facial expression (2904 facial expressions). In this article, we propose an alternative methodology to identify the one-to-many correspondences between socio-emotional states and non-verbal behaviors by building and analyzing a corpus of virtual agent’s non-verbal behavior directly rated by users.

One of the major issues in the synthesis of behavior using a corpus-based approach is collecting datasets, which can be di cult, time consuming and expensive to collect and annotate. A growing interest in using crowdsourcing to collect and annotate datasets has been observed in recent years [ 18 ]. Crowdsourcing consists of outsourcing tasks to an undefined distributed group of people, often using Internet to recruit participants informally or through formal paid mechanisms such as Amazon’s Mechanical Turk [ 19 ]. Online tools for crowdsourcing have been developed to allow people to annotate human behaviors (e.g. in [ 20 ]). Moreover, an evaluation of the crowdsourcing workers’ annotations showed that their qualities are comparable to expert annotators [ 20 ]. In order to build a rated corpus of virtual agents’ non-verbal behaviors, we have implemented a toolbox to easily develop online crowdsourcing tools. The objective of such a crowdsourcing tool is to o↵ er the possibility to users to directly configure the virtual agent’s non-verbal behaviors conveying particular socio-emotional states. For instance, the users may have the task to define the virtual agent’s gestures and facial expressions corresponding to the expression of certain attitudes such as friendliness or dominance. This method avoids the traditional approach of creating a repertoire of socio-emotional states by asking users to label a set of predefined non-verbal behaviors. Instead, users are placed at the heart of the non-verbal behavior creation process. Even if a finite set of animations is pre-defined, the tool gives the users the impression to create the non-verbal behavior they believe corresponds to a given socio-emotional state.

To create such a crowdsourcing tool, the toolbox covers di↵ erent functionalities: – the construction of an audiovisual corpus of virtual agent’s non-verbal behaviors containing all the possible combinations of modalities (Section 2); – the framework to develop and distribute the tool online (Section 3). As use cases, we present two developed online crowdsourcing tools that have been used to construct a repertoire of virtual smiles and to define virtual agent’s non verbal behaviors associated to social attitudes (Section 4). We conclude and discuss the limits of this method in Section 5. 2

The GretaModular Platform to create corpora of virtual agents’ non-verbal behaviors The first step to develop the online crowdsourcing tool is to generate the videos of virtual agents displaying di↵ erent combinations of facial expressions, gestures, and postures. For this purpose, the platform we are using to animate a virtual agent is GretaModular, a significantly improved version of the Greta system [ 21 ].

GretaModular o↵ ers several modules, each dedicated to particular functionality. The core modules, based on the SAIBA framework [ 22 ], include an Intent Planner, a Behavior Planner and a Behavior Realizer to compute multimodal expressions of communicative intentions. Additional peripheral modules endow the system with several useful functionalities. For instance, the Gesture Editor and the Facelibrary Viewer enable one to easily define new gestures and facial expressions of virtual agents. Others modules, such the BML and FML file readers, the Character Manager, the Ogre3D Player and the Video Capture module, facilitate the creation of a corpus of virtual agent animations. Moreover, the flexible architecture of the platform has been implemented with a graphical user interface that allows a simple manipulation of the modules by drag and drop. New modules may be easily developed and plug in to add new functionalities to the system.

In GretaModular, a repertoire of signals contains the description of non-verbal behavior (facial expressions, gestures, postures) in BML (Behavior Markup Language) [ 23 ]. The links between non-verbal behaviors and communicative intentions are specified in a lexicon. Communicative intentions are encoded with FML (Function Markup Language) [ 24 ]. Furthermore, expressivity parameters can be used to modulate the qualitative execution of non-verbal behaviors (e.g. fluidity of gestures) [ 25 ].

Using GretaModular, one may create a corpus in 5 steps (Figure 1). The appearance of the virtual agent has to be first chosen. The physical appearance of the virtual agent may be selected in the Character Manager module of GretaModular. The available virtual agents are illustrated Figure 2. The second step consists in creating or completing the repertoire of signals in BML. In GretaModular, a large set of signals has already been defined for the virtual agents of the platform (Figure 2). One may easily design new signals using the Gesture Editor and the Facelibrary Viewer of the GretaModular platform. Thirdly, the di↵ erent links between non-verbal behaviors and communicative intentions are defined in the lexicon. Moreover, di↵ erent expressive parameters of the virtual agent’s non-verbal behavior may be configured. To generate the videos of the corpus, we have developed a specific module, named the Plan Capture Controller Module, built on the top of the video capture module. This module takes as input an FML file describing the communicative intentions (e.g. emotions, beliefs) that the virtual agent has to express through its verbal and non-verbal behavior [ 21 ]. Given that communicative intention may be expressed through di↵ erent signals and modalities, the module computes and loads all the possible animations corresponding to the FML file (based one the lexicon in which the correspondences between communicative intentions and behaviors are described), plays them and captures each one of them in separate video files. For instance, the intention to greet may be expressed by a head nod or a hand shake, with or without a smile. For this communicative intention, the Plan Capture Controller module creates 4 di↵ erent video files of a virtual agent that greets in 4 di↵ erent manners. The Plan Capture Controller module may generate the animations with di↵ erent values of the expressive parameters. Finally, with GretaModular, one may rapidly create a corpus of videos of virtual agents with di↵ erent appearances displaying multimodal non-verbal behaviors with di↵ erent expressive parameters.

So far, we have created two corpora of virtual agent’s non-verbal behaviors. A first corpus has been dedicated to the virtual agent’s smiles. One hundred and ninety two di↵ erent animations of a smiling virtual agent face have been generated. The smiles varied on several morphological and dynamic parameters defined from the theoretical and empirical research on human smiles [ 26–28 ]: the cheek raising (Action Unit 6 - AU6), the lip press (Action Unit 24 -AU24), the amplitude of the smile (Action Unit 12 - AU12), the symmetry of the lip corners, the mouth opening (Action Unit 25 - AU25), the duration of the smile and the velocity of the rise and of the decay of the smile. We have considered two or three discrete values for each of these parameters: small or large smile (for the amplitude); open or close mouth; symmetric or asymmetric smile; tensed or relaxed lips (for the AU24); cheekbone raised or not raised (for the AU6); short (1.6 seconds) or long (3 seconds) total duration of the smile, and short (0.1 seconds), average (0.4 seconds) or long (0.8 seconds) beginning and ending of the smile (for the rise and decay). All the possible combinations of these discrete values have been generated to create the corpus of the virtual agent’s smiles.

A second corpus has been created to study the non-verbal behaviors conveying the social attitudes of dominance, submissiveness, friendliness and unfriendliness. For this purpose, we have generated 1440 videos corresponding to all the possible combinations of the following parameters identified as cues of social attitudes [ 29–33 ]: type of facial expressions (positive: smile, negative: frown or neutral), the activated modalities (arm gestures, head gestures, both or none), the amplitude of arm gestures (small, medium or wide), the power of arm gestures (weak, normal or strong), the head position (upward, downward, tilted aside or straight) and the presence of gaze avoidance (yes or no). The corpus contains animations of two di↵ erent virtual agents, one with a female appearance and one with a male appearance.

Online Crowdsourcing tools for the user design of virtual agent’s non-verbal behaviors In order to create crowdsourcing tools based on virtual agent’s non-verbal behaviors corpora, we have created a framework using Flash technology to enable broad distribution on the web. The framework allows one to develop a web application in which users have the task to define the non-verbal behaviors of a virtual agent associated to particular socio-emotional states. The interface of the application is composed of 4 parts (Figures 3 and 4): 1. the upper part contains a description of the task; 2. the left part contains a video showing the virtual agent animation, in a loop; 3. the right part contains a panel with the di↵ erent non-verbal parameters that the user can change to define the virtual agent’s non-verbal behavior. Any time the user changes the value of one of the parameters, a corresponding video is automatically played; 4. the bottom part contains a Likert scale that allows users to indicate their satisfaction with the created animation.

To develop the crowdsourcing tool, one has to define the tasks of the users. The panel of the non-verbal parameters has to correspond to the parameters considered in the creation of the corpus of virtual agents’ non-verbal behaviors (Section 2). The videos displayed according to the selected parameters are directly extracted from the corpus. The Flash framework includes a connection with a database to record the responses of the users.

Using this framework, two crowdsourcing tools have been developed: E-Smilescreator and GenAttitude. The interfaces of these tools are illustrated in Figures 3 and 4. The objective with the E-smiles-creator tool is to study the morphological and dynamic characteristics of di↵ erent smile types. Through the interface of the E-smiles-creator (Figure 3), the users have the tasks to create di↵ erent types of smile (amused, polite, and embarrassed). To create each of these smiles, the users select the parameters of the smile with the radio buttons (Panel 3, Figure 3). These parameters correspond to those used to create the corpus of virtual smiles. The corresponding video contained in the corpus is automatically loaded and played (Panel 2, Figure 3). With the GenAttitude tool (Figure 4), the objective is to identify virtual agents’ non-verbal behaviors corresponding to the expression of di↵ erent attitudes. The users have the tasks to configure the non-verbal behavior of the virtual agent corresponding to the expression of a particular social attitude (dominant, submissive, hostile, or friendly) for a given communicative intention. For instance, the users have the tasks to configure the non-verbal behavior of the agent when it is asking something with a dominant attitude. The parameters of the non-verbal behavior (Panel 3, Figure 4) correspond to those used to create the corpus of videos (Section 2).

Finally, with the crowdsourcing tools, users unconsciously rated videos of the corpus (of the virtual agents’ non-verbal behaviors) with pre-defined labels (emotions or social attitudes). However, not all the videos of the corpus are rated. Only those that appear as relevant for the pre-defined labels are rated. 4

Analysis of the collected data on virtual agents’ non-verbal behaviors Through the developed crowdsourcing tools presented above, we have collected 1044 smile descriptions (from 348 participants among which 195 females; mainly French, with a mean age of 30 years) and 925 non-verbal behavior descriptions corresponding to social attitudes (from 170 participants among which 50 females, mainly French, with a mean age of 29 years), in one week. The participants were recruited via online mailing lists (they were not payed). The average level of satisfaction of the participants (5,3 on a Likert Scale of 7 points, Panel 4) shows that the participants were globally satisfied by the interface to create the animations. Moreover, the positive comments posted by the participants show that their user experience was funny and enjoyable. We have analyzed the collected data to create a repertoire of non-verbal behaviors conveying di↵ erent emotions and social attitudes.

The levels of satisfaction indicated by the participants (Panel 4, Figures 3 and 4) was used to give higher weight to the non-verbal behaviors with a high level of satisfaction1. We made the assumption that the non-verbal behaviors with a high level of satisfaction were more reliable that those with low level. In fact, we oversampled the corpora such as each created non-verbal behavior was duplicated n times, where n is the level of satisfaction associated with this non-verbal behavior. For instance, a smile with a level of satisfaction of 7 was duplicated 7 times whereas a smile with a level of satisfaction of 1 was not duplicated. The resulting data sets were composed of 5517 descriptions of smiles and 4947 non-verbal behavior descriptions conveying social attitudes.

To analyze the collected data and construct computational models, di↵ erent methods have been explored. Concerning smiles, we used a decision tree learning algorithm to identify the di↵ erent characteristics of the amused, polite, and embarrassed smiles in the corpus. The decision tree has the advantage to be well-adapted to qualitative data and to produce results that are interpretable and that can be easily implemented in a virtual character. The nodes of the decision tree correspond to the smile characteristics and the leaves are the smile types. Di↵ erent leaves correspond to the same smile type enabling one to identify one-to-many correspondences between smile types and facial expressions. The virtual agent may then express the same type of smile in di↵ erent manners during an interaction to avoid repetition of the exact smile pattern. Previous research has shown that the non-repetitive behaviors of a virtual agent improves its perceived believability [ 6 ]. A perceptive study has validated most of the smiles as appropriate in amused, polite or embarrassed situations. The smiles decision tree and the validation study are described in more details in [ 34 ]. 1 Note that the data could be analyzed without oversampling.

For the second corpus, we have explored another method of analysis: the corpus of the virtual agents’ non-verbal behaviors associated to social attitudes has been used to create a Bayesian network. A Bayesian network is a directed acyclic graph that represents causal relations between variables, the strength of these relations being represented by conditional probabilities. The structure of the network has been defined based on a statistical analysis of the corpus. The input nodes of the model are the social attitudes (dominant, submissive, friendly, or hostile) and the communicative intentions. The outputs are the characteristics of the non-verbal behavior that should convey a given communicative intention with a particular attitude. The Bayesian network directly represents the causee↵ ect relations between our input variables (communicative intentions and social attitudes) and output variables (the non-verbal behavior parameters). The parameters of the model (i.e. the probability of the edges) are directly extracted from the built oversampled corpus. The probabilistic nature of such a model enables us to introduce variabilities in the outputs, particularly relevant for modeling human-like uncertain behavior. Once again, the model may be used to determine one-to-many correspondences between attitudes and non-verbal behaviors. Moreover, the model provides a probability that the virtual agent’s non-verbal behavior is perceived with the expected attitude. Also, the same Bayesian network can be used to infer the probabilities for the input variables given the output values. This could be use to retrieve the most likely attitude and intention given the nonverbal behavior parameter values. The Bayesian model is detailed in [ 35 ]. 5

Conclusion In this article, we have presented a toolbox that enables one to create a crowdsourcing tool to build corpus of virtual agents’ non-verbal behaviors. We have presented two use cases that aimed at analyzing the links between the characteristics of a signal (the smile) or of a multimodal behavior to the expression of emotions or social attitudes. Instead of asking users to rate a set of virtual agent animations, we have proposed an approach in which the user has the impressions to directly design the virtual agent’s non-verbal behavior. The large number of participants, their levels of satisfaction and their positive posted comments indicate that the proposed tasks and interface are satisfying and attractive. The size of the obtained corpus enables one to apply di↵ erent methods from a statistical analysis to machine learning techniques.

In future works, we aim at improving the interfaces of the crowdsourcing tools. In the current version, the values of the non-verbal behavior’s parameters are selected through radio buttons. Continuous values indicated with sliders could enable us to obtain a more fine-grained description of the virtual agent’s non-verbal behaviors. Moreover, the Bayesian network model resulting from the GenAttitude tool has to be evaluated during interaction with users to ensure that the non-verbal behaviors convey the expected attitudes. Acknowledgment This research has been supported by the European projects NoE SSPNet and IP REVERIE. The authors would like to thank Andr´e-Marie Pez and Pierre Philippe for the implementation of the GretaModular platform.

References

1. Johnson , W.L. , Rickel , J.W. , Lester , J.C. : Animated pedagogical agents: Faceto-face interaction in interactive learning environments . International Journal of Artificial Intelligence in Education 11 ( 2000 ) 47 - 78

2. Andr´e, E., Klesen , M. , Gebhard , P. , Allen , S. , Rist , T. : Integrating models of personality and emotions into lifelike characters . In: A↵ ective interactions , Springer ( 2001 ) 150 - 165

3. Bates , J.: The role of emotion in believable agents . Communication of ACM 37 ( 7 ) ( July 1994 ) 122 - 125

4. Beale , R. , Creed , C. : A↵ ective interaction: How emotional agents a↵ ect users . International Journal of Human-Computer Studies 67 ( 9 ) ( 2009 ) 755 - 776

5. Partala , T. , Surakka , V. : The e↵ ects of a↵ ective interventions in human-computer interaction . Interacting with Computers 16 ( 2 ) ( 2004 ) 295309

6. Niewiadomski , R. , Hyniewska , S. , Pelachaud , C. : Constraint-based model for synthesis of multimodal sequential expressions of emotions . IEEE Transactions on A↵ ective Computing 2 ( 3 ) ( 2011 ) 134 - 146

7. Bickmore , T.W. , Picard , R.W.: Establishing and maintaining long-term humancomputer relationships . ACM Trans. Comput.-Hum. Interact . 12 ( 2 ) ( June 2005 ) 293 - 327

8. Kasap , Z. , Moussa , M.B. , Chaudhuri , P. , Magnenat-Thalmann , N.: Making them remember emotional virtual characters with memory . IEEE Computer Graphics and Applications (March 2009 ) 20 - 29

9. Albrecht , I. , Schroder , M. , Haber , J. , Seidel , H.: Mixed feelings : expression of non-basic emotions in a muscle-based talking head . Special issue of Journal of Virtual Reality on Language, Speech and Gesture 8 ( 4 ) ( 2005 ) 201 - 212

10. Ekman , P. , Friesen , W.V. : Unmasking the Face. A guide to recognizing emotions from facial clues . Prentice-Hall, Inc., Englewood Cli↵ s, New Jersey ( 1975 )

11. Ostermann , J.: Face animation in mpeg-4 . In Pandzic, I., Forchheimer , R., eds. : MPEG-4 Facial Animation - The Standard Implementation and Applications . Wiley, England ( 2002 ) 17 - 55

12. Mehrabian , A. : Basic Dimensions for a General Psychological Theory: Implications for Personality, Social, Environmental, and Developmental Studies . Oelgeschlager, Gunn & Hain, Cambridge, Mass ( 1980 )

13. Scherer , K. : Appraisal processes in emotion: Theory, methods , research. ( 2001 ) 92 - 119

14. Niewiadomski , R. , Pelachaud , C. : Towards multimodal expression of laughter . In: The 12th International Conference on Intelligent Virtual Agents, Santa Cruz , USA ( 2012 ) 231 - 244

15. Ding , Y. , Radenen , M. , Artiere , T. , Pelachaud , C. : Speech-driven eyebrow motion synthesis with contectual markovian models . In: To appear in Proc. of ICASSP . ( 2013 )

16. Castellano , G. , Mancini , M. , Peters , C. , McOwan , P. : Expressive copying behavior for social agents: A perceptual analysis . IEEE Transactions on Systems, Man and Cybernetics 42 ( 3 ) ( 2012 ) 776 - 783

17. Grammer , K. , Oberzaucher , E. : The reconstruction of facial expressions in embodied systems . ZiF : Mitteilungen ( 2006 )

18. Yuen , M.C. , King , I. , Leung , K.S.: A survey of crowdsourcing systems . In: Proceedings of the IEEE International conference on Social Computing (SocialCom) . (oct. 2011 ) 766 - 773

19. Mason , W. , Suri , S. : Conducting behavioral research on Amazon's Mechanical Turk . Behavior Research Methods 44 ( 1 ) ( June 2011 ) 1 - 21

20. Park , S. , Mohammadi , G. , Artstein , R. , Morency , L.P. : Crowdsourcing micro-level multimedia annotations: the challenges of evaluation and interface . In: Proceedings of the ACM multimedia 2012 Workshop on Crowdsourcing for multimedia. CrowdMM '12 , New York, NY, USA, ACM ( 2012 ) 29 - 34

21. Niewiadomski , R. , Bevacqua , E. , Mancini , M. , Pelachaud , C. : Greta: an interactive expressive ECA system . In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2. AAMAS '09 , Richland , SC , International Foundation for Autonomous Agents and Multiagent Systems ( 2009 ) 1399 - 1400

22. Kopp , S. , Krenn , B. , Marsella , S. , Marshall , A.N. , Pelachaud , C. , Pirker , H. , Thrisson , K.R. , Vilhjlmsson , H.: Towards a common framework for multimodal generation: The behavior markup language . In: Proceedings of the international conference on Intelligent Virtual Agents (IVA) , Springer-Verlag Berlin, Heidelberg ( 2006 ) 21 - 23

23. Kopp , S. , Krenn , B. , Marsella , S. , Marshall , A. , Pelachaud , C. , Pirker , H. , Tharisson , K. , Vilhjalmsson , H.: Towards a common framework for multimodal generation: The behavior markup language . In Gratch, J., Young , M. , Aylett , R. , Ballin , D. , Olivier , P., eds.: Intelligent Virtual Agents. Volume 4133 of Lecture Notes in Computer Science . Springer Berlin / Heidelberg ( 2006 ) 205 - 217

24. Heylen , D. , Kopp , S. , Marsella , S. , Pelachaud , C. , Vilhjlmsson , H.: The next step towards a function markup language . In: Proceedings of the international conference on Intelligent Virtual Agents (IVA) . ( 2008 )

25. Mancini , M. , Pelachaud , C. : Dynamic behavior qualifiers for conversational agents . In Pelachaud, C. , Martin , J.C. , Andre , E. , Chollet , G. , Karpouzis , K. , Pele , D., eds.: Intelligent Virtual Agents. Volume 4722 of Lecture Notes in Computer Science . Springer Berlin / Heidelberg ( 2007 ) 112 - 124

26. Ambadar , Z. , Cohn , J.F. , Reed , L.I. : All Smiles are Not Created Equal: Morphology and Timing of Smiles Perceived as Amused, Polite, and Embarrassed/Nervous. Journal of Nonverbal Behavior 17 - 34 ( 2009 ) 238 - 252

27. Ekman , P. : Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W.W. Norton & Company edn. ( 2009 )

28. Hoque , M. , Morency , L.P. , Picard , R.W.: Are you friendly or just polite? - analysis of smiles in spontaneous face-to-face interactions . In: Proceedings of the international conference on A↵ ective Computing and Intelligent Interaction (ACII) , Berlin, Heidelberg, Springer-Verlag ( 2011 ) 135 - 144

29. Briton , N.J. , Hall , J.A. : Beliefs about female and male nonverbal communication . Sex Roles 32 ( 1995 ) 79 - 90

30. Burgoon , J.K. , Buller , D.B. , Hale , J.L. , de Turck , M.A. : Relational Messages Associated with Nonverbal Behaviors . Human Communication Research 10 ( 3 ) ( 1984 ) 351 - 378

31. Burgoon , J.K. , Le Poire , B.A. : Nonverbal cues and interpersonal judgments: Participant and observer perceptions of intimacy, dominance, composure, and formality . Communication Monographs 66 ( 2 ) ( 1999 ) 105 - 124

32. Carney , D. , Hall , J. , LeBeau , L.: Beliefs about the nonverbal expression of social power . Journal of Nonverbal Behavior 29 ( 2005 ) 105 - 123

33. Hess , U. , Thibault , P. : Why the same expression may not mean the same when shown on di↵ erent faces or seen by di↵ erent people . In Tao, J., Tan , T., eds.: A↵ ective Information Processing . Springer London ( 2009 ) 145 - 158

34. Ochs , M. , Niewiadomski , R. , Brunet , P. , Pelachaud , C. : Smiling virtual agent in social context . Cognitive Processing , Special Issue on“Social Agents” 13 ( 22 ) ( 2012 ) 519 - 532

35. Ravenet , B. , Ochs , M. , Pelachaud , C. : From a user-created corpus of virtual agent's non-verbal behavior to a computational model of interpersonal attitudes . In: To appear in the proceedings of the Intelligent Virtual Agents (IVA) conference . ( 2013 )