A facial imitation framework for the simultaneous face control of a virtual avatar and a humanoid robot

Abstract

Facial expression imitation (FEI) for humanoid robots is an active research field in the context of human robot interaction (HRI). Virtual avatars can enhance and simplify the experimental HRI setup in terms of cost and performance, avoiding possible long-term mechanical degradation of the physical robot in use. Moreover, the presented framework allows to conduct comparison studies aimed at investigating the role of embodiment in the interaction with a robot versus its digital twin, which is a critical factor to establish a successful social bond with the robot, as in the case of numerous clinical applications.

Keywords

• Human-robot interaction
• facial expression imitation
• virtual avatar
• Facial Action Coding System (FACS)

1. Introduction

CEUR

ceur-ws.org provides the opportunity to assess the value of embodiment in the context of emotional communication between a human being and an artificial interlocutor.

2. Proposed work

As shown in Fig.1, the proposed framework is composed of three main systems: the real-time acquisition of images from a camera, the analysis of the extracted images to obtain the Action Units (AUs) of the detected subject, the send and execution of the facial movements to the avatar and to the physical robot. In the acquisition phase, tFrhaeme Grabber (FG, Algorithm1), when executed, asks the user to input the desired frame acquisition ra( t=e 5 fps if not specified) and the name of the folder where the frames will be stored. TsheetupFolder() function is then called, creating both a main folder and a temporary one for storing the frames. If folders already exist, the program informs the user that specified folders already exists. Next, tsheetupCamera() function is called, initializing the webcam and configuring its resolution (wi dt=h 640 pixel, heightℎ = 480 pixel) and frame rate. Finally, the getFrames() function starts capturing frames from the webcam. For each captured frame, a unique name is generated, including the frame number and timestamp. The frame is then saved as an image in the temporary folder and copied to the main folder. This process continues until the user interrupts the program with a keyboard interruption. When this occurs, the program disconnects the webcam, closes alOlpenCV windows, and removes both the main and temporary folders. If an error occurs during the frame capture (e.g., if the webcam is unavailable), the program notifies the user that it cannot initiate a new webcam recording. The second program, i.eE.,vtehnet Handler (EH, Algorithm2) is a file monitoring system that responds to the creation of new files by sending them to a server for processing and subsequently forwarding the results to another server. Itwautscehsdtohgemodule to monitor a specified directory for new files. The program starts by defining the directory to monitor and then enters a waiting loop until the specified directory exists. Once this condition is met, it begins monitoring it using thOneMyWatch class. This class utilizes the Observer class fromwatthcehdog module to monitor the directory. OTnhMeyWatch class has a run method that initiates the observation and waits for file system events. When a file system event is detectedo,nt_haeny_event() method of theHandler class is called. This method checks if the event corresponds to the creation of a new ifle. If a new file is detected, its path is passed to tehmeaution() function, which sends the file in a binary representatiotno a local server for processing and returns an XML response. The XML response is then sent to another server usinsgetnhdeToAbel() andsendToAvatar() function. This process continues until the user interrupts the program or an error occurs. Two Flask web applications [ 15 ], i.e., ReceiverAvatar (RAv, Algorithm3) andReceiverAbel (RAb, Algorithm4), are structured as web services that receive XML data, extract AUs values, and send them in the appropriate format to the avatar and the robot usingsetnhdeAUsAbel() andsendAUsAvatar() functions. Using Flask applications enable the system with a high versatility and scalability, because, when they receive a post request to the relative write endpoint, they call the write function, which extracts the data fro the request, performs some formatting steps, sends the data to the avatar or to Abel using the relative sendAUs() function, and returns a response to the original request. Another Flask application using theEmotiva API is responsible for predicting facial AUs and estimated emotions from a single image . Emotiva is a Facial Expression Recognition (FER) software able to analyze human attentive and afective states 1[ 4 ]. A post call is made each time the event handler detects the capture of a frame in the specified folder. The virtual avatar used in this framework is based on the project, an open-source -based 3D face animation system11[ , 12 ]. It is a software that enables the simulation of realistic facial expressions by manipulating specific AUs as defined in the FA CS. includes an API suitable for generating real-time dynamic facial expressions for a three-dimensional character. It can be easily integrated into existing systems without requiring prior experience in computer graphics. Algorithm 1 Frame Grabber 1: function setupFolder( _ ) 2: if _ is not specified then 3: _ ← ‘/correct/path/to/frame_folder’ 4: 5: Algorithm 2 Image sender 1: function sendAUs(s) 2: ← content of the post request to ‘/AUs_write_port’ 3: return response 4: function detectNewImage(event) 5: if new image in the foldetrhen 6: send to Flask receiver server Algorithm 3 Avatar’s receiver 1: initialize Flask instance 2: function write(‘/write_port’, method=‘POST’) 3: ← content of the post request to ‘/AUs_extraction_port’ 4: ← extract list from 5: send and movement speed to avatar 6: return ‘Data received’ Algorithm 4 Abel’s receiver 1: initialize Flask instance 2: function write(‘/write_port’, method=‘POST’) 3: ← content of the post request to ‘/AUs_extraction_port’ 4: ← extract list from 5: send and movement speed to Abel 6: return ‘Data received’

Frame Grabber

Frame Event Handler (Image Sender)

Watchdog frame_folder

Avatar Receiver Abel Receiver

Avatar Abel

3. Experimental results

The proposed framework was developed with Python on PC platform. We run the application with a frame rate= 5 fps using the integrated webcam, taking images of6s4i0zex 480 pixel. In Table1, AUs used for the experiment are listed. In our tests, the facial expressions assumed by the digital avatar as well as by the robot successfully followed the AUs extracted by the Emotiva API and, although they were noisy data acquired with non-specific equipment, it was possible to efectively control the movement of both the virtual and physical agents simply changing the user facial expression. This is shown in Fig.2c and2d in the case of the avatar, and in F2iega.nd2f in case of the robot. The landmarks are represented as yellow dots superimposed on the two images of the subje2catasn(Fdig. 2b). Additionally, a rectangle is used to identify the faces present in the field of view. The values of the AUs in both cases are also displayed in Figu2rgeasnd2h. To improve the quality of control, we plan, instead of using the PC camera, to use a Kinect camera directly connected to Abel for image acquisition, which has a higher resolution, and to increase the number of AUs involved in the agent control. To set up the experiment accurately it is necessary to be in a very bright environment, preferably under direct light to increase the contrast of the acquired image. It is also advisable to choose a suficiently high frame rate to achieve real-time control of the robot and the avatar. Selecting values that are to low can lead to latency issues in the avatar system. Additionally, during the experiment, the subject’s face should not exit the camera’s field of view or rotate more than an angle of about 30° from the central position. Processing a partial face is not supported. If this occurrence happens, the results are deemed unreliable, and the user receives an alert message. In case of detection of multiple subjects in the field of view, a processing is performed for each visible face. In the context of the presented framework, this could generate conflicts in the control of the avatar and the robot, since there is no decision-making algorithm included in this framework. This problem is solved by the integration of the proposed framework with the high-level cognitive processing (i.e., the Plan block of Abel’s control architecture13[]) that makes the artificial agents able to focus their attention on a specific subject according to specific attention rul1e6s].[ (a) Happy expression (b) Sad expression (c) Happy expression, avatar face (d) Sad expression, avatar face (e) Happy expression, Abel face (f) Sad expression, Abel face (g) Happy expression AUs and emotions values

(h) Sad expression AUs and emotions values

4. Conclusions and future developments

The use of a digital avatar introduces several advantages, such as simplifying certain phases of development and testing which would normally involve the correspondent physical robot, helping to prevent the inevitable wear and tear of the robot’s electronic and mechanical components, and the high scalability and afordability of this technology. On the other hand, we are aware of the importance and the influence of a social robot corporeality in HRI, especially in several clinical applications (e.g., [ 17, 18, 19, 20 ]. The presented architecture allows exploratory interaction studies where it will be possible to compare two systems in which the perception and information processing remain identical, with the only difering variable being the representation and embodiment of the artificial agent. These studies will lead to a methodological evaluation using standard scales (e.g., the Godspee2d1]m) ethods [ comparing the cases of Abel and the digital avatar. Moreover, the degrees of freedom of the avatar are limited e.g., it cannot make asymmetric expressions, and it cannot move any part of the body but the face. To improve this aspect, the next steps of the project will be also to modify the code of the avatar at a lower level, separating the right and left part of its face, and to build the graphical component and the control of other expressive body parts, such as neck, arms and hands.

Acknowledgments

Thanks to the developers of Emotihvtatps://emotiva.ita/nd openFACShttps://github.com/phuselab/ openFACS. Research partly funded by PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 - ”FAIR - Future Artificial Intelligence Research” - Spoke 1 ”Human-centered AI”, funded by the European Commission under the NextGeneration EU programme. [21] C. Bartneck, E. Croft, D. Kulic, Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots, International Journal of Social Robotics 1 (2009) 71–81. do1i:0.1007/s12369-008-0001-3.

References

[1] S. Li , W. Deng , Deep facial expression recognition: A survey , IEEE transactions on afective computing 13 ( 2020 ) 1195 - 1215 .

[2] D. Canedo , A. J. Neves , Facial expression recognition using computer vision: A systematic review , Applied Sciences 9 ( 2019 ) 4678 .

[3] P. Ekman , W. V. Friesen , Facial action coding system , Environmental Psychology & Nonverbal Behavior ( 1978 ).

[4] C. Breazeal , D. Buchsbaum , J. Gray , D. Gatenby , B. Blumberg , Learning from and about others: Towards using imitation to bootstrap the social understanding of others by robots , Artificial life 11 ( 2005 ) 31 - 62 .

[5] T. Wu , N. J. Butko , P. Ruvulo , M. S. Bartlett , J. R. Movellan , Learning to make facial expressions , in: 2009 IEEE 8th International Conference on Development and Learning , 2009 , pp. 11 - 06 .. doi: 1109/DEVLRN. 2009 . 5175536 .

[6] S. Boucenna , P. Gaussier , P. Andry , L. Hafemeister , A robot learns the facial expressions recognition and face/non-face discrimination through an imitation game , International Journal of Social Robotics 6 ( 2014 ) 633 - 652 .

[7] A. Meghdari , S. B. Shouraki , A. Siamy , A. Shariati, The real-time facial imitation by a social humanoid robot , in: 2016 4th International Conference on Robotics and Mechatronics (ICROM) , IEEE, 2016 , pp. 524 - 529 .

[8] H. Kobayashi , F. Hara , Facial interaction between animated 3d face robot and human beings , in: 1997 IEEE International Conference on Systems, Man, and Cybernetics . Computational Cybernetics and Simulation , volume 4 , 1997 , pp. 3732 - 3737 vol. 4 . d1o0i.:1109/ICSMC. 1997 . 633250 .

[9] D. Li , C. Sun , F. Hu , D. Zang , L. Wang , M. Zhang , Real-time performance-driven facial animation with 3ds max and kinect , in: 2013 3rd International Conference on Consumer Electronics, Communications and Networks , 2013 , pp. 473 - 476 . d1o0i .:1109/CECNet. 2013 . 6703372 .

[10] N. Rawal , D. Koert , C. Turan , K. Kersting , J. Peters , R. Stock-Homburg , Exgennet: Learning to generate robotic facial expression using facial expression recognition , Frontiers in Robotics and AI 8 ( 2022 ) 730317 .

[11] V. Cuculo , A. D'Amelio , Openfacs: An open source facs-based 3d face animation system , in: Y. Zhao , N. Barnes , B. Chen , R. Westermann , X. Kong , C. Lin (Eds.), Image and Graphics , Springer International Publishing, Cham, 2019 , pp. 232 - 242 .

[12] openFACS, 2023 . URL: https://github.com/phuselab/openFAC.S

[13] L. Cominelli , G. Hoegen , D. De Rossi, Abel: integrating humanoid body, emotions, and time perception to investigate social interaction and human cognition , Applied Sciences 11 ( 2021 ) 1070 .

[14] Emotiva , 2023 . URL:https://emotiva.i.t/

[15] Flask , 2023 . URL:https://flask.palletsprojects.com/en/2..3.x/

[16] L. Cominelli , D. Mazzei , D. E. De Rossi, Seai: Social emotional artificial intelligence based on damasio's theory of mind , Frontiers in Robotics and AI 5 ( 2018 ) 6 .

[17] S. Shamsuddin , H. Yussof , L. Ismail , F. A. Hanapiah , S. Mohamed , H. A. Piah , N. I. Zahari , Initial response of autistic children in human-robot interaction therapy with humanoid robot nao , in: 2012 IEEE 8th International Colloquium on Signal Processing and its Applications , 2012 , pp. 188 - 193 . doi: 10 .1109/CSPA. 2012 . 6194716 .

[18] S. Shamsuddin , H. Yussof , L. I. Ismail , S. Mohamed , F. A. Hanapiah , N. I. Zahari , Initial response in hri-a case study on evaluation of child with autism spectrum disorders interacting with a humanoid robot nao , Procedia Engineering 41 ( 2012 ) 1448 - 1455 .

[19] A. Tapus , A. Peca , A. Aly , C. Pop , L. Jisa , S. Pintea , A. S. Rusu , D. O. David , Children with autism social engagement in interaction with nao, an imitative robot: A series of single case experiments , Interaction studies 13 ( 2012 ) 315 - 347 .

[20] L. J. Wood , A. Zaraki , B. Robins , K. Dautenhahn , Developing kaspar: a humanoid robot for children with autism , International Journal of Social Robotics 13 ( 2021 ) 491 - 508 .