This paper introduces a multimodal framework designed to enhance empathetic communication in human-robot interactions. Our approach integrates psychological profiling-leveraging publicly available social network data and the DISC personality framework-with real-time emotion recognition across facial, audio, and textual modalities. Facial expressions are analyzed using convolutional neural networks and MTCNN-based face detection, while audio signals are processed through Mel-Frequency Cepstral Coefficients and support vector machines. Textual inputs are evaluated via sentiment analysis using advanced language models. These individual emotional assessments are fused through a fuzzy aggregation method, emphasizing non-verbal cues to derive a comprehensive and adaptive emotional profile. The resulting digital agent tailors its verbal responses and interaction style to align with the user's personality traits and current emotional state, offering a more natural and supportive communication experience.
Empathetic communication is a fundamental aspect of human interaction, fostering social
bonding, trust, and cooperation. In recent years, advancements in artificial intelligence and robotics
have enabled the development of socially interactive robots that can engage in empathetic
communication with humans. Such robots have the potential to revolutionize various domains—
including healthcare, education, and social companionship—by providing emotional support and
enhancing human well-being [
In human–robot interaction (HRI), empathy is typically defined as the robot’s ability to
recognize, interpret, and appropriately respond to human emotions through both verbal and
nonverbal cues [
Despite these promising outcomes, significant challenges remain in designing robots that
can truly engage in empathetic communication. One key issue is determining the appropriate
degree of anthropomorphism required for effective empathy expression. While some research
suggests that human-like facial expressions are essential for conveying empathy [
This study proposes a method for identifying the human agent interacting with the digital agent using visual data and verbal inquiries. Thereafter, by examining publicly available information online, the digital agent constructs a psychological profile of the human agent. The psychological profile is essential for a more advanced contextualization of communication. The digital agent evaluates visual, auditory, and textual data to comprehensively ascertain the emotional condition of the human agent. The emotional state facilitates the refinement of communication context to enhance empathetic alignment with the human agent. The ultimate stage of contextualization entails employing psychological methods to actualize this empathy in the human agent, thereby reinforcing the empathetic connection. The accurate definition of the context enables the digital agent to completely adjust to the human agent by selecting an appropriate communicative register and enhancing vocal expression to convey the correct emotions, preserving every nuance of meaning and formality within the empathetic communication established by the human agent. The following subsections address the tracking of the time-invariant psychological profile and its applications, the analysis of emotions through facial, vocal, and textual expressions, and the identification of the instantaneous time-variant emotional profile.
Traditional methods of personality assessment often require self-reported questionnaires, which can be time-consuming and subject to biases. Advances in natural language processing (NLP) and personality analytics have led to automated profiling systems that infer personality traits from online data. One such tool is Crystal Knows@, a personality prediction engine that utilizes the DISC framework to categorize individuals based on their publicly available professional information.
This study presents a Python-based approach to interfacing with Crystal’s API for automated psychological profiling. The script operates by collecting a LinkedIn profile URL extracting the human agent and using it to query the Crystal API, which then returns a structured personality profile based on the DISC classification. To achieve this, the methodology follows a series of steps. First, the human agent provides information to obtain a LinkedIn profile URL along with a valid Crystal API key for authentication. The script then processes an HTTP POST request containing the URL and sends it to the API using the `requests` library. Upon successful retrieval, the API returns a JSON response containing the subject’s personality classification, which is then parsed and displayed in a structured format for easy interpretation. To ensure robustness, the script includes error handling mechanisms that display appropriate messages in case of failures, such as invalid API credentials, insufficient data, or server errors.
Crystal's DISC framework categorizes personality into 16 distinct types, each defined by varying degrees of Dominance (D), Influence (I), Steadiness (S), and Conscientiousness (C). These types are represented by specific archetypes that offer insights into individual behaviors and communication styles. The Dominance (D) category includes the Captain (assertive and ambitious), Driver (decisive and persuasive), Initiator (charismatic and resourceful), and Influencer (energetic and adventurous). The Influence (I) category features the Motivator (enthusiastic and outgoing), Encourager (warm and light-hearted), Harmonizer (patient and accommodating), and Counselor (empathetic and supportive). The Steadiness (S) category comprises the Supporter (calm and respectful), Planner (predictable and detail-oriented), Stabilizer (reserved and cautious), and Editor (meticulous and independent). Lastly, the Conscientiousness (C) category includes the Analyst (methodical and private), Skeptic (logical and efficient), Questioner (competitive and strategic), and Architect (strong-willed and purposeful). By understanding these personality types, the digital agent can enhance self-awareness, improve team dynamics, and refine interpersonal relationships.
To effectively adapt to DISC profiles, communication should be tailored to each personality type. When interacting with individuals high in Dominance (D), it is best to be direct, concise, and focused on results, avoiding unnecessary details that may be seen as time-wasting. For those with Influence (I) traits, engaging with enthusiasm, recognizing their contributions, and allowing space for social interaction fosters better communication. Individuals with Steadiness (S) prefer a patient approach, reassurance, and stability, so sudden changes should be minimized to avoid discomfort. Finally, those high in Conscientiousness (C) value detailed and accurate information, requiring time to process and respond thoughtfully.
The psychological profile of the human agent is associated to a reference composed by name, surname and face to store it for future use.
The proposed face identification system utilizes the `face_recognition` library to detect and recognize human faces in real-time video streams. Initially, a dataset of known faces is loaded from a predefined directory, where each image is processed to extract facial encodings using deep learning-based feature extraction. During execution, frames from a video source are captured and preprocessed by resizing and converting them to RGB format. The system detects facial landmarks, extracts encodings, and compares them against the stored database using a distance-based similarity metric. If a match is found, the system labels the detected face accordingly; otherwise, it assigns an "Unknown" label and prompts the human agent to provide their name and surname. This information, along with the extracted facial encoding, is then stored in the database for future recognition. The results, including bounding boxes and names, are overlaid onto the video stream and displayed in real time. The system operates continuously, allowing for dynamic face recognition, and can be terminated by the user via a key press.
Leveraging emotional insights in conjunction with a subject's psychological profile, such as one derived from the DISC model, can significantly enhance the effectiveness of your interactions.
For more detailed instructions, the proposed system is built upon the Facial Expression Recognition (FER) framework, leveraging convolutional neural networks (CNNs) for the classification of emotions from facial images. The implementation follows a structured pipeline comprising image acquisition, face detection, emotion classification, and real-time visualization. A modular architecture is adopted, wherein face detection is performed using the Multi-Task Cascaded Convolutional Networks (MTCNN) detector to ensure robust localization, while the FER model classifies emotions into seven predefined categories: Angry, Disgust, Fear, Happy, Sad, Surprise, and Neutral. Real-time processing is facilitated through the OpenCV library, enabling frame capture via a webcam and subsequent display of annotated results. The system is implemented in Python, employing OpenCV for video acquisition and the FER library for emotion classification. The workflow consists of initializing the FER detector with MTCNN, capturing frames, detecting faces and extracting bounding boxes, applying the FER classifier for emotion prediction, overlaying the detected emotion onto the video frame, and displaying the annotated frame in real time, with termination triggered by user input.
The proposed approach consists of three main steps: feature extraction, classification, and prediction, following a supervised learning paradigm in which a machine learning model is trained to recognize emotions based on extracted speech features. The initial phase involves feature extraction from the raw speech signal, utilizing Mel-Frequency Cepstral Coefficients (MFCCs), which are widely adopted in speech processing due to their effectiveness in capturing the perceptual characteristics of human hearing. The extraction process comprises audio preprocessing, where the input audio file is loaded using the Librosa library while maintaining its original sampling rate, followed by the computation of 40 MFCC coefficients from the signal, and statistical aggregation through the calculation of mean MFCC values across time frames to generate a fixed-length feature vector. The extracted features are then used to train a supervised classifier, specifically a Support Vector Machine (SVM) with a radial basis function (RBF) kernel, chosen for its robustness in speech emotion recognition tasks. The approach is evaluated using the Toronto Emotional Speech Set (TESS) dataset, which contains 2800 speech samples spoken by two actresses and classified using the same emotional scale applied in facial emotion analysis, ensuring consistency in multimodal emotion recognition.
In the proposed system, emotion analysis is conducted by leveraging OpenAI’s ChatGPT-4o API, which processes textual input to generate a statistical evaluation of emotional content. The methodology involves sending a structured request containing the target text to the API, which The vector of emotional states is composed of seven probability values ranging from 0 to 1 associated with the seven emotions: Fear, Angry, Disgust, Sad, Neutral, Happy, (positive) Surprise, as shown in (2), where the time dependence is omitted for the sake of simplicity. It would be possible to defuzzify the result by identifying a single most probable emotion, but it was preferred to maintain the emotion defined in a fuzzy manner. In general, the first four emotions are considered indicative of negative feedback, while the last three emotions are considered indicative of positive feedback. (1) (2) then analyzes linguistic features, semantic context, and sentiment polarity to classify the underlying emotions. The model provides a probabilistic distribution of detected emotions, allowing for a quantitative assessment rather than a binary classification.
The emotional state of the human agent is evaluated in a fuzzy manner as an overlap of the assessments obtained from facial, audio, and text emotional analyses with appropriate weights. To appropriately define these weights, it is assumed that the human agent may attempt to mask their emotional state; therefore, greater weight is given to the communication that is more difficult to mask, namely non-verbal communication, then to the tone of voice, and finally to the text chosen for communication, according to the formula indicated in (1), where e(t), ef(t), ea(t), and et(t) are, respectively, the vectors of the overall emotional state, the one obtained from facial analysis, the one obtained from audio analysis, and the one obtained from the neutral language analysis of the text at time t for the human agent.
The communicative form of the digital agent is guided by the psychological profile of the human agent, but some behavioral changes can be expected to vary with the emotional state of the human agent.
A first category of strategies consists of acknowledging emotions. In particular, two approaches are
used: verbal validation and reflective listening [
The flowchart in Figure 1 illustrates the proposed automated system for personalized human-robot interaction based on facial recognition, emotional analysis, and psychological profiling. The process begins with face detection and recognition; if the person is unidentified, their name and surname are collected, stored in a database, and their LinkedIn profile is retrieved. If a LinkedIn profile exists, the system uses the Crystal API to generate a psychological profile, which is also stored. Subsequently, face and audio inputs are analyzed for emotional content using multiple modalities: acoustic features, textual context, and facial expressions. A fuzzy aggregation method integrates these emotional cues with the psychological profile to determine the user's state. This state is then used to generate a context-aware response by querying ChatGPT, which is converted into a vocal message using a text-to-speech module and delivered to the user. The flowchart outlines a structured pipeline for adaptive and emotionally intelligent human-computer dialogue.
Figure 2 presents the pseudocode for face recognition. Figure 3 presents the pseudocode for psychological profile extraction. Figure 4 presents the pseudocode for face emotion recognition. Figure 5 presents the pseudocode for audio emotion recognition. Figure 6 presents the pseudocode for text emotion recognition.
Based on the proposed methodology, a digital agent capable of empathetic interaction in the
verbal domain with a human agent has been developed. To achieve this result, the human agent is
analyzed and decomposed into a time-invariant component during the single interaction, the
psychological profile, and a time-variant component during the single interaction, the emotional
state. There are works in the literature on profiling through social networks [
A second critical issue is related to the fact that individuals present an altered image of
themselves on social networks. It would be appropriate to implement a profile-building method
based solely on direct interaction; however, this approach might also prove ineffective if the
human subject exhibits different psychological aspects when interacting with a digital agent
compared to another human agent. On the other hand, there are various other psychological
profiling software through social networks, including, for example, IBM Watson Personality
Insights, Apply Magic Sauce API, Portrait, or Humantic AI and Kosinski et al [
There are several works in the literature on multimodal emotion analysis, both hierarchical [
As indicated in Figure 1, an empathetic Text-to-Speech (TTS) was used to communicate emotions through an appropriate tone and timbre of voice, speed, cadence, and rhythm. In particular, the Hume.AI service was used, as it is one of the most advanced empathetic TTS systems currently available. This technology has reached a high level of maturity.
This study has several limitations that should be acknowledged. First, the use of LinkedIn and the Crystal API for creating psychological profiles may introduce inaccuracies, as the information provided in these profiles can be selective or distorted. Users often curate their professional profiles to present a particular image, which may not fully reflect their actual personality traits or behavioral tendencies. Future work should consider integrating additional data sources or validation methods to improve the reliability of personality assessments.
Another limitation is the emphasis placed on non-verbal signals in emotional expression. While non-verbal cues play a significant role in human communication, their interpretation varies across cultures and individuals. This variability may lead to inconsistencies in how emotions are perceived and responded to by the system. Further studies should investigate the influence of cultural and personal differences on the effectiveness of non-verbal cues in human-robot interaction. Then the response is slow due to the hardware used and the type of cloud-based implementation: it might be interesting to develop an edge computing application to reduce latency.
Finally, although the system is designed to adapt the tone and pace of speech according to the user's emotional state, the impact of these modifications on interaction quality remains unclear. The study does not provide an evaluation of whether these adjustments enhance engagement or lead to unnatural behavior in the robot. Future research should assess user perceptions and the overall effectiveness of vocal adaptation in improving interaction dynamics.
The effectiveness of the approach in real cases was not explored in this preliminary work, which should be addressed in future developments.
It is also important to note that psychological profiling, although widely used and applicable in various professional and non-professional contexts, presents delicate aspects related to privacy and the decisions of both the subject interacting with the digital agent and those experienced by the subject themselves. To avoid improper behavior by the digital agent, it is possible to refer to various ethical guidelines or advanced regulations valid in certain geographical areas, such as the European General Data Protection Regulation (GDPR) generally for all subjects, but especially for cognitively weaker subjects.
In this work, we presented a novel multimodal framework for enhancing empathetic communication in human–robot interaction. By integrating psychological profiling derived from social network data with real-time emotion recognition from facial, audio, and textual inputs, our system offers a comprehensive understanding of a user's emotional state. This integration allows the digital agent to adapt its communication style dynamically, delivering responses that are not only context-aware but also emotionally resonant. Our architecture leverages advanced face recognition techniques, robust audio and text analysis, and a fuzzy aggregation method to synthesize these diverse modalities. The resulting framework has shown promising potential in improving interaction quality across various applications—from healthcare and education to social companionship. By prioritizing non-verbal cues, which are often more difficult to mask, the system effectively tailors its behavior to meet the emotional needs of users. Despite these advances, several challenges remain. Issues such as processing latency, the reliability of social network-based profiling, and the optimization of multimodal fusion require further exploration. Future work will focus on refining these components, incorporating more sophisticated machine learning techniques, and conducting comprehensive user studies to validate the system's effectiveness in real-world scenarios. In conclusion, our framework lays the groundwork for next-generation human–robot interactions that are not only functionally efficient but also emotionally intelligent, paving the way for more natural, supportive, and adaptive digital communication interfaces.
The author(s) have not employed any Generative AI tools.