The following project concerns the proposal of a positive mood-supporting application, based on multi-sensorial stimuli. The first goal of this project is to study the diferences, relations, and interactions between emotion and mood, trying to understand how emotion recognition methods and eliciting stimuli could be adjusted in the mood domain. This is a fundamental step in building a system meant to detect the user's mood and monitor it over time while trying to support a positive mood by presenting proper multisensorial stimuli. Therefore, two other intermediate steps are essential: i) building a multimodal mood detection framework, using wearable devices combined with short questionnaires; and ii) defining which audiovisual characteristics a multi-sensorial stimulus should exhibit to empower its positive mood-support efect, exploiting diferent learning models. A profiling questionnaire will be submitted the first time a user accesses the system. Then a multimodal mood detection takes place with an Ecological Momentary Assessment (EMA) of the current mood and data derived from smartphones and wearable devices such as environmental, behavioural, and physiological data. Since the user's current mood is defined, the system automatically selects the proper audiovisual stimuli to suggest the improvement of the mood, if needed. After a mood support session, a user's feedback will be required to improve the efectiveness of the system and ensure a better set of stimuli for the specific user. A case study will be considered to understand the eficacy of this system in real-world applications. In particular, an automotive scenario will be developed, exploiting virtual reality stimuli during a driving simulation.
Emotion recognition is a widely investigated field [
In the psychiatric disorders field [
Smartphones can meet the momentary necessity of this type of assessment, due to their handy nature
[
The available mood self-reports are mainly based on or taken directly from the questionnaires designed
for related fields such as emotion, afect, and personality [
Audiovisual stimuli can help to positively support mood, playing a part in determining a quieter state of a subject. The available datasets are intended to elicit emotions, or afect a user state, and are not specifically designed to influence moods [ 18]. They include unimodal stimuli, for example, International Afective Picture System (IAPS) [ 19], Geneva Afective PicturE Database (GAPED) [ 20], Nencki Afective Picture System (NAPS) [21], EmoPics [22], International Afective Digitized Sounds (IADS) [ 23], except for Experimental MOVies for Induction of Emotions (E-MOVIE) [24], which employs audio and visual stimuli.
My project proposal regards a system intended to detect a user’s current mood and automatically ofer multi-sensorial stimuli to support a positive influence on the user’s state. To detect the mood, considering its definition, daily, long-term, and non-invasive monitoring of the user’s state seems essential. An optimal solution would be a smartphone application, that also employs wearable devices, in order to collect information about the user comprising self-assessment reports and the analysis of recorded multimodal data.
A machine learning model feed with hand-crafted features will be developed to study which kind of multi-sensorial stimuli ofer the best positive support and which stimuli features should be included in audiovisual production best practices. This system will leverage a personalised approach by implementing user-adaptive models for mood detection and stimuli suggestion.
The proposed system can be employed in diferent fields with the aim of improving user’s well-being. For example, a more favourable ambiance built with positive mood support multi-sensorial stimuli can support the psychological well-being of patients during rehabilitation or senior citizens. Also, the automotive field can benefit from the proposed system, where driving simulations can measure the support efect in stressful situations.
Interpreting the proposed system as a potential medical assistance item, the European AI Act regulations will be followed thoroughly, especially considering the system potentiality of reducing stress in working environments. This application would also be in line with the 2030 Agenda for Sustainable Development of the United Nations and in particular with the goal of improving healt and well-being in work environments. Ethical considerations are clearly paramount. For example, it will be essential to ensure user anonymity during data processing, with pre-processed data stored on a cloud server. A rigorous validation process will be implemented to ensure a positive impact on users’ moods. Additionally, user volunteers will be observed, allowing them to determine their benefits from the support system. Legal experts will be consulted to ensure compliance with the EU Artificial Intelligence Act and the General Data Protection Regulation (GDPR).
To realize the final system design, several steps must be addressed: each of them is categorized into distinct topic blocks, as shown in Figure 1. Each block represents an innovative approach and will be accompanied by specific milestones to track progress.
Firstly, a deep study of the state of the art will be carried out to better understand mood and emotion. Their influence and dependence on each other are open questions and need to be further investigated. The main aim is to understand if (and how) we can study mood leveraging on emotional response to stimuli and if it is possible to employ emotion recognition models during long periods of user observation. Moreover, it is important to understand if the emotionally eliciting stimuli can be used to support a positive mood, as long as it is repeated over time. This will require the study of diferent emotion stimuli datasets and the selection of suitable starting stimuli. After this initial step, a protocol to measure the mood will be defined and the preparation of a state-of-the-art paper is intended to conclude this first phase.
A smartphone is hypothesized to be a suitable tool to support the definition of a framework to detect a user’s current mood, allowing the development of an ecological system for passive monitoring. This handy and real-life approach would be a new way to detect mood for common users, considering that previous studies were mainly conducted on clinical cases, such as patients with mood disorders, in controlled contexts.
As previously introduced, the system includes multimodal mood profiling, thus the application will require the following data at registration: demographics, such as age, gender, education level, and the predominant personality trait, by answering the 10-item Big Five Inventory Personality Test [25]. Subsequently, the current mood will be accessed through an EMA questionnaire, exploiting the Pick-A-Mood validated questionnaire.
Moreover, environmental and contextual (e.g., weather and location), behavioural (e.g., facial expression and speech analysis [26]), and physiological data (e.g., photoplethysmographic signals and physical activity [27]) will be collected through wearable and portable devices such as smartphones and smartwatches. The type of collected data strongly depends on the type of available wearable device, thus a comparative analysis between diferent devices will be performed.
The literature will be exploited to find suitable handcrafted features and learning models to use on the previously identified literature datasets and to determine which audiovisual characteristics are mostly responsible for the positive efect of the multi-sensorial stimuli. With positive and negative efect labelling, a supervised learning model will be applied to predict the classes eficiently and determine the best-ranked features during the classification.
The model will use carefully selected handcrafted features based on humanistic approaches instead of relying on less interpretable deep features. In cinematography, best practices help create engaging content, which can be translated into interpretable handcrafted features. This study builds on my previous thesis, where I explored integrating humanistic and psychological perspectives into computer vision algorithms. By analyzing these features, we can create production guidelines for developing updated datasets across various fields, focusing on areas like communication efectiveness and interestingness, as well as supporting positive mood, which is the aim of this project.
For this project, specific stimuli will be chosen to define a general dataset for a population, to improve the efectiveness of the positive mood support. This study in the future will allow the creation of new datasets, with updated styles or specially made for specific types of populations. The acquired cues will aid in creating datasets with new types of stimuli, such as tridimensional environments for virtual reality, necessitating user participation for data validation.
In afective computing literature, multimodal models are widely used to recognize emotions [
The feature set and learning model will be carefully selected in previous steps to ensure that the application can provide real-time responses. Common problems from systems that integrate heterogeneous data from diferent sources will be considered. This structure will also adapt to individual users to support a user-centric approach, not only in detecting current moods but also in suggesting appropriate stimuli. The model will ofer positive support stimuli to enhance emotional well-being. The efectiveness of these stimuli will be measured by monitoring mood changes over time and gathering feedback from users.
I am currently collaborating with the Haptics and Virtual Prototyping Lab in the Department of Mechanical Engineering at Politecnico di Milano, under the direction of Professor Francesco Ferrise. This research group primarily focuses on virtual and augmented reality, with active involvement in the automotive field. We will test the mood support system by creating virtual reality scenarios that simulate stressful driving situations. In these scenarios, specific audiovisual stimuli could serve as efective tools to reduce driver tension and promote safer driving conditions. Research has shown that daily stress is associated with negative mood changes on the same day [29, 30].
Through this application scenario, we will evaluate the eficacy of the system, considering various metrics to identify the most appropriate ones for the specific final application.
Computational models play an important role in this project: an innovative mood profiling and detection system would integrate multimodal data, such as personality portray, recently studied EMA questionnaires, and environmental, contextual, behavioural, and physiological data thanks to portable and wearable devices.
Using consumer wearable devices has the advantage of being economically available to extended groups of people, but also the weakness of less precise data [31]; complementing those data with EMA and diferent kinds of data, not only the psychological ones, will help with the accuracy of mood detection. Multimodal approaches are adopted in the emotion recognition field; this study would also allow us to understand the correlations between mood and emotions over time. The handy approach permits the investigation of subjects with no particular pathologies while living in real-world situations and not in controlled environments.
Machine learning models will also help to characterize the most engaging stimuli, thanks to hand-crafted features designed from the video-making best practices. The obtained creative cues will also help to create new datasets in the future and improve virtual scenarios such as in driving simulation. Computational models will also ofer challenges to overcome, such as a user-centric approach for adaptive mood detection, suitable eficiency for a smartphone application, and collecting data from diferent devices. current methods and design of a pictorial self-report scale, Journal of Design Research 14 (2016) 241–279. doi:10.1504/JDR.2016.079751. [15] C. E. Izard, P. B. Read, Measuring emotions in infants and children: based on seminars sponsored by the Committee on Social and Afective Development During Childhood of the Social Science Research Council, volume 1, Cambridge University Press, 1982. [16] D. Watson, L. A. Clark, A. Tellegen, Development and validation of brief measures of positive and negative afect: the panas scales., Journal of personality and social psychology 54 (1988) 1063. [17] G. Puertas, V. Patel, T. Marshall, Are visual measures of mood superior to questionnaire measures in non–western settings?, Social psychiatry and psychiatric epidemiology 39 (2004) 662–666. [18] A. Alnafjan, M. Hosny, Y. Al-Ohali, A. Al-Wabil, Review and classification of emotion recognition based on eeg brain-computer interface system research: A systematic review, Applied Sciences 7 (2017) 1239. doi:10.3390/app7121239. [19] P. J. Lang, M. M. Bradley, B. N. Cuthbert, et al., International afective picture system (iaps): Technical manual and afective ratings, NIMH Center for the Study of Emotion and Attention 1 (1997) 3. [20] E. S. Dan-Glauser, K. R. Scherer, The geneva afective picture database (gaped): a new 730-picture database focusing on valence and normative significance, Behavior research methods 43 (2011) 468–477. [21] A. Marchewka, Ł. Żurawski, K. Jednoróg, A. Grabowska, The nencki afective picture system (naps): Introduction to a novel, standardized, wide-range, high-quality, realistic picture database, Behavior research methods 46 (2014) 596–610. [22] M. Wessa, P. Kanske, P. Neumeister, K. Bode, J. Heissler, S. Schönfelder, et al., Emopics: Subjektive und psychophysiologische evaluation neuen bildmaterials für die klinisch-bio-psychologische forschung, Zeitschrift für Klinische Psychologie und Psychotherapie 39 (2010) 77. [23] M. M. Bradley, P. J. Lang, International afective digitized sounds (iads): Stimuli, instruction manual and afective ratings (tech. rep. no. b-2), Gainesville, FL: The Center for Research in Psychophysiology, University of Florida (1999). [24] A. Mafei, A. Angrilli, E-movie-experimental movies for induction of emotions in neuroscience:
An innovative film database with normative data and sex diferences, Plos one 14 (2019) e0223124. [25] R. R. McCrae, O. P. John, An introduction to the five-factor model and its applications, Journal of personality 60 (1992) 175–215. [26] R. Jahangir, Y. W. Teh, F. Hanif, G. Mujtaba, Deep learning approaches for speech emotion recognition: state of the art and research challenges, Multimedia Tools and Applications 80 (2021) 23745–23812. [27] S. Wan, L. Qi, X. Xu, C. Tong, Z. Gu, Deep learning models for real-time human activity recognition with smartphones, Mobile Networks and Applications 25 (2020) 743–755. [28] K. Ezzameli, H. Mahersia, Emotion recognition from unimodal to multimodal analysis: A review,
Information Fusion 9 (2023). doi:10.1016/j.inffus.2023.101847. [29] A. DeLongis, S. Folkman, R. S. Lazarus, The impact of daily stress on health and mood: psychological and social resources as mediators., Journal of personality and social psychology 54 3 (1988) 486–95. [30] A. A. Stone, J. M. Neale, S. Shiftman, Daily assessments of stress and coping and their association with mood, Annals of Behavioral Medicine 15 (1993) 8–16. URL: https://doi.org/10.1093/abm/15.1.8. doi:10.1093/abm/15.1.8. [31] S. Gashi, P. Oldrati, M. Moebus, M. Hilty, L. Barrios, F. Ozdemir, P. Consortium, V. Kana, A. Lutterotti, G. Rätsch, et al., Modeling multiple sclerosis using mobile and wearable sensor data, npj Digital Medicine 7 (2024) 64.