Development of a Prototype AI System for Real-time Emotion Prediction and Mental State Adjustment

Abstract

Our emotional state influences our daily performance. Recent reports have highlighted a hormetic relationship between stress and cognitive performance [1, 2], suggesting that understanding the optimal balance of emotional states, rather than adopting a simple negativepositive emotion perspective, could potentially enhance the optimization of behavioral performance. Our ultimate goal is to construct a system that discerns individual emotional states from physiological information, and generates music, visuals, or conversations as means to facilitate the individual's transition toward their desired emotional state. To achieve this, it is essential to evaluate complex emotional states on different emotional axes and assess possibly continuously fluctuating emotional states in real-time as possible.

1. Introduction

3. Emotion Estimation System During Music Listening

Here, we present an emotion estimation system that provides predicted emotional values on six emotional axes, similar to Ishikawa et al. [ 6 ]. In our system, notably, we have incorporated the real-time prediction capability, allowing us to update the predicted values every 0.5 seconds. Model construction: We trained our model on 2,322 instances from 54 participants, each giving physiological (EEG and ECG) and emotional rating data. Participants listened to music for a minute while recording EEG and ECG, then rated their emotion on a 15-point scale for six emotional axes. Explanatory variables were taken from the 10 seconds of EEG and ECG data immediately preceding the emotion rating, and the emotion ratings served as the objective variables. The model was trained using XGboost due to its computational efficiency, smaller resource requirement, and faster training speed, enabling real-time estimation. Implementation of emotion estimation: Users wear EEG and ECG devices and transmit the measured data to a computer via a smartphone using Bluetooth. Once the computer accumulates 10 seconds of data, it begins estimating the emotional state. The emotional state estimate is then

2024 Copyright for this paper by its authors. CEUR

ceur-ws.org updated every 0.5 seconds based on the preceding 10 seconds of data. This system, as shown in Figure 1, enables the real-time estimation of emotional state transitions during music or visual content consumption.

4. Prospects and potential applications

In the future, we plan a comprehensive evaluation of our models' accuracy using metrics, such as MAPE and other relevant measures, aiming for less than 20% error in both training and realworld application. Future developments will merge AI to generate music or visuals, guiding users to their desired emotional states, potentially enhancing presentations, work efficiency, and mental health.

Acknowledgements References

This work was partially supported by Innovative Science and Technology Initiative for Security (JPJ004596), Acquisition, Technology, and Logistics Agency, Japan.

References

[1] A. Oshri , Z. Cui , M. M. Owens , C. A. Carvalho , and L. Sweet, “ Low-to-moderate level of perceived stress strengthens working memory: Testing the hormesis hypothesis through neural activation , ” Neuropsychologia , vol. 176 , p. 108354 , Nov . 2022 , doi: 10.1016/j.neuropsychologia. 2022 . 108354 .

[2] K. A. James , J. I. Stromin , N. Steenkamp , and M. I. Combrinck , “ Understanding the relationships between physiological and psychosocial stress, cortisol and cognition,” Front. Endocrinol., vol. 14 , 2023 , doi: 10.3389/fendo. 2023 . 1085950 .

[3] K. Fuseda , H. Watanabe , A. Matsumoto , J. Saito , Y. Naruse , and A. S. Ihara , “ Impact of depressed state on attention and language processing during news broadcasts: EEG analysis and machine learning approach , ” Sci. Rep ., vol. 12 , no. 1 , p. 20492 , Nov . 2022 , doi: 10.1038/s41598-022-24319-x.

[4] Y. Yokota , T. Soshi , and Y. Naruse , “ Error-related negativity predicts failure in competitive dual-player video games , ” PLOS ONE , vol. 14 , no. 2 , p. e0212483 , Feb . 2019 , doi: 10.1371/journal.pone. 0212483 .

[5] Y. Yokota and Y. Naruse , “ Temporal Fluctuation of Mood in Gaming Task Modulates Feedback Negativity: EEG Study With Virtual Reality,” Front. Hum. Neurosci. , vol. 15 , p. 536288 , Jun . 2021 , doi: 10.3389/fnhum. 2021 . 536288 .

[6] Y. Ishikawa et al., “Learning Cross-Modal Factors from Multimodal Physiological Signals for Emotion Recognition,” in PRICAI 2023: Trends in Artificial Intelligence , vol. 14325 , F. Liu , A. A. Sadanandan , D. N. Pham , P. Mursanto , and D. Lukose, Eds., in Lecture Notes in Computer Science , vol. 14325 , Singapore: Springer Nature Singapore, 2024 , pp. 438 - 450 . doi: 10 .1007/ 978 -981-99-7019-3_ 40 .

[7] T. Chen and C. Guestrin , “ XGBoost: A Scalable Tree Boosting System,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , San Francisco California USA: ACM, Aug. 2016 , pp. 785 - 794 . doi: 10 .1145/2939672.2939785.