Towards Affective Lifelogging with Information Fusion Jason J. Jung Department of Computer Engineering Chung-Ang University, Seoul, Korea j2jung@gmail.com Abstract. Recently, many of context-aware services are trying to exploit the emotional contexts of the target users. The aim of this conceptual paper is to discuss affective lifelogging framework which can recognize the emotions by in- tegrating multimodal information from multiple sources. Moreover, we will men- tion the open problems on affective lifelogging. Keywords: Lifelogging · Affective computing · Information fusion · Stream syn- chronization. 1 Introduction Lifelogging (also known as Quantified self [3]), which is a historical dataset of user activities (and behaviors), has been regarded as an important information for under- standing their personal contexts (e.g., interests and patterns). Especially, with various smart devices and wearable devices, it has been much easier for users to record their lifeloggings. Given a particular domain, lifelogging has been studied for various appli- cations, e.g., storification [2] and MyMovieHistory [1]. It is important for context-aware services to recognize emotional states of the target users (whether they are happy or sad). In this study, we are interested in affective lifel- ogging of users. By analogy, it means that the history of user’s emotional states can be recorded. However, it is impractical to ask users to answer what their current emotional state is. It means that affective lifelogging has to be in a non-intrusive manner. Thereby, this work is focusing on collecting all possible lifelogging datasets, and discovering meaningful patterns for affective lifelogging, as shown in Fig. 1. The outline of this paper is as follows. Sect. 2 shows the main idea on affective lifelogging frameworks. In Sect. 3 and Sect. 4, we will address the open problems on capturing the emotional state of users, and draw our conclusion of this on-going work, respectively. 2 Affective lifelogging We design the affective lifelogging framework with multimodel data streams. Definition 1 (Multimodel streams). Given a set of data streams S = {si |i ∈ [1, N]}, multimodel streams can be represented as S = hsi , τ, δ i|si ∈ S, (1) 2. 연구개발의 계획의 우수성 2.2 핵심 기술 • 시스템 시나리오 2 Jason J. Jung EEG Blood Pressure BPM Temperature Photo stream Location Tweets Busy Time Busy Fig. 1. Multimodal streams from multiple data sources Semantic Life Vault 딥러닝 기반 어펙티브 라이프로깅 프레임워크 - 13 -- 18 - - 융복합 콘텐츠 Social 감성인지와 Social Intelligence 모델 활용 Life Logging 기반 기술 개발 where τ and δ indicate the timestamps and data mode, respectively. This work is focus- ing on collecting all possible lifelogging datasets, and discovering meaningful patterns for affective lifelogging, as shown in Fig. 1. For example, the streams are including – various bio-signals (e.g., EEG, blood pressure, and BPM) by wearable devices, 2. 연구개발의 계획의 – location by 우수성 GPS-enabled smart devices, – photos by camera, and 2.2– 핵심 texts기술 by social media. • 감성 이벤트 패턴 마이닝 Event Pattern EEG Library 혈압 심박수 Affective Event Pattern Mining "# "$ "% 체온 사진 ! – size window 위치 텍스트 Happy Busy Sleepy Fig. 2. Event-driven learning for affective lifelogging Semantic Life Vault 딥러닝 - 10 the Thereby, this framework can capture 기반 어펙티브 라이프로깅 프레임워크 -- 15 - timestamps when the emotion is recog- - 융복합 콘텐츠 Social 감성인지와 Social Intelligence 모델 활용 Life Logging 기반 기술 개발 nized with high confidence, and regard them as the events (for segmenting the multiple data streams), as shown in Fig. 2. Towards Affective Lifelogging with Information Fusion 3 Definition 2 (Event). A set of events E is represented as E = {ei |τ@  ei  τA } (2) where τ@ and τA are the beginning and ending of an event, respectively. 2.1 Learning by discovering correlation Once the events are detected, the multiple multimodal streams S can be segmented. By measuring the correlation among the multiple multimodal streams in the same segment, we can find the event pattern library (shown in Fig. 2). Thus, this event-driven approach is similar to the labeling process for training. As another important issue, we are focusing on relative scaling factor (RSF). When the correlation is computed, we want to consider the unique characteristics of the data stream. 2.2 Applications and services With the proposed affective lifelogging framework, several applications and services will be developed with respect to the target user. 2. 연구개발의 계획의 우수성 – single individuals (shown in Fig. 3) 2.3 –예상 a group 결과물of users in a certain location and time (shown in Fig. 4) • 감성 히스토리 기록 04:15 13:14 01:10 03:11 05:33 hours hours hours hours hours Semantic Life Vault 딥러닝 기반 어펙티브 라이프로깅 프레임워크 - 16 Fig. 3. Personal service - 융복합 콘텐츠 Social 감성인지와 Social Intelligence 모델 활용 Life Logging 기반 기술 개발for-- affective 21 - lifelogging 2. 연구개발의 계획의 우수성 2.3 예상 결과물 4 Jason J. Jung • 사회 감성 지도 Emotion1 Emotion2 Emotion3 Emotion4 Semantic Life Vault 딥러닝 기반 어펙티브 라이프로깅 프레임워크 - 융복합 콘텐츠 Social 감성인지와 Social Intelligence 모델 활용 Life Logging 기반 기술 개발- 17 -- 22 - Fig. 4. Spatiotemporal visualization 3 Open problems Since it is the on-going work, we want to mention the following open problems. 1. Evaluation issue: In order to evaluate the proposed framework, we need to collect real world data from users. 2. privacy: Most seriously, it is almost impractical to ask users to recall their own emotion states in the past. 4 Conclusion In this paper, we have introduced a conceptual design on affective lifelogging frame- work. The proposed framework is based on event-based segmentation for the multiple multimodal stream. Acknowledgment This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (NRF-2017R1A2B4010774). References 1. Hong, M., Jung, J.J.: Mymoviehistory: Social recommender system by discovering social affinities among users. Cybernetics and Systems 47(1-2), 88–110 (2016) Towards Affective Lifelogging with Information Fusion 5 2. Jung, J.E., Hong, M., Hoang, L.N.: Serendipity-based storification: from lifelogging to story- telling. Multimedia Tools Appl. 76(8), 10345–10356 (2017). https://doi.org/10.1007/s11042- 016-3682-x, https://doi.org/10.1007/s11042-016-3682-x 3. Swan, M.: The quantified self: Fundamental disruption in big data science and biological discovery. Big Data 1(2), 85–99 (2013)