Recently, many of context-aware services are trying to exploit the emotional contexts of the target users. The aim of this conceptual paper is to discuss affective lifelogging framework which can recognize the emotions by integrating multimodal information from multiple sources. Moreover, we will mention the open problems on affective lifelogging.
Lifelogging (also known as Quantified self [
It is important for context-aware services to recognize emotional states of the target users (whether they are happy or sad). In this study, we are interested in affective lifelogging of users. By analogy, it means that the history of user’s emotional states can be recorded.
However, it is impractical to ask users to answer what their current emotional state is. It means that affective lifelogging has to be in a non-intrusive manner.
Thereby, this work is focusing on collecting all possible lifelogging datasets, and discovering meaningful patterns for affective lifelogging, as shown in Fig. 1.
The outline of this paper is as follows. Sect. 2 shows the main idea on affective lifelogging frameworks. In Sect. 3 and Sect. 4, we will address the open problems on capturing the emotional state of users, and draw our conclusion of this on-going work, respectively.
We design the affective lifelogging framework with multimodel data streams. Definition 1 (Multimodel streams). Given a set of data streams S = fsiji 2 [1; N]g, multimodel streams can be represented as 2. 연구개발의 계획의 우수성 2.2
핵심 기술 2 •Jas시on스J.템Jun시g나리오 Temperature
EEG Blood Pressure
BPM Photo stream Location
Tweets S-e융m러a복n합tic 콘Lif텐e츠VauSlotciah티l감브e성r라인e지이b와프yS로o,cial Inte레liigs임enc워ef크모r델a활m용 eLifewLogoginrgk기반c기a술n개발capt-u1re0 t--h1e5- timestamps when the emotion is recog딥 닝 기반 어T펙 깅t프h nized with high confidence, and regard them as the events (for segmenting the multiple data streams), as shown in Fig. 2.
Busy
Busy where t and d indicate the timestamps and data mode, respectively. This work is focusing on collecting all possible lifelogging datasets, and discovering meaningful patterns for affective lifelogging, as shown in Fig. 1. For example, the streams are including – various bio-signals (e.g., EEG, blood pressure, and BPM) by wearable devices, 2. 연구–개lo발c의ati계on획b의y 우G수PS성-enabled smart devices,
– photos by camera, and 2.2– 핵te심xts기b술y social media.
•
감성 이벤트 패턴 마이닝 EEG 혈압 심박수 체온 사진 위치 텍스트 Happy
Busy
Sleepy "# "$ "%
Affective Event Pattern Mining ! – size window Definition 2 (Event). A set of events E is represented as ei tAg (2) where t@ and tA are the beginning and ending of an event, respectively. 2.1
Once the events are detected, the multiple multimodal streams S can be segmented. By measuring the correlation among the multiple multimodal streams in the same segment, we can find the event pattern library (shown in Fig. 2). Thus, this event-driven approach is similar to the labeling process for training.
As another important issue, we are focusing on relative scaling factor (RSF). When the correlation is computed, we want to consider the unique characteristics of the data stream. 2.2
With the proposed affective lifelogging framework, several applications and services will be developed with respect to the target user. 2. 연구개발의 계획의 우수성
– single individuals (shown in Fig. 3) 2.3 –예a상g결ro과up물of users in a certain location and time (shown in Fig. 4) • 감성 히스토리 기록 04:15 hours 13:14 hours 01:10 hours 03:11 hours 05:33 hours S-e융m러a복n합tic 콘Lif텐e츠VauSlotcia티l감브성라인지이와프S로ocial Inte레lig임enc워e 크모델 활F용iLgife.Lo3g.ginPg 기e반rs기o술n개a발l serv-ice16for- a21ff-ective lifelogging 딥 닝 기반 어펙 깅 프 Emotion1 Emotion2 Emotion3 Emotion4 S딥-e융m러a복n닝합tic 기콘Lif텐반e츠Va어uSlo펙tcia티l감브성라인지이와프S로oci깅al In프te레lig임enc워e 크모델 활용 Life Logging 기반 기술 개발 - 17 -- 22
Fig. 4. Spatiotemporal visualization 3
Since it is the on-going work, we want to mention the following open problems. 1. Evaluation issue: In order to evaluate the proposed framework, we need to collect real world data from users. 2. privacy: Most seriously, it is almost impractical to ask users to recall their own emotion states in the past. 4
In this paper, we have introduced a conceptual design on affective lifelogging framework. The proposed framework is based on event-based segmentation for the multiple multimodal stream.
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (NRF-2017R1A2B4010774). 1. Hong, M., Jung, J.J.: Mymoviehistory: Social recommender system by discovering social affinities among users. Cybernetics and Systems 47(1-2), 88–110 (2016)