=Paper=
{{Paper
|id=Vol-2166/afcai18-paper6
|storemode=property
|title=Towards Affective Lifelogging with Information Fusion
|pdfUrl=https://ceur-ws.org/Vol-2166/afcai18-paper6.pdf
|volume=Vol-2166
|authors=Jason Jung
|dblpUrl=https://dblp.org/rec/conf/afcai/Jung18
}}
==Towards Affective Lifelogging with Information Fusion==
https://ceur-ws.org/Vol-2166/afcai18-paper6.pdf
Towards Affective Lifelogging with Information Fusion
Jason J. Jung
Department of Computer Engineering
Chung-Ang University, Seoul, Korea
j2jung@gmail.com
Abstract. Recently, many of context-aware services are trying to exploit the
emotional contexts of the target users. The aim of this conceptual paper is to
discuss affective lifelogging framework which can recognize the emotions by in-
tegrating multimodal information from multiple sources. Moreover, we will men-
tion the open problems on affective lifelogging.
Keywords: Lifelogging · Affective computing · Information fusion · Stream syn-
chronization.
1 Introduction
Lifelogging (also known as Quantified self [3]), which is a historical dataset of user
activities (and behaviors), has been regarded as an important information for under-
standing their personal contexts (e.g., interests and patterns). Especially, with various
smart devices and wearable devices, it has been much easier for users to record their
lifeloggings. Given a particular domain, lifelogging has been studied for various appli-
cations, e.g., storification [2] and MyMovieHistory [1].
It is important for context-aware services to recognize emotional states of the target
users (whether they are happy or sad). In this study, we are interested in affective lifel-
ogging of users. By analogy, it means that the history of user’s emotional states can be
recorded.
However, it is impractical to ask users to answer what their current emotional state
is. It means that affective lifelogging has to be in a non-intrusive manner.
Thereby, this work is focusing on collecting all possible lifelogging datasets, and
discovering meaningful patterns for affective lifelogging, as shown in Fig. 1.
The outline of this paper is as follows. Sect. 2 shows the main idea on affective
lifelogging frameworks. In Sect. 3 and Sect. 4, we will address the open problems on
capturing the emotional state of users, and draw our conclusion of this on-going work,
respectively.
2 Affective lifelogging
We design the affective lifelogging framework with multimodel data streams.
Definition 1 (Multimodel streams). Given a set of data streams S = {si |i ∈ [1, N]},
multimodel streams can be represented as
S = hsi , τ, δ i|si ∈ S, (1)
2. 연구개발의 계획의 우수성
2.2 핵심 기술
• 시스템 시나리오
2 Jason J. Jung
EEG
Blood
Pressure
BPM
Temperature
Photo
stream
Location
Tweets Busy
Time
Busy
Fig. 1. Multimodal streams from multiple data sources
Semantic Life Vault
딥러닝 기반 어펙티브 라이프로깅 프레임워크 - 13 -- 18 -
- 융복합 콘텐츠 Social 감성인지와 Social Intelligence 모델 활용 Life Logging 기반 기술 개발
where τ and δ indicate the timestamps and data mode, respectively. This work is focus-
ing on collecting all possible lifelogging datasets, and discovering meaningful patterns
for affective lifelogging, as shown in Fig. 1. For example, the streams are including
– various bio-signals (e.g., EEG, blood pressure, and BPM) by wearable devices,
2. 연구개발의 계획의
– location by 우수성
GPS-enabled smart devices,
– photos by camera, and
2.2– 핵심
texts기술
by social media.
• 감성 이벤트 패턴 마이닝
Event Pattern
EEG
Library
혈압
심박수 Affective Event
Pattern Mining
"# "$ "%
체온
사진
! – size window
위치
텍스트 Happy Busy Sleepy
Fig. 2. Event-driven learning for affective lifelogging
Semantic Life Vault
딥러닝 - 10 the
Thereby, this framework can capture
기반 어펙티브 라이프로깅 프레임워크 -- 15 - timestamps when the emotion is recog-
- 융복합 콘텐츠 Social 감성인지와 Social Intelligence 모델 활용 Life Logging 기반 기술 개발
nized with high confidence, and regard them as the events (for segmenting the multiple
data streams), as shown in Fig. 2.
Towards Affective Lifelogging with Information Fusion 3
Definition 2 (Event). A set of events E is represented as
E = {ei |τ@ ei τA } (2)
where τ@ and τA are the beginning and ending of an event, respectively.
2.1 Learning by discovering correlation
Once the events are detected, the multiple multimodal streams S can be segmented. By
measuring the correlation among the multiple multimodal streams in the same segment,
we can find the event pattern library (shown in Fig. 2). Thus, this event-driven approach
is similar to the labeling process for training.
As another important issue, we are focusing on relative scaling factor (RSF). When
the correlation is computed, we want to consider the unique characteristics of the data
stream.
2.2 Applications and services
With the proposed affective lifelogging framework, several applications and services
will be developed with respect to the target user.
2. 연구개발의 계획의 우수성
– single individuals (shown in Fig. 3)
2.3 –예상
a group
결과물of users in a certain location and time (shown in Fig. 4)
• 감성 히스토리 기록
04:15 13:14 01:10 03:11 05:33
hours hours hours hours hours
Semantic Life Vault
딥러닝 기반 어펙티브 라이프로깅 프레임워크 - 16
Fig. 3. Personal service
- 융복합 콘텐츠 Social 감성인지와 Social Intelligence 모델 활용 Life Logging 기반 기술 개발for-- affective
21 - lifelogging
2. 연구개발의 계획의 우수성
2.3 예상 결과물
4 Jason J. Jung
• 사회 감성 지도
Emotion1
Emotion2
Emotion3
Emotion4
Semantic Life Vault
딥러닝 기반 어펙티브 라이프로깅 프레임워크
- 융복합 콘텐츠 Social 감성인지와 Social Intelligence 모델 활용 Life Logging 기반 기술 개발- 17 -- 22 -
Fig. 4. Spatiotemporal visualization
3 Open problems
Since it is the on-going work, we want to mention the following open problems.
1. Evaluation issue: In order to evaluate the proposed framework, we need to collect
real world data from users.
2. privacy: Most seriously, it is almost impractical to ask users to recall their own
emotion states in the past.
4 Conclusion
In this paper, we have introduced a conceptual design on affective lifelogging frame-
work. The proposed framework is based on event-based segmentation for the multiple
multimodal stream.
Acknowledgment
This work was supported by the National Research Foundation of Korea (NRF) grant
funded by the Korea government (MSIP) (NRF-2017R1A2B4010774).
References
1. Hong, M., Jung, J.J.: Mymoviehistory: Social recommender system by discovering social
affinities among users. Cybernetics and Systems 47(1-2), 88–110 (2016)
Towards Affective Lifelogging with Information Fusion 5
2. Jung, J.E., Hong, M., Hoang, L.N.: Serendipity-based storification: from lifelogging to story-
telling. Multimedia Tools Appl. 76(8), 10345–10356 (2017). https://doi.org/10.1007/s11042-
016-3682-x, https://doi.org/10.1007/s11042-016-3682-x
3. Swan, M.: The quantified self: Fundamental disruption in big data science and biological
discovery. Big Data 1(2), 85–99 (2013)