Lifelogging (also known as Quantified self [3]), which is a historical dataset of user activities (and behaviors), has been regarded as an important information for understanding their personal contexts (e.g., interests and patterns). Especially, with various smart devices and wearable devices, it has been much easier for users to record their lifeloggings. Given a particular domain, lifelogging has been studied for various applications, e.g., storification [2] and MyMovieHistory [1].
It is important for context-aware services to recognize emotional states of the target users (whether they are happy or sad). In this study, we are interested in affective lifelogging of users. By analogy, it means that the history of user's emotional states can be recorded.
However, it is impractical to ask users to answer what their current emotional state is. It means that affective lifelogging has to be in a non-intrusive manner.
Thereby, this work is focusing on collecting all possible lifelogging datasets, and discovering meaningful patterns for affective lifelogging, as shown in Fig. 1.
The outline of this paper is as follows. Sect. 2 shows the main idea on affective lifelogging frameworks. In Sect. 3 and Sect. 4, we will address the open problems on capturing the emotional state of users, and draw our conclusion of this on-going work, respectively.
We design the affective lifelogging framework with multimodel data streams.
Definition 1 (Multimodel streams). Given a set of data streams S = {s i |i ∈ [1, N]}, multimodel streams can be represented as Thereby, this framework can capture the timestamps when the emotion is recognized with high confidence, and regard them as the events (for segmenting the multiple data streams), as shown in Fig. 2.
Definition 2 (Event). A set of events E is represented as
where τ and τ are the beginning and ending of an event, respectively.
Once the events are detected, the multiple multimodal streams S can be segmented. By measuring the correlation among the multiple multimodal streams in the same segment, we can find the event pattern library (shown in Fig. 2). Thus, this event-driven approach is similar to the labeling process for training.
As another issue, we are focusing on relative scaling factor (RSF). When the correlation is computed, we want to consider the unique characteristics of the data stream.
With the proposed affective lifelogging framework, several applications and services will be developed with respect to the target user.
single individuals (shown in Fig. 3) a group of users in a certain location and time (shown in Fig. 4)
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (NRF-2017R1A2B4010774).
1. Evaluation issue: In order to evaluate the proposed framework, we need to collect real world data from users. 2. privacy: Most seriously, it is almost impractical to ask users to recall their own emotion states in the past.