Collecting lifelogs comprising data related to human life over a long period of time has progressed in recent years due to the widespread use of inexpensive small sensors. Understanding longterm daily life can be helpful in healthcare applications and for improving quality of life, for example. In this research, we propose a method for comparing two periods of daily life, which makes it possible to find similar and diferent life periods in lifelogs. Since a human life includes various behaviors, we propose an approach for comparing two periods of daily life based on the diferences in the behaviors performed in each period. We extracted episodes corresponding to several behaviors from motion data acquired with wearable sensors. Frequent episodes corresponding to frequent behaviors can be extracted using conventional episode mining methods. However, when comparing daily lives, not only a behavior frequency, but also its duration is important. Hence, we introduce long-duration episodes corresponding to long-running behaviors. In the proposed method, the similarity of two daily life periods is calculated based on diferences of long-duration and frequent episodes. We demonstrate with experiments using real-life data that the proposed method can establish the similarity between two periods of daily life correctly.
The collection and utilization of lifelogs comprising long-term
data about human life have advanced in recent years due to the
widespread use of inexpensive and small sensors. For example,
MyLifeBits is known as a research project focusing on lifelogs
[
Many studies have been reported on recognizing and
visualizing human activities from lifelogs using data mining techniques
[
To compare daily life over a few weeks, it should be
characterized with the behaviors performed during this period. When
many similar or diferent behaviors are exhibited in two distinct
periods, these periods can be treated as having similar or
diferent daily lives, respectively. Motion data acquired by a wristband
device equipped with an acceleration sensor is used in this study
to characterize human behavior as a series of events. Therefore,
episodes corresponding to diferent behaviors can be established
by applying an episode mining algorithm to motion data. For
example, an episode mining algorithm proposed by [
In this study, we propose an approach for comparing two periods of long-term daily life. We introduce a procedure for ifnding long-duration episodes by evaluating the sum of durations rather than the frequency of occurrence. We then compare two periods of daily life using both frequent and long-duration episodes extracted from the entire lifelog of motion data. To achieve this, we calculate the similarity between each pair of episodes that are locally long-duration or frequent in each period. The efectiveness of the proposed method is evaluated with experiments using real-life data. 2
This study evaluates the similarity between two periods of daily life of one user. We assume that human life consists of various behaviors, which we define as a series of activities performed with purpose. For example, the daily life of a student during an academic term consists of behaviors such as going to school, taking classes, and studying. During the summer vacation period, the student’s daily life consists of behaviors such as studying, working part-time, and hanging out with friends.
In this study, we compare two periods of daily life according to the diference of the behaviors the user performs during each period. Hence, the problem of daily life comparison can be formulated as comparing the similarity between a set of behaviors performed in periods 1 and 2. To characterize behaviors, motion data are used in this study. 3
We used the life recorder UW-301BT from Hitachi Systems as a sensor device (Figure 1) to collect motion data. Three-axis acceleration of the arm movement was measured as the device was worn on the wrist during the day and while sleeping. The output of this device is motion status data indicating a segment, in which the same type of motion status continues. This data is continuously output without interruption or overlapping. The degree of activity intensity is expressed in the following nine types of motion statuses: “rest”, “quiet (sitting quietly)”, “deskwork (sitting task)”, “light work (standing work)”, “work”, “exercise”, “walking”, “jogging”, and “not-wearing”. These statuses describe the intensity of activity level in order from “rest” to “exercise”. “Walking” indicates that a periodic activity was performed. “Jogging” indicates that a periodic and hard activity was performed.
A motion status data D = ⟨d1; : : : ; dn ⟩ is an ordered list of motion status events. A motion status event di = (mj ; Tsi ; Tei ) with 1 i n is a set consisting of a motion status, starting, and ending date-time. Here, mj 2 M (M is a set of motion statuses) is a motion status. Tsi and Tei are starting and ending date-time, where Tsi < Tei with 1 i n and Tei Tsi+1 with 1 i n 1. The duration of motion status di is Tei Tsi . Table 1 shows an example of motion status events. The motion status event in the first row means that “walking” continued for 3 minutes from “2018-10-15 08:02” to “2018-10-15 08:05”. The motion status event in the second row means that “deskwork” began soon after “walking”. The motion status data consisting of motion status events in Table 1 are arranged in order of starting date-time: ⟨(walking, 2018-10-15 08:02, 2018-10-15 08:05), (deskwork, 201810-15 08:05, 2018-10-15 08:44), : : : , (light work, 2018-10-15 16:40, 2018-10-15 16:55)⟩.
Conventional sensor devices equipped with an accelerometer
do not output motion status data. The majority of such devices
output the amount of activity per unit time as a time-series data.
A method for detecting segments, in which characteristic
activities are performed, from the amount of activity per unit time
has been proposed [
A behavior constituting a user’s daily life is a series of activities performed with purpose. In other words, a behavior appears in the motion status data as an ordered pattern of some motion statuses. For example, consider the behavior of going shopping by walking from home to a store. This behavior consists of walking from home to the store, shopping in the store, and walking from the store to home. In the motion status data, this behavior appears as “walking”, “light work”, and “walking”. In this way, behaviors can be expressed in the order of completion of corresponding motion statuses.
The motion status data represents a single event sequence,
while the motion status pattern represents the order of motion
statuses. Therefore, an episode extracted by applying episode
mining to the motion status data corresponds to a pattern of
the motion status indicating a behavior. Many episode mining
methods have been proposed [
We consider that frequent and long-duration episodes
correspond to behaviors performed mundanely. Exceptional
behaviors that are not normally performed can be identified by
browsing the lifelog. However, these exceptional behaviors are not
useful for comparing long periods of daily life, and hence are
excluded in this study. In this study, we use a formalism of MANEPI
[
Episode: Let α = ⟨m1; : : : ; mk ⟩ be an episode, where mj 2 M. The length of α is the number of motion statuses. Episode refers to the ordered pattern, in which each motion status appears in the order from m1 to mk . An episode α = ⟨m1; : : : ; mk ⟩ is a sub-episode of another episode β = ⟨m1′; : : : ; ms′ ⟩ if there exists 1 j1 < < jk s such that mi = mj′i for all i with 1 i k. For example, episode ⟨walkinд; deskwork; liдhtwork⟩ means that motion statuses appear in the order of “walking”, “deskwork”, and “light work”. Episode ⟨walkinд; deskwork⟩ is a sub-episode of episode ⟨walkinд; deskwork; liдhtwork⟩. Occurrence: In the motion status data D, if each motion status mj of the episode α is contained in D preserving the order, it denotes that α appears in D. The occurrence of α , denoted by occ (α ), is a segment, where α appears in D. occ (α ) is denoted as [Tsm1 ; Temk ] using the starting date and time of a motion status event including m1 and the ending date and time of a motion status event including mk . The duration of occ (α ) = [Tsm1 ; Temk ] is Temk Tsm1 . In Table 1, one of the occurrences of episode α = ⟨walkinд; deskwork; liдhtwork⟩ is occ (α ) = [2018-10-15 08:44, 2018-10-15 10:34].
To exclude redundant occurrences where the duration
required for one behavior becomes too long, we use a constraint
related to the span of an occurrence [
Moreover, we use a gap constraint, maxдap, that is the
maximum length of the time interval between two consecutive
motion status events in an occurrence [
Among the occurrences of episode α , a set of all occurrences satisfying maxspan and maxдap is the occurrence list of α , denoted in this paper as OCC (α ). In Table 1, the occurrence list of episode α = ⟨walkinд; deskwork; liдhtwork⟩ satisfying maxspan = 300 minutes and maxдap = 120 minutes is OCC (α ) = f[2018-10-15 08:02, 2018-10-15 10:34], [2018-1015 08:44, 2018-10-15 10:34], [2018-10-15 12:53, 2018-10-15 14.47], [2018-10-15 12:53, 2018-10-15 16:55], [2018-10-15 14:25, 2018-1015 16:55]g.
we consider episodes with minimal and non-overlapping
occurrences [
For occurrences [ts ; te ]; [ts′; te′ ] 2 OCC (α ), the relationship is non-overlapping if te < ts′ _ te′ < ts . A set of minimal and non-overlapping occurrences of episode α is denoted as MAMO (α ). Here, MAMO (α ) MO (α ). When several occurrences overlap, we preferentially select the occurrence of the earliest possible starting date-time as MAMO. For example, in MO (⟨walkinд; deskwork; liдhtwork⟩), [2018-10-15 12:53, 2018-10-15 14:47] and [2018-10-15 14:25, 2018-10-15 16:55] are overlapped. Set of minimal and nonoverlapping occurrences of ⟨walkinд; deskwork; liдhtwork⟩ is MAMO (⟨walkinд; deskwork; liдhtwork⟩) = f [2018-10-15 08:44, 2018-10-15 10:34], [2018-10-15 12:53, 2018-10-15 14:47]g.
The frequency of episode α , f req (α ), is the number of occurrences in MAMO (α );
f req (α ) = jMAMO (α ) j: For example, f req (⟨walkinд; deskwork; liдhtwork⟩) = 2. An episode satisfying a user-specified minimum value of the frequency, min f req, is called a frequent episode.
tdur (α ) = Total duration and long-duration episode: The sum of the durations of all occurrences contained in MAMO (α ) is named as the total duration of α , denoted by tdur (α ); ∑ (te
ts ): [ts;te ]2MAMO (α ) For example, tdur (⟨walkinд; deskwork; liдhtwork⟩) = 224 minutes. An episode satisfying a user-specified minimum value of the total duration, mintdur , is called a long-duration episode. 5
The problem of mining frequent episodes can be formulated as
extracting all episodes satisfying min f req. In this paper,
frequent episodes are extracted using MANEPI [
A set of minimal occurrences MO (α; m) is generated by deleting
non-minimal occurrences from OCC (α; m). By selecting
nonoverlapping occurrences from MO (α; m), we can attain a set
of minimal and non-overlapping occurrences MAMO (α; m). If
MAMO (α; m) satisfies min f req, ⟨α; m⟩ is outputted as a frequent
episode. This process of extending episodes is repeated.
Redundant candidate episodes can be pruned because the downward
closure for frequent episode mining [
The problem of mining long-duration episodes can be formulated as extracting all episodes satisfying mintdur . Longduration episodes can be extracted in the same way as MANEPI. However, to extract all long-duration episodes, it is necessary to examine episodes which do not satisfy min f req. A longduration episode is an episode satisfying mintdur and having an unbounded frequency. An episode that does not satisfy min f req can also be long-duration. Consider Table 1 with min f req = 3 and mintdur = 200 minutes. For episode α = ⟨walkinд; deskwork; liдhtwork⟩, f req(α ) = 2, and tdur (α ) = 224 minutes. α is not extracted by frequent episode mining, since α does not satisfy min f req. However, α is a long-duration episode since α satisfies mintdur . In long-duration episode mining, we have to consider the total duration rather than frequency.
However, the Apriori property does not hold for the total duration. For episodes α and β (α is a sub-episode of β), the total duration of β may be longer than that of α . For example, in Table 1, tdur (⟨walkinд; liдhtwork⟩) = 81 minutes and tdur (⟨walkinд; deskwork; liдhtwork⟩) = 224 minutes. Even if α does not satisfy mintdur , β must be examined; therefore, pruning cannot be performed using the total duration to extract longduration episodes. However, it is not necessary to examine all candidate episodes. For an episode that becomes a long-duration episode, the frequency is minimum when the duration of all occurrences is maxspan. In other words, the lower bound of the frequency of a long-duration episode is ⌈ mmainxtsdpuarn ⌉. For example, when mintdur = 1000 minutes and maxspan = 300 minutes, an episode X such that f req(X ) < 4 (= ⌈ 1300000 ⌉) cannot be a long-duration episode. Therefore, it is suficient to extract episodes with this lower bound of frequency in MANEPI. Hereafter, the minimum frequency in a long-duration episode mining is denoted as low f req: mintdur low f req = ⌈maxspan ⌉: When an episode satisfying low f req is extracted, it is output as a long-duration episode if it satisfies mintdur .
The procedure of our method for mining long-duration episodes is as follows.
mintdur , maxspan, maxдap) Input: a motion status data D, a frequency threshold of an episode min f req, a total duration threshold of an episode mintdur , a duration constraint of an occurrence maxspan, a gap constraint of an occurrence maxдap Output: all long-duration episodes 1: F I := All motion statuses that satisfy low f req 2: Generate minimal occurrence list MO for each motion status in F I 3: foreach motion status h 2 F I do 4: foreach motion status m 2 F I do 5: ExtendEpisode(h, m) 6: end 7: end
Input: episode α , motion status m Output: long-duration episode β 8: Generate episode β by appending m to α 9: MO (β ) = ∅ 10: foreach occ [oas ; oae ] 2 MO (α ) do 11: Extract the occurrence [oms ; ome ] 2 MO (m) such that oms < oms′ ^ oms oae ^ oms′ > oae for any [oms′ ; ome′ ] 2 MO (m) 12: if (ome oas maxspan) ^ (oms oae maxдap) then 13: Append [oas ; ome ] to MO (β ) 14: endif 15: end 16: Delete occ [os ; oe ] from MO (β ) such that oe == oe′ ^ os > os′ for any [os ; oe ]; [os′ ; oe′ ] 2 MO (β ) 17: MAMO (β ) = ∅ 18: foreach occ [os ; oe ] 2 MO (β ) do 19: if (os oe′ for any [os′ ; oe′ ] 2 MAMO (β )) then 20: Append [os ; oe ] to MAMO (β ) 21: endif 22: end 23: if jMAMO (β )j low f req then 24: tdur (β ) = ∑[os;oe ]2MAMO (β ) (oe os ) 25: if tdur (β ) mintdur then 26: Output β as long-duration episode 27: endif 28: foreach motion status m′ 2 F I do 29: ExtendEpisode(β, m′) 30: end 31: endif
The episode that satisfies low f req can become a longduration episode; hence, we have to examine episodes that satisfy low f req. First, all motion statuses that satisfy low f req are extracted (line 1), and the minimal occurrence lists of these motion statuses are generated at the same time (line 2). Here, for a motion status m, OCC (m) and MO (m) are equivalent. Then, an episode extended with one motion status is examined in the ascending order of short episodes (line 3-7). We can extract all long-duration episodes using this procedure.
In ExtendEpisode procedure for extending episode α by motion status m, episode ⟨β⟩ = ⟨α; m⟩ is generated (line 8), occurrence of m is added to occurrences of α to generate the occurrence list of β (line 9-15), non-minimal occurrences are deleted (line 16), and the minimal occurrence list of β, MO (β ), is generated. Then, non-overlapping patterns are sequentially selected from the head of MO (β ) (line 17-22), and MAMO (β ) is generated. When MAMO (β ) satisfies the lower bound of frequency for long-duration episodes (line 23), the total duration of β is calculated (line 24). When the total duration of β satisfies the minimum threshold of the total duration (line 25), β is output as a long-duration episode (line 26). Since an episode that satisfies low f req can become a long-duration episode, ExtendEpisode procedure is repeated (line 28-30). 6
We compare two periods Pd1 and Pd2 of daily life by measuring their similarity based on motion status data D, minimum frequency min f req, minimum total duration mintdur , maximum occurrence duration maxspan, and maximum gap maxдap.
We focus on behaviors performed mundanely in the user’s daily life. First, long-duration and frequent episodes are extracted from the entire motion status data. In the two periods, episodes corresponding to actual behaviors are selected. Then, the similarity the two periods is evaluated based on the diference of the episodes included in each period. The procedure of the proposed method is as follows.
(1) Extracting global long-duration episodes and global frequent episodes: Extract long-duration and frequent episodes from the entire motion status data D. We call these episodes as global long-duration and global frequent episodes, respectively. Non-maximal episodes are deleted from these global episodes. Here, a maximal episode is an episode that is not a sub-episode of any other episode. (2) Taking out local long-duration episodes and local frequent episodes: Episodes that are locally long-duration episodes and locally frequent episodes in each period are selected from their global counterparts. Locally long-duration and frequent episodes in a period Pdi are long-duration and frequent episodes that satisfy long-duration and frequent conditions mintdur and min f req in this period, respectively.
For each global long-duration or frequent episode д, all occurrences satisfying the following two conditions are selected from MAMO (д): starting date-time of occ (д) date-time of the first day of period Pd1, ending date-time of occ (д) date-time of the last day of period Pd1.
When the total duration or frequency of the selected occurrences satisfies the relative mintdur or min f req ratio of period Pdi to the entire motion data, д is appended to LLPdi or LFPdi , respectively. Here, LLPdi and LFPdi are the sets of local long-duration and frequent episodes satisfying the relative condition in the period Pdi . (3) Evaluating similarity of daily lives:
The similarity between two periods is calculated from the set of their corresponding local long-duration and local frequent episodes using the Jaccard index.
Here, the Jaccard index is calculated as jLLPd1 \LLPd2 j+jLFPd1 \LFPd2 j , where LLPdi is a set jLLPd1 [LLPd2 j+jLFPd1 [LFPd2 j of local long-duration episodes of period Pdi and LFPdi is a set of local frequent episodes of period Pdi ; i = 1; 2.
In (1), long-duration and frequent episodes are extracted from the entire motion status data. These episodes are not extracted from the motion status data of each period to be compared. Although the considered comparison period is several weeks, mintdur and min f req become too small in each period. Episodes that are long-duration or frequent only in the considered period are incidentally extracted. Therefore, episodes that do not correspond to daily behaviors may be extracted. Hence, we extract global long-duration episodes and global frequent episodes corresponding to daily behaviors from the entire motion status data.
Our method uses only maximal episodes to compare two periods of daily life since the similarity between these periods becomes high when using non-maximal episodes. Suppose there are episodes α and β corresponding to certain behaviors, and episode γ is a sub-episode of both α and β . Furthermore, episode α appears only in one of the two considered periods, while episode β appears only in the other period. Episode γ appears in both two periods. Let episodes α and β represent diferent behaviors in the two periods. Then, assuming that the two periods are similar would be incorrect since γ is a sub-episode of both α and β . However, episode γ becomes a factor to raise the similarity of two periods of daily life erroneously since γ appears in both periods. Hence, episode γ should not be considered as it is not an episode corresponding to a certain behavior. Therefore, only maximal episodes are considered in this study.
Our method compares two periods of daily life using local long-duration and frequent episodes in each period. In (2), all long-duration and frequent episodes are selected from global frequent and long-duration episodes, respectively. For example, consider the case where the entire motion status data period is 600 days, min f req is 1000, mintdur is 72000 minutes, and the number of days in the period Pd is 30 days. When the total duration of the selected occurrences satisfies 7200 63000 , the episode is determined as local long-duration episode in period Pd and appended to LLPd . When the frequency of the selected occurrences satisfies 1000 63000 , the episode is determined as local frequent episode in period Pd and appended to LFPd .
In (3), our method outputs the Jaccard index as the similarity measure between two periods of daily life. When two periods are similar, the number of common long-duration and frequent episodes increases and the similarity becomes a value close to 1. On the other hand, when the user’s daily life is diferent in the two periods, the similarity is close to 0. 7
We examine whether the proposed method can compare two periods of daily life using real-life data. In particular, we use motion status data collected from six participants. The period covered by each data set varies from 0.8 to 6.5 years (average is about 2.2 years). In this experiment, each motion status was divided into two motion statuses at the median of the duration. Five motion statuses, namely, “rest”, “quiet”, “deskwork”, “light work”, and “walking”, were divided. Other motion statuses were not divided due to their low frequency. Therefore, the total number of motion statuses was 14.
We prepared 18 pairs of the periods in which similar daily lives were confirmed and 27 pairs of the periods in which diferent daily lives were confirmed. For each pair of the periods, their similarity was calculated using the proposed method. Here, we set the involved parameters as follows: min f req is 4 per week, mintdur is 480 minutes per week, maxspan is 240 minutes, and maxдap is 45 minutes.
First, we evaluated the efect of using maximal episodes. Figures 2 and 3 show the similarity between the pair of periods evaluated using the proposed method with only maximal and all episodes, respectively.
It can be noticed from Figures 2 and 3 that the proposed method can compare two periods of daily life correctly. The similarity of the two periods, which the two respective subjects spent in the same way, is high, whereas that of diferent daily lives is low. When only maximal episodes are considered, the similarity of similar daily lives is higher than that of diferent daily lives. The error ratio, which we define as the ratio of cases in which the similarity of similar daily lives is smaller than that of different daily lives for each participant, was 0%. This means that the proposed method using only maximal episodes can compare two periods of daily life correctly. On the other hand, the similarity becomes high when all episodes are used even for different daily lives. The error ratio when using all episodes was 43%. This means that we cannot correctly compare daily lives in many cases when all episodes are used. Note that non-maximal episodes are included when considering all episodes. A nonmaximal episode is a sub-episode of multiple maximal episodes. This means that a non-maximal episodes is a part of multiple behaviors. Therefore, even if a maximal episode is long-duration or frequent in one period, its sub-pattern often becomes longduration or frequent in two diferent periods. This trend becomes higher as the length of an episode shortens.
Next, we evaluated the efect of using both long-duration and frequent episodes. Figures 4 and 5 show the similarity between two periods of daily life when using only long-duration or frequent episodes.
Figures 2, 4, and 5 show that the range of the similarity of daily lives increases when using only either long-duration or frequent episodes. In particular, there are cases when the diference between the similarity of similar daily lives and that of diferent daily lives is bigger providing that only long-duration episodes are used. However, the lowest similarity value of similar daily lives decreases, and the maximum similarity value of diferent daily lives increases, which results in wrong comparison of daily lives. The error ratios when using only either long-duration or frequent episodes were 0:9% and 8%, respectively. This means that daily lives cannot be compared correctly when using only one of the episode types. The most accurate results can be attained when using both long-duration and frequent episodes, even if accuracy for one of them is low. 8
In this study, we proposed a method for comparing two periods of daily life based on episode mining of a lifelog of motion data. Conventional episode mining algorithms can extract frequent episodes, which correspond to frequently occurring behaviors. To characterize a human daily life, behaviors lasting for a long period of time are also important. Hence, we proposed an algorithm for mining long-duration episodes, which are evaluated based on their total duration rather than frequency. In this way, long-duration episodes with low frequency can be extracted.
The proposed method for comparing two periods of daily life uses both long-duration episodes and frequent episodes. For global long-duration and global frequent episodes extracted from the entire motion data, local long-duration episodes and local frequent episodes are selected for each period. Then, the similarity between two periods of daily life is calculated based on the sets of local long-duration and local frequent episodes for each period. By using only maximal episodes, our method avoids using redundant episodes. Experimental results on real-life data showed that the proposed method can correctly compare periods of daily life.
In this paper, the time slot during which an episode persists is not considered. In the future, we plan to extend our method to distinguish episodes using time slots, during which a pattern persists.
This work was supported by JST CREST Grant Number JPMJCR1503, Japan.