Daily living studies usually require a physical environment such as cameras, sensor networks, or experimental space. Moreover, it is difficult to collect data by flexibly changing the conditions. In the future, for the analysis of daily life, it is necessary to combine data obtained from a physical space that can acquire real data and a virtual space that can flexibly change conditions and perform many experiments. In this study, using virtual space to enable various analyses of daily living activities, we propose a method to construct and augment knowledge graphs (KGs) based on the simulation results of daily living.
With the spread of virtual reality (VR) head-mounted displays and 3D game engines, it is possible to simulate daily activities using virtual space. In the future, there will be an increasing demand for analyzing daily activities using both the physical spaces that can collect real data and virtual spaces that can flexibly change conditions and perform many experiments.
In this study, we proposed VirtualHome2KG1. It is a system used for constructing and augmenting the KGs of daily living activities using virtual spaces (Fig. 1). The proposed system executes the simulation of the arbitrary agent’s activities and then records the spatiotemporal changes in virtual spaces at that time. This data is converted into the Resource Description Framework (RDF) formatted KG based on the designed ontology. Moreover, we proposed the method of human behavioral data augmentation by generating KGs of sequences consisting of multiple activities using the Markov chain. We evaluated the augmented KGs whether it is a natural sequence of activities in daily life using crowdsourcing.
In this study, we used VirtualHome [1] platform for simulating daily living activities in 3D virtual spaces. VirtualHome provides a dataset of daily household activities (e.g., “prepare coffee” and “watch horror movie”) consisting of multiple actions (e.g, “walk” and “sit”). Activity data is described as a sequence consisting of action, the object name, and object ID as follows: “[WALK] <remotecontrol> (108)”, “[GRAB] <remotecontrol> (108)”, . . . , “[WATCH] <tv> (106).” In this context, “activity” is a coarse-grained event and “action” is a fine-grained event to compose the activity. We use this dataset to execute a simulation in the VirtualHome and output the home situation at the time each action in the activity is executed. For example, the state of television is updated from “OFF” to “ON” when “SWITCHON tv (106)” is executed. In addition, we record the execution time of each action. 2.2
HomeOntology [2] has been developed based on the VirtualHome’s activity dataset. We reused the Activity class of the HomeOntology for representing activities. In this ontology, 12 subclasses have been defined such as EatingDrinking, HygieneStyling, and Leisure. In addition, 591 classes have been defined as subclasses of these 12 classes. The HomeOntology does not support representing spatiotemporal information. Hence, there is no way to represent the time it takes to execute an action and the three-dimensional coordinates. There is also no way to represent states, affordances, and attributes of objects. For the first issue, we solved it by reusing Time Ontology2 and X3D ontology3. We have defined new classes and properties for these representations for the second issue.
Figure 2 shows a part of an example of constructed KG based on our ontology. This figure is a part of the KG that represents the execution result of “Listen
to music,” which is one of the activity data provided by the VirtualHome. In actuality, there are more than 400 objects and their states. 2.3
The constructed KG is a cutout of the situation of single activity in daily life. However, in actual daily life, a person executes different activities sequentially. Hence, sequential data consisting of multiple activity connections are essential for analyzing daily life and lifestyle tendencies. Thus, we propose a method for extracting activities from the constructed KG, combine them without inconsistency between before and after situations of the activity, and then generate a KG of the activity sequence.
chain to generate the scenario of daily life probabilistically. In N -th Markov chain, i-th activity Xi occurs based on following conditional probability: P r(Xi) = P r(Xi | Xi−1, Xi−2, . . . , Xi−N ). In this study, a simple Markov chain with N = 1 generates an sequence. First, we collected sequential data of daily activities using Lancers4 as a crowdsourcing platform to calculate the transition probability among the activities. Then, we designed the crowdsourcing task that input the order of daily activities when a person living alone stays home in the morning, noon, and night. Workers divide indoor daily life into six activities in the morning, noon, and night and input their order in each pattern of weekdays and holidays. Thus, 36 activities are input per worker. As a result, 100 data
Scenario (a) (b) (c) (d) (e) Answer rate points were collected. Since each answer described the order of activities in the morning, noon, and night on weekdays and holidays, respectively, the total number of activity sequence data became 600. Finally, we calculated the transition probability based on this data and generated new activity sequences using the Markov chain. Because the generated activity sequence consists of the previously stated class names, it is necessary to instantiate each element. Thus, we instantiated activity sequences by randomly extracting subclasses (and instances with the same names as the subclass) of the class corresponding to each element in the generated activity sequence. An example of the generated activity sequences is given as follows: Do work → M ake toast → Eat dinner → Clean f loor → Listen to music → T ake nap. The activity class and its subclasses of the HomeOntology developed based on the VirtualHome dataset were reused in this step. We set the maximum length of the sequence to 6.
Second, the :State instances of objects in the situation before executing the first action and after executing the final actions of each activity are obtained using a SPARQL Protocol and RDF Query Language (SPARQL) query.
Then, two consecutive elements activityi (1 ≤ i ≤ 5) and activityi+1 are extracted from the activity sequence. The objects’ final states after executing activityi are compared with the objects’ first states before executing the other activityi+1. If there is no inverse state (e.g., CLOSED and OPEN) of the same objects, two activities can be executed consecutively. Finally, triples < activityi, nextActivity, activityi+1 > are created, and the related states are updated using a SPARQL DELETE/INSERT query.
activity sequence is natural as a routine of daily life. The following five activity sequences were generated as a sample and evaluated using crowdsourcing. (a) Go to toilet → Relax on sof a → Sleep → Eat dinner → Clean f loor →
Change sheets and pillow cases (b) Go to sleep → Clean f loor → W ash sink → Sleep → Sleep (c) Change sheets and pillow cases → Listen to music → Apply lotion →
Do work → Do work (d) M ake toast → Eat dinner → W ash sink → Listen to music → Do work →
Do work
(e) W atch movie → Sleep
The options in the questionnaire are as follows: the activity sequence (
The proposed system can generate KGs helpful to search for changes in daily life behavior and indoor environments with spatiotemporal information. Since each action that constitutes an activity is connected to the state of an object, it is possible to trace the trajectory of actions and objects in daily life. In addition, it is possible to analyze “Which objects are grabbed often?”, “Which objects often change their state from clean to dirty?”, and “How long does the agent sit for in a day?” Therefore, we believe the proposed system can be applied to the health care domain by combining external knowledge. As another use case, our system provides a mechanism to prepare a training dataset for activity recognition. 4
In this study, we proposed a method and presented a system for constructing and augmenting a KG using a 3D virtual space for advanced analysis of daily activities. Specifically, we designed the ontology to represent daily activities with situational changes in the space and executed simulation using VirtualHome. After that, we generated the KG of the simulation results. In addition, we proposed a method for generating KGs of various daily activities’ scenarios by combining the generated activity KGs to enable a contextual analysis of daily life.
In the future, we will focus on practical applications while also addressing issues with knowledge representation and data quality. We believe that combining our KG with explicit external knowledge in the healthcare domains will enable us to conduct a helpful analysis.
Acknowledgements This paper is based on results obtained from projects, JPNP20006 and JPNP180013 commissioned by the New Energy and Industrial Technology Development Organization (NEDO), and JSPS KAKENHI Grant Number JP19H04168.