Overview of MediaEval 2019: Insights for Wellbeing Task Multimodal Personal Health Lifelog Data Analysis

Abstract

This paper provides a description of the MediaEval 2019 "Multimodal personal health lifelog data analysis". The purpose of this task is to develop approaches that process the environment data to obtain insights about personal wellbeing. Establishing the association between people's wellbeing and properties of the surrounding environment which is vital for numerous research. Our task focuses on the internal associations of heterogeneous data. Participants are expected to process the mixed environment data(e.p weather, air pollution, lifelog images, etc.) to tackle two challenging subtasks. The first one is to develop a hypothesis about the associations within the heterogeneous data and build a system that is able to correctly replace segments of data that have been removed. The second one is to develop approaches to automatically predict personal AQI (Air Quality Index) at specific positions and time durations.

INTRODUCTION

The association between people’s wellbeing and the properties of the surrounding environment is an important area of investigation. Numbers of studies have suggested that human health both physical and mental are highly influenced by the surrounding environment. For example, the environmental elements (pollution, weather) have a high correlation with cardio [ 6 ]. The greenness exposure benefits for mental health[ 8 ].

Although these investigations have a long and rich history[ 9 ] [ 5 ], they have focused on the general population. There is a surprising lack of research that investigates the impact of the environment at the scale of individual people. At personal scale, local information about air pollution (e.g. PM2.5, NO2, O3), weather (e.g. temperature, humidity), urban nature (e.g. greenness, liveliness, quietness), and personal behavior (e.g. psychophysiological data) play an important role. It is not always possible to gather plentiful amounts of such data. As the result, a key research question remains open: Can sparse or incomplete data can be used to gain insight into wellbeing? In other words, is there a hypothesis about the associations within the data so that wellbeing can be understood by using a limited amount data?

Developing hypotheses about the associations within the heterogeneous data contributes towards building good multimodal models that make it possible to understand the impact of environment on wellbeing at the local and individual scale. Such models are necessary since not all cities are fully covered by standard air pollution and weather stations, and not all people experience the same reaction to the same environment situation. Moreover, images captured by the first-person view[ 4 ] could give important cues to help understand that environmental situation in cases in which precise data from air pollution stations is lacking. Thus, the purpose of this task is to develop approaches that process the environment data to obtain insights about personal wellbeing. 2

TASK DESCRIPTION

Participants receive a set of weather and air pollution data, lifelog images, and tags recorded by people who wear sensors, use smartphones and walk along pre-defined routes inside a city and develop approaches that process the data to obtain insights about personal wellbeing. Participants in this task tackle two challenging subtasks:

Segment Replacement. Task participants develop a hypothesis about the associations between the data and build a system that can replace segments of data that have been removed correctly. In particular, the task will have a set of 10 queries; each query gives the participants some records of data and asks them to predict the missing values.

Personal Air Quality. Task participants develop approaches to automatically predict personal AQI (Air Quality Index) at specific positions and time durations using either the underspecified data or the full data from a subset of data sources. The aim of Personal AQI is to measure the wellbeing of individual people with respect to the quality of the air that they are breathing. In particular, the task will have one query; this query gives the participants some records of data and asks them to predict the Personal Air Quality of one group who are moving along a particular route. The AQI in this task is calculate by Taiwan AQI[ 1 ].

For each task, participants can submit up to five runs. At least one run using only data released by the ask organized must be submitted. The participants are allowed to use third-party data for a maximum of two runs. 3

DATA DESCRIPTION

The Insight for Wellbeing task introduces a novel dataset, namely SEPHLA[ 7 ] created by the data collection campaign, namely DATATHON organized in Fukuoka City, Japan [ 3 ] in 2018 and 2019. The SEPHLA is dataset at the individual scale contained as follows: • Walking routes: street names, GPS, time. • Psychophysiological: footsteps, heart rate. • Pollutant concentrations: PM2.5, NO2, O3. • Weather variables: temperature, humidity. • Image data: first-person view images. • Urban perception tags: urban, clean, noisy, greenness, trafifc, • Emotional tags data: excited, depressed, degree of fatigue, breathe The SEPHLA is collected by wearable sensors, lifelog-cameras, and smart-phones attached to each data collector.

In 1st DATATHON from March 10th to April 8th, 2018, 22 participants divided into 5 groups to collect the SEPHLA data in 5 diferent routes.These five diferent routes include five diferent urban scenes: • Root1 Momochihama, seaside area. • Root2 Ohori Park, park with lake and greenness. • Root3 Tenjin, business district in the city. • Root4 Kashi, residential area in the city.

• Root5 Fukuoka Airport, transportation hub.

The images data taken by smartphones during the five routes. Most of the images are taken at the predefined checkpoints.

The 2nd DATATHON was held on March 23th and April 6th, 2019. The 27 participants on the first day and 25 participants on the second day were divided into 5 groups to collect the SEPHLA data. In 2nd DATATHON, every groups started at the same location and freely select the route from the predefined checkpoint to reach the goal point. The images data taken by smartphones and lifelog cameras.

Images data were annotated with the outputs of a visual concept detector, which provided three types of outputs (Attributes, Categories, and Concepts). Two visual concepts, which include attributes and categories of the place in the image, are extracted using PlacesCNN. The remaining one has detected object category and its bounding box extracted by using Faster R-CNN.

All individual information, especially in images, is blurred for privacy purposes. The copyright of SEPHLA belongs to the National Institute of Information and Communications Technology, Japan (NICT) and will be released for participants only for research purposes.

Participants will also receive the sensor station data for the same period as the SEPHLA dataset. The sensor stations data[ 2 ] contains 16 sensor stations in Fukuoka. Each sensor station collected 11 types of air pollutants data (SO2, NOx, NO, NO2, CO, Ox, NMHC, CH4, THC,SPM, PM2.5) with 4 types of weather data (Wind direction (WD), Wind speed (WS), Temperature (TEMP), Humidity (HUM)) every hour.

4 GROUND TRUTH

The ground truth for the dataset of the two subtasks is collected as follows: • For the Segment Replacement subtask: The correlation among data types collected along a route during a special time duration is manually calculated. All data segments with high correlation are extracted and labeled. Some of data types in these segments will be hidden and the rest is released for participants. For images data, concepts, categories, and scene are automatically detected using Google Visual API. • For the Personal Air Quality subtask: A set of specific time segments along the routes is labelled with information based on global AQI provided by Fukuoka City plus local AQI calculated by individual sensing data, as well as with tags contributed by the datathon participants that reflect their perceptions of the urban environment and experienced emotions. Images are also semi-automatically annotated with labels relating to the impact of air pollution and weather on vision such as cloudy, fog, windy, and sunny.

5 EVALUATION

The evaluation for each subtask is defined as follows:

Segment Replacement: The evaluation will calculate the diferences between the predicted values and the ground truth values using a simple measure that applies the arithmetic mean of the normalized Euclidean distances (L2 distance). Min-max normalizing will be applied to scale the values into a suitable range [ 0, 1 ].

Personal Air Quality: The evaluation will count based on the diferences of the predicted classes from right ones, by applying the arithmetic mean of the absolute distances (L1) between the pairs.

6 DISCUSSION AND OUTLOOK

Multimodal personal health lifelog data analysis task

Details on the methods and results of each individual participant team can be found in the working note papers of the MediaEval 2019 workshop proceedings.

References

[1] 2019 . Air Quality Index . https://taqm.epa.gov.tw/taqm/en/b0201.aspx.

[2] 2019 . Atmospheric Environmental Regional Observation System . http: //soramame.taiki.go.jp/.

[3] 2019 . Datathon. http://datathon.jp.

[4] D.T Dang-Nguyen , L Piras , M Riegler , L Zhou , M , Lux, and C Gurrin . 2018 . Overview of ImageCLEFlifelog 2018: Daily Living Understanding and Lifelog Moment Retrieval . In CLEF2018 Working Notes.

[5] M. S Dao and K Zettsu . 2018 . Complex Event Analysis of Urban Environmental Data based on Deep CNN of Spatiotemporal Raster Images . In 2018 IEEE International Conference on Big Data.

[6] D Santani , S Ruiz-Correa , and D Gatica-Perez . 2018 . Looking South: Learning Urban Perception in Developing Cities . ACM Transactions on Social Computing 1 , 3 (Dec 2018 ).

[7] T Sato , M.S Dao , K Kuribayashi , and K. Zettsu . 2019 . SEPHLA: Challenges and Opportunities within Environment âĂŞ Personal Health Archives . In International Conference on Multimedia Modeling . 325 - 337 .

[8] H.J. et al Song. 2019 . Association between Urban Greenness and Depressive Symptoms: Evaluation of Greenness Using Various Indicators . Int. Journals of Environmental Research and Public Health 16 , 2 (Jan 2019 ), 291 - 301 .

[9] P. J Zhao and K Zettsu . 2018 . SConvolution Recurrent Neural Networks for Short-Term Prediction of Atmospheric Sensing Data . In The 4th IEEE International Conference on Smart Data . 815 - 821 .