Overview of MediaEval 2020 Insights for Wellbeing: Multimodal Personal Health Lifelog Data Analysis

Abstract

This paper provides a description of the MediaEval 2020 “Multimodal personal health lifelog data analysis". The purpose of this task is to develop approaches that process the environment data to obtain insights about personal wellbeing. Establishing the association between people's wellbeing and properties of the surrounding environment which is vital for numerous research. Our task focuses on the internal associations of heterogeneous data. Participants create systems that derive insights from multimodal lifelog data that are important for health and wellbeing to tackle two challenging subtasks. The first task is to investigate whether we can use public/open data to predict personal air pollution data. The second task is to develop approaches to predict personal air quality index(AQI) using images captured by people (plus GAQD). This task targets (but is not limited to) researchers in the areas of multimedia information retrieval, machine learning, AI, data science, event-based processing and analysis, multimodal multimedia content analysis, lifelog data analysis, urban computing, environmental science, and atmospheric science.

INTRODUCTION

The association between people’s wellbeing and the properties of the surrounding environment is an essential area of investigation[ 2 ]. Although these investigations have a long and rich history[ 3 ], they have focused on the general population. There is a surprising lack of research investigating the impact of the environment on the scale of individual people. On a personal scale, local information about air pollution (e.g., PM2.5, NO2, O3), weather (e.g., temperature, humidity), urban nature (e.g., greenness, liveliness, quietness), and personal behavior (e.g., psychophysiological data) play an essential role. It is not always possible to gather plentiful amounts of such data as [ 6 ]. As a result, a key research question remains open: Can sparse or incomplete data be used to gain insight into wellbeing? Is there a hypothesis about the associations within the data so that wellbeing can be understood using a limited amount of data? Developing hypotheses about the associations within the heterogeneous data contributes towards building good multimodal models that make it possible to understand the impact of the environment on wellbeing at the local and individual scale. Such models as [ 8 ] [ 5 ] are necessary since not all cities are fully covered by standard air pollution and weather stations, and not all people experience the same reaction to the same environment situation. Moreover, images captured by the first-person view could give essential cues to understand that environmental situations in cases in which precise data from air pollution stations are lacking[ 7 ].

Let us imagine the following scenario. Yamamoto-san is using the Image-2-AQI app to know how harmful air pollution is by merely feeding captured images to the app. Simultaneously, at the urban air pollution center, the air pollution map is updated with Yamamoto-san’s contribution (e.g., images, annotation). Satoh-san, with some clicks on his smartphone, the environmental-based risk map application can show him the excellent route from A to B with less congestion and harmful air pollution as trafic risk prediction [ 4 ]. Simultaneously, less congestion from A to B is due to fewer people coincidentally traveling on the same route. Such simple apps are parts of the human-environment sustainable and co-existing system that have changed people’s pro-environmental behaviors.

The critical research question here is, “does the personal air quality be predicted by using other data that is easy to obtain?” 2

TASK DESCRIPTION

Task participants create systems that derive insights from multimodal lifelog data that are important for health and wellbeing. The ifrst dataset, namely “personal air quality data” (PAQD), includes air pollution data (PM2.5, O3, and NO2) and lifelog data (e.g., physiological data, tags, and images) collected by using sensors boxes, lifelog cameras, and smartphones along the predefined routes in a city. The second dataset, namely “global air quality data” (GAQD), includes weather and air pollution data collected over the city and provided by the government and crawled from related websites.

Personal Air Quality Prediction with public/open data. Participants predict the value of personal air pollution data (PM2.5, O3, and NO2) using only weather data (wind speed, wind direction, temperature, humidity) and air pollution data (PM2.5, O3, and NO2) from public/open data sources (e.g., stations, website). This subtask’s target is to investigate whether we can use public/open data to predict personal air pollution data. The personal air pollution data can be concerned as the regional air pollution data since these data a locally collected by people who carry personal equipment. In other words, the ground truth is data collected by sensor boxes carried by people.

Personal Air Quality Prediction with lifelog data. Participants predict the personal Air Quality Index using images captured by people (plus GAQD). The purpose of this subtask is whether we can use only lifelog data (i.e., pictures of the surrounding environment, annotations, and comments), plus some data from open sources (e.g., weather, air pollution data) to predict the personal air pollution data. 3

DATA DESCRIPTION

Personal air quality dataset (PAQD). PAQD were collected from March to April 2019 along the marathon course of the Tokyo 2020 Olympics and the running course around the Imperial Palace using wearable sensors [ 8 ](Fig. 1). There were five data collection participants assigned to five routes to collect the data. Routes 1–4 were along the marathon course for the Tokyo 2020 Olympics. Route 5 was the running course around the Imperial Palace. The length of each route was approximately 5 km. Each participant started data collection at 9 am every weekday, and it took approximately one hour to walk each route. Collected data contain weather data (e.g., temperature, humidity), atmospheric data (e.g., O3, PM2.5, and NO2), GPS data, and lifelog data (e.g., images, annotation).

Glocal air pollution dataset (GAPD). GAPD contains the atmospheric monitoring station data collected by the Atmospheric Environmental Regional Observation System ( AEROS ) in Japan [ 1 ]. AEROS contains real-time atmospheric data at every hour for 2032 meteorological monitoring stations across Japan. The atmospheric data includes eleven types of air pollutant data (SO2, NOx, NO, NO2, CO, Ox, NMHC, CH4, THC, SPM, and PM2.5), and four types of meteorological data (wind direction, wind speed, temperature, and humidity). 4

GROUND TRUTH AND EVALUATION

The ground truth for the dataset of the two subtasks is collected as follows: • For the Personal Air Quality Prediction with public/open data subtask: some parts of personal (PM2.5, O3, and NO2) data are deleted and saved as the ground truth. • For the Personal Air Quality Prediction with lifelog data subtask: the set of personal AQI are hidden and saved as the ground truth.

We use symmetric mean absolute percentage error (SMAPE), root mean squared error (RMSE), and mean absolute error (MAE) to do the evaluation. Each evaluation metrics defined as follow: (1) (2) (3) = = 1 Õ

| − | =1 ( + )/2

( − )2 =1 1 Õ =1 = | − | Where and mean the real value and the prediction value at time , and n is the total number of predicted data.

For each subtask, the evaluation method is applied as follows: • For the Personal Air Quality Prediction with public/open data subtask: We use the SMAPE /RMSE /MAE for comparing each air pollution factor PM2.5, O3, and NO2 with the ground truth. • For the Personal Air Quality Prediction with lifelog data subtask: We use the SMAPE/RMSE /MAE for comparing predicted AQI to the ground truth. 5

DISCUSSION AND OUTLOOK

Details on the methods and results of each individual participant team can be found in the working note papers of the MediaEval 2020 workshop proceedings.

References

[1] 2020 . Atmospheric Environmental Regional Observation System . http: //soramame.taiki.go.jp/.

[2] Zhao P. J Sato T. Zettsu K. Dang-Nguyen D. T. Gurrin C. Nguyen N. T. Dao , M. S. 2019 . Overview of MediaEval 2019: Insights for Wellbeing Task - Multimodal Personal Health Lifelog Data Analysis . In MediaEval Benchmarking Initiative for Multimedia Evaluation.

[3] Salvador Ruiz-Correa Darshan Santani and Daniel Gatica-Perez. 2018 . Looking South: Learning Urban Perception in Developing Cities . Trans. Soc. Comput 13 , 1 ( December 2018 ), 291 - 301 .

[4] M. Dao N. Nguyen and K. Zettsu . 2019 . Complex Event Analysis for Trafic Risk Prediction based on 3D-CNN with Multi-sources Urban Sensing Data . In 2019 IEEE International Conference on Big Data . 1669 - 1674 .

[5] M. Dao P. Vo , T. Phan and K. Zettsu . 2019 . Association Model between Visual Feature and AQI Rank Using Lifelog Data . In 2019 IEEE International Conference on Big Data . 4197 - 4200 .

[6] T Sato , M.S Dao , K Kuribayashi , and K. Zettsu . 2019 . SEPHLA: Challenges and Opportunities within Environment - Personal Health Archives . In International Conference on Multimedia Modeling . 325 - 337 .

[7] Sajedi H. Ramezani F. Vahdatpour , M. 2018 . Air pollution forecasting from sky images with shallow and deep classifiers . Earth Sci Inform 11 ( 2018 ), 413 - 422 .

[8] P. J Zhao and K Zettsu . 2019 . Decoder Transfer Learning for Predicting Personal Exposure to Air Pollution . In 2019 IEEE International Conference on Big Data . 5620 - 5629 .