When collecting and processing data recorded by sensors for any applications, noisy and missing data is an important problem that need to be address. This paper presents two approaches we use to predict missing air quality data in MediaEval Insight for Wellbeing Task. The first approach based on other data attributes like temperature and humidity, and the second based on data recorded from other sources. Evaluating the experimental results using the average L2 distance, we got the score of 0.9013 for the first approach and 0.0155 for the second approach.
Environmental data can be used to analyse diferent aspects for
the deveopment of the society, including the quality of personal
health [
In the MediaEval Life Well Being 2019 task[
Based on the organization of the data, we found there are 2 main approaches to predict the P M2.5 in the queries. In the first approach, we want to explore if it would be possible to find relationship between P M2.5 values and other obtained attributes (Section 2.1). In the second approach, because in each question, there are a number of people walking in the same region in roughly the same time interval, we propose to combine values from multiple people, to infer the missing data segment of P M2.5 (Section 2.2). 2.1
Inference from other attributes 2.1.1 Using test-set only. After observing over the chart of a dataset, we omitted some features have a mean value close to zero like N O2,33, some category features. We found the feature about the location of all users at any time are not so diferent, so we concluded that data about location and P M2.5 are not had close ∗The first two authors contributed equally to the paper relationships. Because we don’t want unexplainable-relationship between coordinates and P M2.5, we need a solution clear and stable as much as we can. We do not have adequate data about location and P M2.5, also any pretrain model to mapping from location information to what we need. We assume coordinates value have a relationship with other attributions so coordinates’ meaning can be implicitly represented through temperature and humidity feature [Figure 1], so that if we found the right function to mapping from temperature and humidity to P M2.5, we also have the coordinates information in the result, also simplify the data. As a result, we push normalized temperature and humidity data through a multi-layer perceptron model to approach the problem.
2.1.2 Using test-set as validation. We do the same as first run at this run, however, the motivation of this run is that we do not try to overfit the testing dataset of organizing, in this approach we try to generalize the method. We use the dataset development (unrated) in the contest to train set, the oficial dataset to the validation set. This task is preprocessed data most clean-able and optimize the loss on validation (oficial dataset). 2.2
First, we examine and compare the coordinates and trajectories of people within the same group (same question) through time, and ifnd that in most cases, people in the same group walked in roughly the same route, and they were at the same location together at every moment along the way (the start and end times of each person may vary) [Figure 2].
Therefore, we can conclude that given a specific time, the P M2.5 values recorded by people within the same group are highly related because they recorded the P M2.5 value of the same location at the same time, and we could guess the missing P M2.5 values by the corresponding P M2.5 values of other people in the same group. Hoang-Anh Le, Thang-Long Nguyen-Ho, Minh-Triet Tran However, comparing P M2.5 values of all people in the same group, we find that these values vary considerably. Thus, we implement some statistical method to predict missing P M2.5 values from corresponding P M2.5 values of other people.
2.2.1 Average. We predict the missing P M2.5 values by taking the average of P M2.5 values of other people in the group in the corresponding time. [Figure 3]. However, the P M2.5 data of these people are scatter over the time interval and not available for every second. There for we use 1D linear interpolation to predict P M2.5 data for each person at every second before taking the average.
2.2.2 Average with bias. The average of P M2.5 values of all people is only a reasonable prediction for the true P M2.5 value of the environment at that moment. However, most sensors can not produce these true values, each sensor has its own inaccuracy. And since we want to predict the P M2.5 values recorded by a specific sensor, we want to take into account this inaccuracy. Since the random noise are dificult to evaluate, we only consider the bias problem - the sensor consistently records values that lower or higher than the true values by a certain amount (the bias value).
To estimate the bias, we calculate the diference between the average P M2.5 values and the P M2.5 values of that sensor at each moment these values available, and take the average of these differences. After that, we add this bias to the predict values of the previous run.
2.2.3 Average (outlier removed) with bias. We observe that there are some noise in certain sensor that make some recorded P M2.5 values become very high, having very large diferences with the values of other sensor at corresponding time, making the average values become more inaccurate to estimate the true values. To remove these noise, we check at a certain time, if the diference between the value of a sensor with the average value larger than the variance by a threshold factor, than we ignore this value and recalculate the average. We also recalculate the bias value and apply it for this run.
Table 1: Oficial evaluation result (provided by organizers) Approach 1 2 The table above shows the results of each method mentioned earlier. In this table, the scores of each run is the means of L2 distance between the predicted results and the ground truth. Our experiment results show that the second approach (predict based on other people within the group), achieve fairly good results. The result of the first approach (predict based on temperature and humidity) are not so good as the average L2 distances are still quite large. We think the reason is probably because only temperature and humidity could not give us enough information to predict the P M2.5 values, and to have really good predictions, we should combine the information about variations of P M2.5 values through time, the temperature and humidity values and the P M2.5 values of other people in the same group. 4
We propose two simple approaches for the Life Well Being Problem. The first approach uses a neural network to predict P M2.5 values from other factors like temperature and humidity. The second approach using the P M2.5 values recorded by other people at the same location and at the same time. These methods are simple but can predict the missing values quite efectively.
We think these methods could be improved further by combining them together (meaning take into account both the other attributes values and other P M2.5 values), having a more efective noise removal method, or building a more complex regression model. Insight for Wellbeing Task
Research is supported by Vingroup Innovation Foundation (VINIF) in project code VINIF.2019.DA19. We would like to thank AIOZ Pte Ltd for supporting our team with computing infrastructure.