Nowadays, the vast amount of produced mobility data (by sensors, GPS-equipped devices, surveillance networks, radars, etc.) poses new challenges related to mobility analytics. In several application, such as maritime or air-trafic data management, data analysis of mobility data requires weather information related to the movement of objects, as this has significant efect on various characteristics of its trajectory (route, speed, and fuel consumption). Unfortunately, mobility databases do not contain weather information, thus hindering the joint analysis of mobility and weather data. Motivated by this evident need of many real-life applications, in this paper, we develop a system for integrating mobility data with external weather sources. Our system is designed to operate at the level of a spatio-temporal position, and can be used to eficiently produce weather integrated data sets from raw positions of moving objects. Salient features of our approach include operating in an online manner and being reusable across diverse mobility data (urban, maritime, air-trafic). Further, we extend our approach: (a) to handle more complex geometries than simple positions (e.g., associating weather with a 3D sector), and (b) to produce output in RDF, thus generating linked data. We demonstrate the eficiency of our system using experiments on large, real-life data sets.
The ever-increasing rate of generation of mobility data by modern
applications, including surveillance systems, radars, and
GPSenabled devices, poses new challenges for data management
and analysis. To support advanced data analytics, mobility data
needs to be enriched by associating spatio-temporal positions of
moving objects with external data sources, as the data is ingested.
This problem is known as data integration and is particularly
challenging in the era of Big Data [
One significant data integration task common in all domains,
including urban, maritime and air-trafic, is related to weather
information. This is due to the fact that weather plays a critical role
in the analysis of moving objects’ trajectories [
Furthermore, our involvement in several EU projects and the interaction with domain experts has strengthened the above observation. Namely, in fleet management use-cases (cf. project Track&Know1), fuel consumption can be estimated more accurately if weather information is available (mostly rain-related information). In the maritime domain (cf. projects BigDataStack2 and datAcron3), weather typically afects the trajectory followed by a vessel. Last, but not least, in air-trafic management (cf. project datAcron), storm-related information may afect not only the route of an aircraft, but can also result in regulations for lfights and eventually delays, which could probably be predicted. Therefore, a common requirement across all these domains is to have available weather information together with the positions of moving objects.
Unfortunately, despite the significance of integrating mobility data with weather, there is a lack of such publicly available systems or tools that are easy to use. Motivated by this limitation, in this paper, we present the design and implementation of weather integration system, which has several salient features: (a) it works as a standalone and re-usable tool for data integration of mobility data with weather, (b) it is eficient in terms of processing performance, thus making it suitable for application in online scenarios (stream processing), (c) it supports enrichment of complex geometries (e.g., polylines, polygons) with weather data, which is not straightforward.
In summary, we make the following contributions: • We present a generic system for integrating mobility data represented by spatio-temporal positions with weather information, focusing on ease of use and eficient processing. • We show how to extend the basic mechanism to perform weather integration for more complex geometries, such as large 3D sectors, which is not straightforward. • We demonstrate the eficiency of our system by means of empirical evaluation on real-life mobility data sets from diferent domains (urban, maritime and air-trafic).
The remainder of this paper is structured as follows. Section 3 describes how weather data is made available, its format, and internal structure. Section 4 presents the system architecture
of the weather integration service. Section 5 provides various extensions of the basic functionality, thus improving the usability of the system in diferent application scenarios. The experimental evaluation is provided in Section 6, and we conclude the paper in Section 7. 2
The significance of integrating mobility data, in the form of AIS
messages, with weather data has been identified as a major
challenge for enabling advanced analytics in the maritime domain [
In the context of linking mobility data to external data sources,
in order to produce semantic trajectories, one external source
that has been considered is weather. For example, FAIMUSS [
Only few works study the concept of weather data integration,
focusing on real-time applications. Gooch and Chandrasekar [
Kolokoloc [
GRIB (Gridded Binary) format 4 is a standard file format for storing and transporting gridded meteorological data in binary form. The GRIB standard was designed to be self-describing, compact and portable. It is maintained by the World Meteorological Organization (WMO). All of the National Met Services (NMS) use this kind of standardization in order to store and exchange forecast data. GRIB files are provided by National Oceanic and Atmospheric Administration (NOAA), containing data from numerical weather prediction models which are computer-generated.
NOAA ofers several data sets composed of GRIB files, based on one of the provided model data. Four categories of model data are available5; Reanalysis, Numerical Weather Prediction, Climate Prediction and Ocean Models. Model data are represented on a grid (two-dimensional space), divided into cells where each one maps a specific geographical area. Data is associated with every grid cell; weather information is provided for every geographical place being included in the grid. Model data contain also the temporal dimension inasmuch the weather conditions do not remain static accross the globe. In other words, model data are gridded data with spatio-temporal information. The ofered data sets can be considered as three-dimensional cubes with weather data over a time period. In some cases, a fourth dimension is 4http://weather.mailasail.com/Franks-Weather/Grib-Files-Getting-And-Using 5https://www.ncdc.noaa.gov/data-access/model-data/model-datasets included, namely the altitude, for weather information that does not refer to the surface of the earth.
In this work, we use GRIB files based on the Global Forecast System (GFS) which is a type of Numerical Weather Prediction 6 (NWP) data model. NWP is a widely used model for weather forecasting generally, exploiting the current state of weather for making predictions. Current observations are (computer) processed, as they served as an input to mathematical models. Many attributes of the future weather state are produced as an output, such as temperature, humidity, precipitation, etc.
The GFS model7 is composed of four distinct models; the atmosphere model, the ocean model, the land/soil model and the sea ice model. All of these models provide a complete image about the weather conditions around the globe. GFS is produced by the National Centers for Environmental Prediction (NCEP). The data set product type we use in this work is the GFS Forecasts. Also, two other GFS product types exist, the GFS Analysis and the Legacy NCEP NOAAPort GFS-AVN/GFS-MRF Analysis and Forecasts. The products come with some data sets that diferentiate to the grid scale or the covering time period.
In this work, we use the data set of GFS Forecasts product that has the globe partitioned per 0.5◦ degrees on the geographic coordinates (longitude and latitude); also, another GFS Forecasts product exist that has the globe partitioned with 1◦ degrees. Every day the mathematical forecast model is run four times and has one of the following time references; 00:00, 06:00, 12:00 or 18:00. The time reference is the time that the forecast starts. Each of the forecast models cover the weather conditions around the globe for 384 hours (16 days) after its starting time. Specifically, a forecast model covers the weather conditions for 93 diferent timings, called steps. Every step is a distinct GRIB file, containing numerous weather attributes that are instantaneous and aggregates (averages).
The steps start from 000 to 240 (increased by 3) and continue to 252 until 384 (increased by 12). The step number indicates that the weather information contained in the GRIB file refer to the timing of X-hours after the forecast starting time. For example, the 000 step contains only instantaneous variables, referring to the forecast starting time (as the first step, it does not contain aggregate variables). The 003 step contains both instantaneous and aggregate variables. The instantaneous variables refer to the timing of weather attributes after 3-hours from the forecast reference time. The aggregate variables contain the averages of the 3-hours that passed. The same applies to the 006 step, containing both instantaneous and aggregate variables. The instantaneous variables refer to the timing of weather attributes after 6-hours from the forecast reference time. The aggregate variables contain the averages of the 6-hours that passed. The same pattern does not continue for the aggregate variables on the next steps. For instance, the 009 step contain aggregates variables that refer to the averages of weather attributes of the last 3-hours and step 012 contain aggregate variables that refer to the averages of weather attributes of the last 6-hours (the pattern is repeated until the 240 step). The aggregate variables of the steps greater than 240 [252...384] are the averages of weather attributes of the last 12-hours.
In our work, we use the four forecast models of a day with step 003 from the GFS Forecasts product; therefore, every day 6https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/numericalweather-prediction 7https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/globalforcast-system-gfs objectId lon lat 465_18 22.61 40.56
date 2017/04/30 19:03:06 Spatio-Temporal Parser ...
...
Trajectory point objectId lon lat date ... 465_1o8bje2c2tI.d61 l4o0n.56 lat2200111977:0//003:440//633d00ate ... ... 444666555___111666688822224444222____8222888...4444o565888b808666j222e2222___c2222444444t....000I6655111d...01885554532224444l333o0000...n777....5555228895643001111199933377:::000887l//a...005442222009t:::440000111109253//11119999252337777::::0000////00004345222::::45443052000111////26523033111000d7770100:::...554a///...000...035te:::555201///469000111............
862_41 23.79 38.02 201107:5/09:51/001 consists of 4 GRIB files that cover the instantaneous variables of weather attributes at specific timings of a day; 3:00, 9:00, 15:00 and 21:00. Since we use the 003 step, we have access to the 3-hour aggregates variables that cover the time following time periods of a day; 00:00-3:00, 06:00-9:00, 12:00-15:00 and 18:00-21:00. 4
The proposed system operates at the level of a single record, corresponding to a spatio-temporal position, and processes records independently of each other. The spatio-temporal position can be in the 2D space (x, y, t ) of 3D space (x, y, z, t ). At an abstract level, the system uses an external source storing weather data, in order to extract the desired weather attributes w1, w2, . . . , wn that are associated with the specific position. In the case of 2D data, its output is an extended record that consists of the fields: x, y, t , w1, w2, . . . , wn . Obviously, any other additional fields of the input record are maintained also in the output record. In the following, we present our techniques for implementing this data integration process in an eficient way. 4.1
The architecture of the system consists of two parts-mechanisms whose functionality is combined for the data integration service provision. The first part is called Spatio-Temporal Parser and the second part is the Weather Data Obtainer. The overall architecture is illustrated in Figure 1.
The Spatio-Temporal Parser parses sequentially the records of the input data set of mobility data. For each record, a set of basic data cleaning operations are performed. For instance, the spatiotemporal part is checked both for its existence (null or empty values) and its validity (valid longitude and latitude values). If the spatial or temporal information of a record is out of the valid range or missing, the parser ignores the whole record, writes information in an error log, and the parsing procedure continues by accessing the next record. Each record with valid spatial and temporal information is passed to the Weather Data Obtainer Mechanism, which is responsible of getting the values of the weather attributes from the weather data source that contains weather information (GRIB files). Then, the obtained values are concatenated with the current processed record, forming thus an enriched record containing values of weather attributes. Subsequently, the resulting record is written to a new file in the hard disk; the whole procedure generates a new (enriched) data set. The logical separation of parsing from the remaining functionality is useful, since the system can be easily extended to read data from other data formats and sources, such as XML, JSON, or a database.
The Weather Data Obtainer is the mechanism that finds the values of weather attributes given a longitude, a latitude and a date value. In case of 4D mobility data, it also uses the altitude as input. The functionality of the Obtainer is based on GRIB files since its role is to obtain from them weather information for a specific timespan. After the Obtainer has received the spatial and temporal information, its first step is to determine the right GRIB file that should be accessed in order to get the values of the weather attributes; each of the GRIB files contains weather data only for a specific time period. As a result, the covering timespan of the chosen GRIB file should be the closest to the given timestamp of the spatio-temporal position.
The procedure of matching the given timestamp with one of the GRIB files, is achieved by maintaining a Red-Black Tree data structure in-memory, organizing the references (paths) of each GRIB file from a given set. The tree’s node arrangement (key) is determined by the covering time of each GRIB file. Given a timestamp such as 4/12/2016 05:10:00, the tree finds two GRIB Files - f1 and f2 that cover earlier and later time respectively; in our example, these are 4/12/2016 00:00:00 (f1) and 4/12/2016 06:00:00 (f2). Due to the fact that the given timestamp is closer to 4/12/2016 06:00:00, the f2 file is chosen for opening. The formation of the Tree Data Structure is considered as a pre-processing step, prior to processing mobility data.
After a specific GRIB file is selected, it must be opened in order to retrieve the weather attributes associated with the spatial part of the spatio-temporal position. There are two options of accessing the values of weather attributes of a GRIB file. The first is by loading and keeping in memory the weather attribute(s) of interest, while the second is by retrieving the weather attribute(s) from disk. The purpose of loading and keeping in memory is to perform eficiently repeated read operations, but there is a natural trade-of in terms of speed and main memory consumption. In our case, the parameters required to identify the value of a weather attributes are the spatial values (longitude and latitude) of the record at hand. These are used for determining the cell (region that results from grid partitioning) in order to get the values of the weather attributes. 4.2
As GRIB files are binary files, which are accessed by an API, there exists an overhead related to opening a lfie. In case of multiple read operations, this cost can easily dominate the total processing time, especially if many files need to be opened by input records.
To avoid this overhead, we introduce a simple caching mechanism, which practically maintains in memory references to open ifles, in order to avoid repeated open/close operations. In the general case, the caching mechanism is crucial for the performance of the data integration procedure because a GRIB file remains open and can serve many sequential requests. This relieves the Weather Data Obtainer from the task of re-loading the GRIB file in-memory for every record, thus saving significant time. The cache replacement policy adopted is simple LRU.
It should be mentioned that in case of sorted access to mobility data based on the temporal part of the spatio-temporal position, the value of the caching mechanism is negligible. However, there exist cases where the underlying mobility data is not strictly sorted by time. This typically occurs in real-life surveillance data acquisition, where some records corresponding to positions of moving objects may be delayed. In such cases, the caching mechanism can improve performance significantly. 5
In this section, we describe two extensions of the basic system for weather data integration. The first extension concerns integration of weather information with complex geometries, such as 3D sectors and airblocks, which are prevalent in air-trafic management (ATM) applications. The second extension is about providing output in RDF format, thereby generating linked data. 5.1
A useful extension of the proposed system is towards more complex (compared to a single point) geometries. In many cases we may need to associate the trajectory of a moving object (i.e. a LineString geometry) with weather conditions, or a region or a cluster of regions on the surface of the earth (i.e. polygon or multi-polygon geometries).
The first extension of the proposed system is towards
computing the average of values of a selected weather attribute, over a
(potentially 3D) geometry. Specifically, given a geometry д and
a time interval [ts , te ] (not necessarily a proper interval, i.e., it
may hold that ts = te ), where ts is the time instant denoting the
starting time of the interval and te is the time instant that the
interval ends, this extension returns the average of all the values
retrieved for all the points of the geometry. Since the GRIB file
that we use has resolution of 0.5 degrees, we reduce the geometry
to 0.5 degree precision. This will reduce the number of points
and the number of requests to the GRIB file. The same geometry
simplification is applied for altitude of the 3D geometry, i.e., the
z-axis values are reduced to the isobaric levels used in the GRIB
ifle (and for weather attributes that depend on altitude). The
core function used for retrieving the values of selected weather
attributes for a given spatio-temporal position is used for each
point of the geometry, and the average of these values is returned
as result.
The proposed system has also been extended to operate on RDF
data, both as a consumer and as a server. Since RDF is the W3C
standard to be used for Linked Open Data, connecting as a
consumer to RDF triples, it can exploit any positioning data available
on the web, to return it enriched with weather data. Furthermore,
serving RDF positioning data enriched with weather data under a
common schema, can support several tasks from event or pattern
recognition to link discovery between multiple data sets. Exactly
this functionality of our system to generate linked mobility data
with weather has been exploited in the context of the datAcron
project [
When acting as an RDF consumer and given the schema of the data source, the proposed system executes a SPARQL query to retrieve the necessary positioning and temporal values for each point. If the data source provides complex geometries, these can also be exploited, as discussed previously. For each record retrieved by the SPARQL query, the core function retrieves the corresponding values of the selected weather attributes and enriches the record.
When the proposed system serves the enriched data as RDF
triples, it only requires a triple template to be used on an
RDFGen [
Obviously, the proposed system can be used in the same time as consumer and as a server, enriching RDF data with weather attributes and values. Since RDF-Gen can provide consistent triples to any given schema, this extension can enhance any positioning and weather related ontology. 6
In this section, we provide the results of the empirical evaluation performed using real-life data from the urban domain, provided by a fleet management data provider. All experiments were conducted on a computer equipped with 3.6GHZ Intel core i7-4790 processor, 16GB DDR3 1600MHz RAM, 1TB hard disk drive and Ubuntu 18.04.1 LTS operating system. Our code is developed in Java and is available at the following link: https: //github.com/nkoutroumanis/Weather-Integrator. For the access to GRIB files, we use the NetCDF-Java Library 8. 6.1
Data sets. The mobility data set used in this work for the application of the data integration procedure is in the form of CSV files, containing real trajectories of vehicles in the region of Greece. 8https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/ documentation.htm
Figure 4 provides an illustration of the data distribution on the geographical map. Each record constitutes a spatio-temporal point with additional information of the vehicle, such as speed, fuel level, fuel consumed, angle, etc. The records of the CSV files are provided in temporal sort order. This resembles real-life operation, since positions of moving objects are transmitted by devices located on vehicles, even though they are not received in strict temporal order, but with small discrepancies.
Due to the fact that the size of the complete data set is about 130GB and spans one year (July 2017 - June 2018), we take a small sample (consisting of 9 CSV files) whose temporal part is in the time period of January 2018 for performing the first set of experiments. Each file is approximately 550KB and contains about 4,500 records. In addition, we use a larger sample that consists of 10GB of data, having 81,483,834 records in total.
For the weather data, we downloaded 124 GRIB files (31 x 4) corresponding to January 2018. These files were used for obtaining weather data information in the data integration process. We use only the 003 steps of the 4 forecast models per day. The total size of the GRIB files is 8GB.
The resultant (enriched) data set contains the following 13 weather attributes that describe promptly the rain-related weather conditions: • Per cent frozen precipitation surface • Precipitable water entire atmosphere single layer • Precipitation rate surface 3 Hour Average • Storm relative helicity height above ground layer • Total precipitation surface 3 Hour Accumulation • Categorical Rain surface 3 Hour Average • Categorical Freezing Rain surface 3 Hour Average • Categorical Ice Pellets surface 3 Hour Average • Categorical Snow surface 3 Hour Average • Convective Precipitation Rate surface 3 Hour Average • Convective precipitation surface 3 Hour Accumulation • U-Component Storm Motion height above ground layer • V-Component Storm Motion height above ground layer Metrics. Our primary target is to make the integration procedure eficient. For this purpose, we use the following metrics that reflect the mechanism performance: • Execution time: The total required time for the completion of the integration procedure (in minutes). • Throughput: The number of processed records per second (rows/sec). • Cache hit ratio (CHR): The ratio number of cache hits to the total number of records. In other words, this number is the percentage of records that have been enriched with weather information without requiring the corresponding GRIB file to be loaded in-memory. The higher the CHR value, the larger the benefit in execution time.
Methodology. The experimental evaluation is structured as follows. First, we evaluate the performance of our system, in terms execution time and throughput. Thus, we use two samples of the complete data set of diferent size: • Small data set: 5.1MB of data corresponding to the trajectories of few vehicles in January 2018. • Large data set: 10GB of data covering the time span of the complete data set.
The size of the integrated data set with weather is 8.6MB and 16.6GB respectively for the two sample data sets above.
Second, we evaluate the performance of the caching mechanism. Since data is provided sorted in time, the caching mechanism is of little use. Therefore, we randomly shufle the input records, thus making a worst-case scenario where the input data are processed in random order. In this case, two consequent records will access diferent GRIB files with high probability.
Integration
With Indexing Without Indexing
Preprocessing
Table 1 demonstrates some elements about the data integration procedure for the case of the small data set. The first row in the table refers to keeping in-memory the retrieved weather values, whereas the second row corresponds to retrieval from disk. Clearly, the former is the most eficient way to perform the integration task, achieving throughput of 3,570 rows/sec. Instead, the latter approach only processes 26 rows/sec. This gain comes with an overhead in memory consumption, which is almost doubled, but is still manageable. Notice that the input data set is provided sorted in time, therefore the observed throughput of 3570 rows/sec is the best performance that can be achieved on the given hardware. Regarding the pre-processing overhead, namely the construction of the Red Black Tree that indexes the GRIB files, this is in general negligible (see the third row of the table).
Table 2 shows the results when the large data set (10GB) is used. We only employ our approach with in-memory maintenance of the retrieved weather values. Again, the throughput is quite high (2,784 rows/sec), thus showing that our performance results also hold in the case of large data sets. 30 5 140
20 and eficiency. The proposed mode of operation is record-byrecord, which is an abstraction that ofers significant benefits, including fairly easy parallelization. Furthermore, we show that the proposed system is extensible, demonstrating its use to enrich complex 3D geometries with weather (instead of simple points) and the generation of linked data in RDF. Our experiments on real-life data sets show the eficiency of our system.
In our future work, we intend to study the gain in performance that can be attained by means of parallel processing, using a Big data framework, such as Apache Flink of Spark Streaming. Moreover, we will focus on diferent use-cases where our system can be applied, e.g., batch processing to enrich a vast database of historical trajectories with weather information. Also, we will explore in much more detail the issue of enriching complex geometries with weather information, which is not straightforward even for domain experts, especially for large-sized objects that cover large parts of the space (trajectories, sectors, etc.).
This work is supported by projects datAcron, Track&Know, BigDataStack, and MASTER (Marie Sklowdoska-Curie), which have received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 687591, No 780754, No 779747 and No 777695 respectively. is seldom encountered in practice. Finally, Figure 8 shows that higher cache sized also result in higher memory consumption.
In summary, the caching mechanism can be very useful in the case of input data that is not temporally sorted, since it improves performance significantly, at the expense of higher memory consumption. 7
In this paper, we presented a system for integrating mobility data with weather information, which focuses on ease of use