This paper demonstrates some of the latest developments for the visual analysis of large tracking datasets within the Python ecosystem revolving around the Pandas and HoloViz libraries. These software tools enable the quick and interactive notebookbased exploration of large datasets with millions of location records on commodity hardware.
Movement datasets collected by systems tracking vehicles,
people, goods or wildlife have the potential to improve our
understanding of complex mobility systems [
Visualizations of movement data often sufer from visual
clutter [
Recent years have seen an increase in movement data analysis
functionality in Python, including libraries such as
MovingPandas [
In this paper, we demonstrate cutting-edge tools for the
interactive exploration of large tracking datasets in Python notebooks
building on the data handling capabilities of Pandas[
Panel objects from Pandas, Xarray, or other data structures
• HoloViews[
The key tool for the visualization of large datasets in the following examples is HoloViz Datashader. Computation-intensive steps in the Datashader process are written in Python that is compiled to machine code using Numba5 to increase performance. Furthermore, Datashader distributes the computations across CPU cores and processors (using Dask6) or GPUs (using CUDA7). Conventional visualization packages for interactive plotting used 1https://blog.rstudio.com/2016/10/05/r-notebooks/ 2http://holoviz.org 3https://holoviz.org/background.html 4https://bokeh.org 5https://numba.pydata.org 6https://dask.org 7https://developer.nvidia.com/cuda-zone in notebook environments (for example, Folium8 used by scikitmobility) pass all data records directly to the browser where they are then displayed. This enables interaction with each data point but it quickly runs into limitations on how much data can be visualized. Datashader instead generates a fixed-size (rasterized) data structure (regardless of the original number of records) and transfers that to the browser for display.
In the following section we illustrate the capabilities of the HoloViz family of tools for the purposes of movement data exploration with a focus on the moving object trajectories using a dataset of vessel tracking data (AIS) published by the Danish Maritime Authority with more than 12 million location records. Afterwards, we discuss current limitations that movement data analysts should be aware of and have an outlook at what might come next.
The analysis examples covered in this demo are primarily moverfocused. The first examples show point-based visualisations of individual location records, as well as segment-based visualizations of consecutive records in their geographic context. Later examples show non-spatial visualizations that can provide further insight into other important characteristics of tracking datasets, such as sampling intervals and mover identification. The data requirements for these examples are minimal: columns for mover ID (MMSI in our AIS data example), timestamp (Timestamp), location (Longitude and Latitude), and optionally speed (SOG which is short for ‘speed over ground’ used in the histogram in Figure 1).
Classic scatter plots, as shown in Figure 1, are popular tools (particularly for early steps in the movement data exploration process) to assess the spatial extent of the dataset, find outliers, and identify gaps in the data coverage. This commonly reoccurring task should therefore be straightforward to support data analysts in their work. Indeed, as Listing 1 shows, creating a scatterplot from a Pandas DataFrame and linking it to another plot (a histogram of speed values in this case) requires only a few lines of code9. To plot the locations on top of background map tiles, the original latitude and longitude values need to be reprojected to a metric coordinate system (Pseudo-Mercator, EPSG:3857). For this purpose, Datashader provides the convenience function lnglat_to_meters.
The linked functionality is automatically created by calling link_selections. The linking is based on the common IDs of records in the DataFrame df that is visualized in both plots. Another noteworthy feature is the simple way of arranging plots. For example, arranging plots side-by-side in one notebook cell output is achieved by connecting them using a simple plus sign (+). To add geographic context, background map tiles can be added as shown in the following example in Listing 2. The diferent layers of the map plot are combined using simple multiplication operators (*).
Figure 1 also illustrates how the user can apply spatial filters (by drawing a bounding box in the map view) and attribute-based iflters (by selecting certain bin ranges in the histogram). This way, location records of vessels travelling faster than 10 knots are highlighted in the map. 8https://python-visualization.github.io/folium/ 9Linked brushing demo notebook at https://github.com/anitagraser/movingpandasexamples/blob/bmda2021/tech-demos/linked-brushing.ipynb Listing 1: Source code for the linked scatter plot and histogram in Figure 1 import p a n d a s a s pd import h v p l o t . p a n d a s from d a t a s h a d e r . u t i l s import l n g l a t _ t o _ m e t e r s from h o l o v i e w s . s e l e c t i o n import l i n k _ s e l e c t i o n s
Segment-based visualizations (Figure 2-3) enable more advance
analysis tasks, such as the identification of gaps in trajectories or
discovery of large jumps and other common movement data
quality issues [
For more advanced segment-based analysis, for example, to filter segments based on their length (to detect gaps and jumps), as shown in Figure 3, additional computing steps need to be added. This step (using a custom compute_segment_info function in the notebook11) is currently considerably more computationally expensive than the plotting steps because of the necessary pair-wise operations such as distance computations between consecutive records of the same mover.
As the previously presented point and segment density map
examples show, Datashader is eficient at visualizing large sets
of points or paths in two-dimensional plots by rasterizing the
inputs. However, Datashader is of course not limited to plots
in geographic space. For example, Figure 4 shows a coordinate
change plot which visualizes the location change between
consecutive location records. These coordinate change plots can
help identify issues related to systematic changes in sampling
intervals [
A convenient hvplot tool for exploring the relationships between trajectory properties are scatter matrixes (a combination of scatter plots and histograms). For example, a scatter matrix of trajectory duration versus start time, as shown in Figure 5, can help identify issues related to unstable object IDs. These issues can be caused by reassignment of IDs at certain points in time (often at midnight) or in certain intervals which lead to visible start time or duration clusters, respectively. These scatter matrixes can be used to visualized the relationship between two or more DataFrame columns. The scatter matrix in Figure 5 is not datashaded, however, since trajectories are aggregates of the raw tracking data, the resulting DataFrame of trajectory properties (traj_df ) is much smaller than the original DataFrame of individual location records (df ) and can therefore be plotted using conventional means, that is, using hvplot without Datashader. 3
Notebook-based visualizations continue to advance and many
of the limitations listed in prior work (such as a lack of linked
views [
Using regular Pandas DataFrame, the size of the dataset is limited by the available memory since the whole DataFrame has to fit into memory. This may be addressed by adopting Dask
DataFrames which can break down larger datasets into manageable chunks.
While Bokeh plots can usually be exported (that is saved as images), this is currently not possible for map plot, as indicated by a lack of a save icon in the Bokeh tool bar in Figures 2/3/6. This makes the process of generating plots for publications less convenient.
While linked plots allow for intuitive data exploration, their
interactive nature makes it hard to keep track of the changes
and to reproduce results [
Further research will look into the possibility of creating notebook-based animations of movement data, similar to what is possible in desktop GIS such as QGIS with TimeManager extension. This may be achieved using hvPlot’s support for streaming DataFrames or other similar tools that still need to be evaluated.