Accurate forecasting in many industries, such as dairy production, is a key challenge to improve investment decisions, market planning, energy use, and environmental protection. As most real-world phenomena take place over time, many prediction tasks involve the use of time series or collections thereof. In this study, we propose a nearestneighbour data augmentation approach which draws on information from a set of related time series. Given an input series, we identify other series which have similar pro les using a k-nearest neighbour approach, and use their data to augment the input series. This has the e ect of both smoothing outlying observations and lling missing observations. We show that, by combining this method with a criterion to re ne the cases for which augmentation is to be applied, overall forecasting accuracy can be improved. Our approach is evaluated on diverse real-world time series data from a number of di erent domains.
Time series data has attracted a signi cant research e ort in recent years,
perhaps because of its centrality in accurate forecasting in a variety industry
settings, from nancial services to smart agriculture. Many di erent approaches to
time series prediction have been proposed, along with di erent methods for data
pre-processing and augmentation [
One technique to enhance the performance of time series forecasting is the
application of a k-nearest neighbours (k-NN) approach, which is typically
applied to reduce noise and the impact of outliers on training sets. The use of k-NN
can additionally improve training set quality when the volume of observations
Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
is limited. Recent research has, to the best of our knowledge, used perturbed
examples of existing observations to generate neighbours for the application of
this approach, as opposed to using real-world occurrences [
To harness this additional information provided for a collection of entities,
we propose using an approach motivated by k-NN regression, which attempts
to make a prediction for a continuous target by averaging cases in the same
neighbourhood. In our case, we generate a forecast for a given input series by
incorporating information coming from an aggregated set of forecasts generated
for neighbouring entities. Speci cally, we apply this approach using the popular
Prophet forecasting algorithm [
Our analysis considers the importance of both the neighbourhood size k and the volume of available data. Our ndings presented in Section 4 indicate that not all data is suited to this augmentation approach. Therefore, we also provide a means to pro le entities, using a validation window, to ensure that augmentation is applied to instances more suited its application. Using a combination of k-NN regression and this ltering criterion, we observe improved accuracy across all datasets, when compared to baseline forecasts produced by Prophet alone. 2
The standard de nition of time series forecasting assumes the existence of a set
of time-ordered observations of a variable Y , denoted y1; y2; : : : yt, where yi is
the value of Y measured at time i, and de nes the predictive task as trying to
forecast the future values of this variable for time stamps s > t. Many variants of
this general task exist, including the use of other measured variables as potential
predictors of the future values of the target series Y . Still, the general assumption
is that there is an unknown function that \maps" past observations to future
values of Y . The learning goal is to approximate a function using some prediction
error criterion and a historical record of observed values. The predictors used
for forecasting the future values of Y are usually the most recent observations of
Y , as the basic assumption of time series forecasting is that of the existence of
some form of correlation between successive observations of the series. This is the
methodology employed in most approaches to time series forecasting, including
the well-known ARIMA models [
High-dimensional multivariate data can sometimes produce superior
forecasting results to those achieved using linear time series. For instance, in the
context of single-farm prediction of milk production, the most successful models
are the surface tting model and the NARX (Nonlinear autoregressive model
with exogenous inputs (RMSE 75.5kg, 365 day horizon)) [
One popular time series forecasting procedure is Prophet [
The k-nearest neighbour algorithm is a well-known non-parametric method
used for classi cation and regression. It has been labelled as a lazy learner as
the algorithm does not learn a discriminative function, but rather stores the
instances for later use. Given some features or explanatory variables of a new
instance to be regressed on, k-NN nds the k training instances that are closest
to the new instance according to some distance metric.In the context of time
series data, k-NN has been employed in a range of disparate ways. For instance,
its use can be witnessed in local prediction: by breaking down domains into
local neighbourhoods and tting to each neighbourhood separately. This type
of methodology can be found in bootstrapping as far back as Jayawardena and
Lai [
A variety of techniques have been proposed for manipulating time series
representations to improve the outputs of downstream tasks. While data
augmentation has largely been used to generate additional training data in the context
of image datasets, Oh et al. [
Selection criteria. Early experimentation indicated that data augmentation was not universally appropriate for every entity in a dataset. Consequently, we have developed an approach for ltering out entities that are less suited to augmentation. These might correspond to entities for which forecasting accuracy is already high, or outliers for which relevant neighbouring cases are unavailable. We can consider the time-series to be composed of distinct windows, with the term `window' referring to a slice of the time series (which in the given datasets corresponds with a single year). Using a single window of the data as a validation set, we apply augmentation for a range of values of k, followed by the forecasting model in each case. We can then determine whether applying k-NN resulted in any positive change in forecasting performance (e.g., as measured by mean absolute error (MAE)), and, if it did, what value of k produced the best result. If no value of k produced a positive change then the entity is ltered from the augmentation process. We refer to the entities for which augmentation is applied as \contested", and those for which it is not applied as \uncontested".4 However, if the approach produced a positive result when using only one value of k, an 4 Given that we are interested in the e ectiveness of both the ltering technique, and data augmentation, we consequently adopt this terminology to add clarity and distinguish between the original results that form our baselines, which are entirely unaugmented, and the test results which consist of both augmented and unaugmented cases. Select and clean
data Calculation of k nearest neighbours
Produce augmented time seires using each value of n
Did augmentation improve validation set forecast? NO
YES
Overal evaluation Filter from augmentation process Would augmentation haveimproved subsequent forecasts? missing value apply interpolation viaclosest neighbour
Apply augmentation using most successful value
of n Did forecast accuracy improve? NO
YES
NO 'uc_loss' result 'uc_won' result 'loss' result
YES 'won' result additional check was performed to see if the di erence in MAE for this entity's validation window was above average (across all entities for the validation window). If this happened to be the case, then such entities were also discarded for use with the proposed approach. A comparison between entities which were and were not selected using this basis will be provided in Section 4. 3.2
We now discuss ve di erent datasets from domains where collections of related entities, represented as time series, naturally occur. We chose a diverse range of domains in order to prove the applicability of our approach. With the exception of the socioeconomic dataset, each of these domains is represented as univariate time series (while the lattermost, for the purposes of this research, is split into two separate univariate datasets).
Agricultural data. Our rst dataset concerns milk production from 400 farms
in Ireland for one dairy processor, as measured across four consecutive years. This
will be referred to as the `dairy' dataset in our experiments. Cases describe daily
readings for the volume of milk collected on each farm. Each farm constitutes a
separate entity and the predicted variable is the volume of milk collected from
each entity. In any given year, milk supply forecasting is a fundamental driver for
the dairy sector. Dairy processors use their forecasts to establish pricing levels,
contracts with supplier farms, and the production requirements for their
factories. Forecasting consequently strongly in uences farm management (the
prevention of under or over production, and also the manner of production adopted),
the consumption of resources in the sector (e.g. fertiliser use, tanker-transport
use for milk collections), and the processing e ciency of factories (avoiding waste
from surplus milk supplies). Processors can drive sustainability changes through
accurate and precise forecasting. However, such forecasting is highly challenging
for many reasons: the forecast must be made over 1000s of farms which di er in
their herd-pro les; the land farmed and farm-management practices is for the
full year in advance, not incrementally as the year unfolds; and planning can
encounter disruption from climate-change and disease outbreaks.
Meteorological data. The second dataset used here, involves the monthly
average temperatures, recorded in degrees Fahrenheit, of 18 major cities from
regions spread across the world, dating from the years 2000 and 2013. This
corresponds to a subset of the Berkeley Earth climate change dataset [
Energy data. The fth and nal dataset considered relates to the Natural Gas
Gross Withdrawals and Production, measured in monthly million cubic feet per
day, as compiled by the U.S. Energy Information Administration (EIA) [
Measures. Using the datasets described earlier we evaluated our proposed method using several measures. The principle measure used was the mean absolute error (MAE), in which a lower gure indicates a better performance. The baseline method in the experiments is the non-augmented Prophet forecast model applied to a single query time series.
Setup. For the purposes of our experiments, the rst three windows in each dataset were used as training data, the next window as a validation set, and the subsequent windows as test data (the latter of which is used for Tables 3-4). For each window of the test data, we have a predicted and actual value. Forecasts were generated for each entity to provide a base rate and then further forecasts generated using training data composed as as an arithmetic mean for a range of neighbours. The base rate in these evaluations was the non-augmented predicted value provided by Prophet. The window of observations and predictions was then advanced by one period, with the length of a period being equal to a calendar year. The process was then repeated again ab initio, until the methodology was run over the full length of the respective dataset.
A potential limitation when using Euclidean distance as a metric to measure time series similarity is that when large absolute distances exist between di erent time series, it becomes harder to identify time series with similar trends and seasonality. In order to deal with this issue, we applied a normalisation in which the time series being compared were scaled, in proportion to the ratio of means. Missing data can also be an issue for forecasting in real-world datasets. While a potential solution to this can be to ll a missing value with the preceding day's value, in our method we instead replace the missing value with a mean of the values of k-nearest entities (i.e., interpolation).
Prophet has no native means to handle multiple entities' time-series-data for tting an individual entity's time series forecast, so our proposed approach is consequently applied prior to the provision of data to the model. Prophet allows the use of a custom list of holidays and seasonalities in the model. However, the use of holidays as a parameter is only appropriate for time series whose values are discredited to days as opposed to months or years. As only one of our datasets contained daily data (the agricultural dataset), and the importance of holidays in this context was unclear, we do not utilise this functionality in our forecasts. 4.2
Number of neighbours. In our experiments, we consider augmentation using k 2 [1; 5] neighbours. As outlined in Section 3.1, the validation window was used to establish whether or not the baseline forecasts would be \contested" using the data augmented via the proposed k-NN approach (i.e., whether or not additional neighbours should be used). However, the validation window is also used to establish what value of k should be used to this end. A frequency count of the value of k which produced the best result within the validation window was performed for all entities which passed the selection criterion. The results of this for each dataset can be viewed in Table 2. When the number of entities to provide this summation of k values was at least double the range of k values available (i.e., a clear majority), a global value of k was used for all further forecasts for that dataset (otherwise a local value of k was used).
Table 3 shows the di erence in performance between these forecasts when the uncontested baseline forecasts are used, versus when the selection criteria determines that the augmentation approach should be used. In the table, uc lost and uc won refer to the uncontested cases which have not been augmented, and whether or not the proposed approach would have produced superior or inferior
Dataset results had it been used. These results relate to the test data of all chosen entities, where the maximum value is the number of windows multiplied by the number of entities for a given dataset. Where the augmentation approach is adopted we can also see, for a given forecast, whether or not the change has resulted in an increase or decrease in accuracy (i.e., won and lost respectively).
Table 3 reveals considerable di erences between datasets. For instance, the dairy dataset would have seen overall more forecasts improve than deteriorate through k-NN without augmentation applied to its data, but the weather dataset would have seen two thirds of its non-augmented forecasts obtain worse results than that of the baseline. Overall, the proposed selection criteria appears to have achieved a reliable basis for excluding many of the forecasts less suited to an augmentation approach. Indeed, only one dataset (employment adj.) shows more cases chosen by the selection criteria su ering from reduced accuracy than the number of those bene ting.
Moving beyond a simple count of wins and losses to the overall magnitude of these changes (Table 4), again we observe that uncontested forecasts, which fail the selection criteria, in general would have seen a performance decrease with the application of k-NN. Overall, we can see that there is not a single dataset in which the overall MAE declines for the selected forecasts. An overall summary of this information is given in Table 5. We report both the average MAE and the average reduction in error across entities that conform to the selection criteria.
If we look at a single sample entity in Fig. 2 that quali es for the application of the augmentation approach in the weather dataset (speci cally depicting Warsaw, with y-axis representing MAE value), we can see the approach's accuracy relative to both the baseline Prophet MAE and the hypothetical performance using di erent values of k. Using the validation window corresponding to 2001, all values of k showed improvement over the baseline Prophet forecast, and as the weather dataset applies a local rather than universal value for k on the test set, the value of k that performed best in this instance was chosen (k = 5). While the forecasting performance for all values of k broadly match the peaks and troughs of the baseline, signi cant improvement can be seen across the different windows. In this case the nearest neighbours which combine to provide the augmentation training data for MAE5 are, in ascending order: Wroclaw, Kiev, Kherson, Stockholm, and Uppsala.
Fig. 3 shows augmentation performance for a single farm from the dairy dataset where x axis represents time, and y axis represents volume of milk produced in litres. In the plot we can see how close the di erent augmentation approaches (y^1-y^5), as well as the baseline Prophet forecasting (y^) come to predicting the actual production values (y) for this window. As the dairy dataset utilised a global value of k, y^5 was the model chosen for all entities that were selected for augmentation. Notably, not all values of k in this instance achieve improved results over the baseline, with both y^1 and y^2 being generally inferior to the baseline. However, y^5 provides a good match for the actual production values of this farm. 5
Despite the strong performance of the baseline Prophet algorithm, our ndings indicate that potential improvements can be made with an aggregation approach to the training data using a k-NN methodology, in situations where we have multiple related entities represented as time series. Our results indicate that, while this has demonstrable bene t, it is by no means universally successful. Nevertheless, there is clear promise o ered by using di erent time series with similar pro les, in domains where forecasting is challenging due to noisy or missing data.
Although this work has established the potential use of this methodology, questions remain concerning the potential use of di erent distance metrics for the measurement of neighbours, and whether the use of additional validation windows could provide su cient data to adapt a more precise aggregation technique. While the various datasets explored provided a diverse ranges of entities, it was beyond the scope of this research to establish how many entities are required for this methodology to be e ective. Furthermore, we intend to extend this research by considering how k-NN in this context can ultimately aid in providing use explanations of forecasting outputs to end users.
Acknowledgements. This publication has emanated from research conducted with the nancial support of (i) SFI and the Department of Agriculture, Food and Marine on behalf of the Government of Ireland to the VistaMilk Research Centre under Grant Number 16/RC/3835, and (ii) SFI to the Insight Centre for Data Analytics under Grant Number 12/RC/2289 P2.