The ability of Artificial Neural Networks (ANNs) to learn accurate patterns from large amount of data has spurred interest of many researchers and industrialists alike. The promise of ANNs to automatically discover and extract useful features/patterns from data without dwelling on domain expertise although seems highly promising but comes at the cost of high reliance on large amount of accurately labeled data, which is often hard to acquire and formulate especially in time-series domains like anomaly detection, natural disaster management, predictive maintenance and healthcare. As these networks completely rely on data and ignore a very important modality i.e. expert, they are unable to harvest any benefit from the expert knowledge, which in many cases is very useful. In this paper, we try to bridge the gap between these data driven and expert knowledge based systems by introducing a novel framework for incorporating expert knowledge into the network (KINN). Integrating expert knowledge into the network has three key advantages: (a) Reduction in the amount of data needed to train the model, (b) provision of a lower bound on the performance of the resulting classifier by obtaining best of both worlds, and (c) improved convergence of model parameters (model converges in lesser number of epochs). Although experts are extremely good in solving different tasks, there are some trends and patterns, which are usually hidden only in the data. Therefore, KINN employs a novel residual knowledge incorporation scheme, which can automatically determine the quality of the predictions made by the expert and rectify it accordingly by learning the trends/patterns from data. Specifically, the method tries to use information contained in one modality to complement information missed by the other. We evaluated KINN on a real world traffic flow prediction problem. KINN significantly superseded performance of both the expert and as well as the base network (LSTM in this case) when evaluated in isolation, highlighting its superiority for the task.
Humans tend to learn from examples specific to the
problem, similar to DNNs, as well as from different sources
of knowledge and experiences
Therefore, a natural step forward is to combine both these
separate streams of knowledge i.e. knowledge extracted
from the data and the expert’s knowledge. As a matter of
fact, supplementing DNNs with expert knowledge and
predictions in order to improve their performance has been
actively researched upon. A way of sharing knowledge among
classes in the data has been considered in zero-shot-learning
The residual scheme employed in KINN to incorporate expert knowledge inside the network has three key advantages: (a) Significant reduction in the amount of data needed to train the model, since the network has to learn a residual function instead of learning the complete input to output space projection, (b) a lower bound on the performance of KINN based on the performance of the two subsequent classifiers achieving the best of both worlds, and (c) improvements in convergence of the model parameters as learning a residual mapping makes the optimization problem significantly easier to tackle. Moreover, since the DNN itself is data driven, this makes KINN robust enough to deal with situations where the predictions made by the expert model are not reliable or even useless.
The rest of the paper is structured as follows: We first provide a brief overview of the work done in the direction of expert knowledge incorporation in the past. We then explain the proposed framework, KINN, in detail. After that, we present the evaluation results regarding the different experiments performed in order to prove the efficacy of KINN for the task of expert knowledge incorporation. Finally, we conclude the paper with the conclusion.
Integrating domain knowledge and experts opinion into the
network is an active area of research and even dates back to
the early 90s. Knowledge-based Artificial Neural Networks
(KBANN) was proposed by
Expert knowledge is incorporated for key phrase
extraction by
Time-series forecasting is of vital significance due to its high
impact, specifically in domains like supply chain
L where W = fWl; blgl=1 encapsulates the parameters of the network and : Rp 7! R defines the map from the input space to the output space. The optimal parameters of the network W are computed based on the empirical risk computed over the training dataset. Using MSE as the loss function, the optimization problem can be stated as: ([xt 1; :::; xt p]; W))2 (1) where X denotes the set of training sequences and x 2 Rp+1. Solving this optimization problem comprising of thousands, if not millions of parameters, requires large amount of data in order to successfully constrain the parametric space so that a reliable solution is obtained.
Humans on the other hand, leverage their real-world knowledge along with their past-experiences in order to make predictions about the future. The aim of KINN is to inject this real-world knowledge in the form of expert into the system. However, as mentioned, information from the expert may not be reliable, therefore, KINN proposes a novel residual learning framework for the incorporation of expert knowledge into the system. The residual framework conditions the prediction of the network on the expert’s opinion. As a result, the network acts as a correcting entity for the values generated by the expert. This decouples our system from complete reliance on the expert knowledge. We evaluated KINN on Caltrans Performance Measurement System (PeMS) data. The data contains records of sensor readings that measure the flow of vehicular traffic on California Highways. Since the complete PeMS dataset is enormous in terms of its size comprising of records from multiple highways, we only considered a small fraction of it for our experiments i.e. the traffic flow on Richards Ave, from January 2016 till March 20161. The dataset contains information regarding the number of vehicles passing on the avenue every 30 seconds. PeMS also contains other details regarding the vehicles, however, we only consider the problem of average traffic flow forecasting in this paper. The data is grouped into 30 minute windows. The goal is to predict average number of vehicles per 30 seconds for the next 30 minutes. Fig. 1 provides an overview of the grouped dataset. The data clearly exhibits a seasonal component along with high variance for the peaks.
LSTMs have achieved state-of-the-art performance in a
range of different domains comprising of sequential data
such as language translation (Weiss et al. 2017), and
handwriting and speech recognition
Partial auto-correlation of the series was also analyzed to
identify association of the current value in the time-series
with its lagged version as shown in Fig. 2. As evident from
the figure, the series showed strong correlation with its past
1http://www.stat.ucdavis.edu/~clarkf/
three values. This is also cemented by the result of the
gridsearch that chose the window size of three. The final
network consisted of three hidden LSTM layers followed by
a dense regression layer. Apart from the first layer, which
used sigmoid, Rectified Linear Unit (ReLU)
Auto-Regressive Integrated Moving Average (ARIMA) is widely used by experts in time-series modelling and analysis. Therefore, we employed ARIMA as the expert opinion in our experiments. Since the data demonstrated a significant seasonal component, the seasonal variant of ARIMA (SARIMA) was used, whose parameters were estimated using the Box-Jenkins approach (Box et al. 2015). Fig. 4 demonstrates the predictions obtained by employing the LSTM model as well as the expert (SARIMA) model on the test set.
The overall predictions made by both the LSTM as well as the expert network seems plausible as shown in Fig. 4(a). However, it is only through thorough inspection and investigation on a narrower scale that strengths and weaknesses of each of the networks are unveiled as shown in Fig. 4(b). The LSTM tends to capture the overall trend of the data but suffered when predicting small variations in the time-series. SARIMA on the other hand was more accurate in predicting variations in the time-series. In terms of MSE, LSTM model performed considerably worse when compared to the expert model. For this dataset, the discovered LSTM model achieved a MSE of 5.90 compared to 1.24 achieved by SARIMA on the test set.
Most of the work in the literature
The KINN framework promotes both the expert model as well as the network to complement each other rather than directly mimicking the expert’s output. This allows KINN to successfully tackle cases where predictions from the expert are not reliable. Finding the best expert or neural network is not the focus here but instead, the focus is to incorporate expert prediction, may it be flawed, in such a way that the neural network maintains its strengths while incorporating strengths of the expert network.
There are many different ways through which knowledge
between an expert and the network can be integrated. Let
x^tp 2 R be the prediction made by the expert. We
incorporate the knowledge from the expert in a residual scheme
inspired by the idea of ResNet curated by
Incorporating expert knowledge in this residual fashion serves a very important purpose in our case. In cases where the expert’s predictions are inaccurate, the network can generate large offsets in order to compensate for the error, while the network can essentially output zero in cases where the expert’s predictions are extremely accurate. With this flexibility built into the system, the system can itself decide its reliance on the expert’s predictions.
We curated a range of different experiments each employing KINN in a unique scenario in order to evaluate its performance under varied conditions. We compare KINN results with the expert as well as the DNN in terms of performance to highlight the gains achieved by employing the residual learning scheme. To ensure a fair comparison, all of the preprocessing and LSTM hyperparameters were kept the same when the model was tested in isolation and when integrated as the residual function in KINN.
In the first setting, we tested and compared KINN’s performance in the normal case where the expert predictions are accurate and the LSTM is trained on the complete training set available. We present the results from this normal case in experiment # 01. In order to evaluate KINN’s performance in cases where the amount of training data available is small or the expert is inaccurate, we established two different sets of experiments starting from the configuration employed in the first experiment. In the first case, we reduced the amount of training data provided to the models for training. We present the findings from this experiment in experiment # 02. In the second case, we reduced the reliability of the expert predictions by injecting random noise. The results from this experiment are summarized in experiment # 03. A direct extension of the last two experiments is to evaluate KINN’s performance in cases where both of these conditions hold i.e. the amount of training data is reduced as well as the expert is noisy. We summarize the results for this experiment in experiment # 04. Finally, we evaluated KINN’s performance in cases where the expert contained no information. We achieved this using two different ways. We first evaluated the case where the expert always predicted the value of zero. In this case, the target was to evaluate the impact (if any) of introducing the residual learning scheme since the amount of information presented to the LSTM network was exactly the same as the isolated LSTM model in the first experiment. We then tested a more realistic scenario, where the expert model replicated the values from the last time-step of the series. We elaborate the findings from this experiment (for both settings) in experiment # 05.
We first tested both the LSTM as well as the expert model in isolation in order to precisely capture the impact of introducing the residual learning scheme. KINN demonstrated significant improvements in training dynamics directly from the start. KINN converged faster as compared to the isolated LSTM. As opposed to the isolated LSTM which required more training time (epochs) to converge, KINN normally converged in only one fouth of the epochs taken by the isolated LSTM, which is a significant improvement in terms of the compute time. Apart from the compute time, KINN achieved a MSE of 0.74 on the test set. This is a very significant improvement in comparison to the isolated LSTM model that had a MSE of 5.90. Even compared to the expert model, KINN demonstrated a relative improvement of 40% in terms of MSE. Fig. 6 showcases the predictions made by KINN along with the isolated LSTM and the expert network on the test set. It is evident from the figure that KINN caters for the weaknesses of each of the two models involved using the information contained in the other. The resulting predictions are more accurate than the expert network on minimas and also captures the small variations in the series which were missed by the LSTM network.
In order to further evaluate the results, error at each timestep is compared for the isolated models along with KINN. To aid the visualization, step-wise error for first 100 timesteps of the test set is shown in Fig. 6. The plot shows that the step-wise prediction error of KINN is less than both the expert model as well as the LSTM for major portion of the time.
However, there are instances where predictions made by KINN are slightly worse than those of the baseline models. In particular, the prediction error of KINN exceeded the error of the expert network for only 30% of the time-steps and only 22% of the time-steps in case of the LSTM network. Nevertheless, even in those instances, the performance of KINN was still on par with the other models since on 99% of the time-steps, the difference in error is less than 1.5.
One of the objectives of KINN was to reduce dependency of the network on large amount of labelled data. We argue that the proposed model not only utilizes expert knowledge to cater for shortcomings of the network, but also helps in significantly reducing its dependency on the data. To further evaluate this claim, a series of experiments were performed. KINN was trained again from scratch using only 50% of the data in the training set. The test set remained unchanged. Similarly, the LSTM network was also trained with the same 50% subset of the training set.
The LSTM network trained on the 50% subset of the training data attained a MSE of 6.36 which is slightly worse than the MSE of network trained on the whole training set. Minor degradation was also observed in the performance of the expert network which achieved a MSE of 1.52. Despite of this reduction in the dataset size, KINN achieved significantly better results compared to both the LSTM as well as the expert model achieving a MSE of 0.89. Fig 7 visualizes the corresponding prediction and error plots of the models trained on 50% subset of the training data.
We performed the same experiment again with a very drastic reduction in the training dataset size by using only 10% subset of the training data. Fig. 8 visualizes the results from this experiment in the same way, by first plotting the predictions from the models along with the error plot. It is interesting to note that since the LSTM performed considerably poor due to extremely small training set size, the network shifted its focus to the predictions of the expert network and made only minor corrections to it as evident from Fig. 8(a). This highlights KINN’s ability to decide its reliance on the expert predictions based on the quality of the information. In terms of the MSE, LSTM model performed the worst. When trained on only the 10% subset of the training set, the LSTM model attained a MSE of 6.68, whereas the expert model achieved MSE of 2.67. KINN on the other hand, still outperformed both of these models and achieved a MSE of 1.53.
In all of the previous experiments, the expert model was relatively better compared to the LSTM model employed in our experiments. The obtained results highlights KINN’s ability to capitalize over the information obtained from the expert model to achieve significant improvements in its prediction. KINN also demonstrated amazing generalization despite of drastic reduction in the amount of training data, highlighting KINN’s ability to achieve accurate predictions in low data regimes. However, in conjunction to reducing dependency of the network on data, it is also imperative that the network does not become too dependent on the expert knowledge making it essential to be accurate/perfect. This is usually not catered for in most of the prior work. We believe that the proposed residual scheme enabled the network to handle erroneous expert knowledge efficiently by allowing it to be smart enough to realize weaknesses in the expert network and adjust accordingly. In order to verify KINN’s ability to adjust with poor predictions from the expert, we performed another experiment where random noise was injected into the predictions from the expert network. This random noise degraded the reliability of the expert predictions. To achieve this, random noise within one standard deviation of the average traffic flow was added to the expert predictions. As a result, the resulting expert predictions attained a MSE of 7.81 which is considerably poor compared to that of the LSTM (5.90). We then trained KINN using these noisy expert predictions. Fig. 9 visualizes the corresponding prediction and error plots.
As evident from Fig. 9(a), KINN still outperformed both the expert as well as the LSTM with a MSE of 3.09. Despite the fact that neither the LSTM, nor the expert model was accurate, KINN still managed to squeeze out useful information from both modalities to construct an accurate predictor. This demonstrates true strength of KITNN as it not only reduces dependency of the network on the data but also adapts itself in case of poorly made expert opinions. KINN achieved a significant reduction of 48% in the MSE of the LSTM network by incorporating the noisy expert prediction in the residual learning framework.
As a natural followup to the last two experiments, we introduced both conditions at the same time i.e. reduced training set size and noisy predictions from the expert. The training set was again reduced to 10% subset of the training data for training the model while keeping the testing set intact. Fig. 10 demonstrates that despite this worst condition, KINN still managed to outperform both the LSTM as well as the noisy expert predictions.
As the final experiment, we evaluated KINN’s performance in cases where the expert predictions are not useful at all. We achieved this via two different settings. In the first setting, we considered that the expert network predicts zero every time. In the second setting the expert network was made to lag by a step of one resulting in mismatch of the time step with the predictions. Putting zero in place of x^tp in Eq. 2 yields: x^t = ([xt 1; xt 2; :::; xt p; 0]; W ) + 0
This is almost equivalent to the normal unconditioned full input to output space projection learning case (Eq. 1) except a zero in the conditioning vector. However, in case of lagged predictions by the expert network, since we stack the expert prediction x^tp in a separate channel, the network assigns a negligible weight to this channel, resulting in exactly the same performance as the normal case.
Table 1 provides the details regarding the results obtained for this experiment. It is clear from the table that in cases where the expert network either gave zero as its predictions or gave lagged predictions, which is useless, the network performance was identical to the normal case since the network learned to ignore the output from the expert. These results highlight that KINN provides a lower bound on the performance based on the performance of the two involved entities: expert model and the network.
These thorough experiments advocate that the underlying residual mapping function learned by KINN is successful in combining the network with the prediction made by the expert. Specifically, KINN demonstrated the ability to recognize the quality of the prediction made by both of the base networks and shifted its reliance according to it. In all of the experiments that we have conducted, MSE of the predictions made by KINN never exceeded (disregarding insignificant changes) the MSE of the predictions achieved by the best among the LSTM and the expert model except in case of completely useless expert predictions, where it performed on par with the LSTM network. Table 1 provides a summary of the results obtained from all the different experiments performed. It is interesting to note that even with a huge reduction in the size of the training set, the MSE does not drastically increase as one would expect. This is due to the strong seasonal component present in the dataset. As a result, even with only 10% subset of the training data, the algorithms were able to learn the general pattern exhibited by the sequence. It is only in estimating small variations that these networks faced difficulty when trained on less data.
We propose a new architecture for incorporating expert knowledge into the deep network. It incorporates this expert knowledge in a residual scheme where the network learns a correction term for the predictions made by the expert. The knowledge incorporation scheme introduced by KINN has three key advantages. The first advantage is regarding the relaxation of the requirement for a huge dataset to train the model. The second advantage is regarding the provision of a lower bound on the performance of the resulting classifier since KINN achieves the best of both worlds by combining the two different modalities. The third advantage is its robustness in catering for poor/noisy predictions made by the expert. Through extensive evaluation, we demonstrated that the underlying residual function learned by the network makes the system robust enough to deal with imprecise expert information even in cases where there is a dearth of labelled data. This is because the network does not try to imitate predictions made by the expert network, but instead extracts and combines useful information contained in both of the domains.
This work is partially supported by Higher Education Commission (Pakistan), ”Continental Automotive GmbH” and BMBF project DeFuseNN (Grant 01IW17002).