Automation needs safety inbuilt in the system such that it behaves at least as well as a diligent human in unforeseen circumstances, if not better. It is therefore necessary that the machine learns to behave intuitively by predicting future occurrences and take actions accordingly. Machine learning techniques, therefore, have to focus on safety issues. Human development and the consequential environmental changes will only push safety requirements higher demanding artificial intelligence to fill in the voids so generated. The purpose of this paper is to study the artificial intelligence perspective on safety challenges and concerns of such systems through an extensive literature review and propose a futuristic and easily adaptable system using deep learning technique. The paper would focus primarily on safety aspects of autonomous vehicles using Bayesian Deep learning method.
Current trends in the automotive industry are introducing
new, increasingly complex software functions into vehicles
Success stories in deep learning have made AVs more
or less a reality, however, commercializing such vehicles
have not yet fructified. Recent accidents, especially those
involving cars driving at as low as SAE Level 2 show,
that there are challenges engineers are still faced with,
and the major impediment that stands in the way of
largescale adoption of AVs, is its associated safety concerns
Although there is no concrete solution in addressing
the safety concern, several researchers have outlined the
safety challenges and proposed recommendations to
consider. Salay et al.
(i) Implicit Specification – Traditional Verification and Validation (V&V) methods (as suggested in ISO 26262 V model) lay great importance on ensuring that the functional requirements specified at the design-time of the system are met. However, NN-based systems depend solely on the training data for inferring the specifications of the model and do not depend on any explicit list of requirements, which can be problematic while applying traditional V&V methods. (ii) Black-Box Structure – While writing the code for a NN, one specifies the details about the layers and the activation functions, but, unlike traditional software, the control flow is not explicitly coded, leading to NNs being referred to as black-box structures. Traditional white-box testing techniques such as code coverage and decision coverage cannot be directly applied to NNs, thus, there is a need to construct paradigms for adaptive software systems.
We distinguish between the existing approaches by categorizing them into two groups: (i) ‘Training phase’, i. e.approaches that are solely used during the development and training phase of the neural network, and (ii) ‘Operational phase’, i. e.those that are used in the run-time environment of the neural network to ensure proper functioning (RQ2).
The existing approaches that fall under this category are:
(i) Train/Validation/Test split – This method is used to
ensure that the developed adaptive system works satisfactorily
for a given set of inputs. The method involves splitting the
available data, to obtain three sets, such that the largest of
the sets is used solely for training, and of the remaining two
sets, one is used for fine tuning the hyperparameters of the
NN, and the second one is used to test the working
neural network to study how well it reacts to previously unseen
data points. Though this method helps verify the working of
the NN, it is not extensive enough to be considered a
guarantee for safety
The solutions that fall under this category can be more accurately referred to as ‘Online monitoring techniques’, that involve the use of one or more monitors working as an oracle to ensure continued proper functioning of the neural network over time (Cukic et al. 2006). The goal here is to ensure that the adaptation dynamics does not cause the network to diverge, thereby triggering unpredictable behavior.
Data Sniffing
Majority of the contemporary approaches, as evident from
“Related Work” section, relate to testing a developed model
before it is deployed in the operational environment.
MLbased components, however, suffer from problems like;
operational data/platform being different from what the model
was trained on, uncertainty about the new inferences gained
from operational data, and even wear-and-tear of
hardware/software. This leaves ML-based components
vulnerable to errors. Thus, it is necessary to focus on
monitoringbased approaches, which are starting to gain recent
interest
To elaborate on specifics of the proposed solution, an end-to-end deep learning model for lane change maneuvers has been chosen. Such a model uses a deep neural net that takes input data from sensors that represent the environment around the ego vehicle, and generate one of three actions allowing the ego vehicle to continue driving in the current lane or to switch to the left or right lane depending on the presence of obstacles.
This proposed solution, referred to as ‘Crash Prediction Network’, involves a neural network model, tasked with determining the likelihood and severity of a crash at any given time step (RQ3). The model takes into consideration multiple features such as output of the perception module of the vehicle, planned trajectory/action of the ego vehicle, predicted (or intended, if available via V2V communication) trajectory of the obstacles, and possibly also information such as number and severity of previous crashes that the ego vehicle and obstacles were involved in. Specifics of the system can be understood by distinguishing between the training and operational (after deployment) phases of the model.
The training phase (as shown in Fig. 1) relies on the model receiving the required input values for the previously described feature set, and also knowing whether a crash occurred or not. Thus, the model requires an architecture that involves a Reinforcement Learning environment, that would allow the model to know the outcome at every time step for a given set of feature values. This would also allow the vehicle to crash often, as is characteristic of RL-agents, especially at the start of training. We, therefore, propose to train the model by allowing it to spar with an RL-agent such that the ego vehicle closely imitates a real-world vehicle that can perform tasks similar to the lane change maneuver use-case described above. At each time step, the RL-agent and the Crash Prediction Network will have access to information about the environment of the vehicle, the Crash Prediction Network will predict whether a crash occurs or not, while simultaneously, the RL agent would interact with the environment to determine whether a crash really occurred or not. Based on the differences in the output of the two networks, the Crash Prediction Network would be updated to eventually be able to predict crashes with a high level of accuracy.
The operational stage (as show in Fig. 2) of this model is designed such that the inputs as usual are fed to the MLbased component responsible to determine the lane change maneuver to be carried out by the ego vehicle. The vehicle, however, does not act directly on the generated lane change action command. The action command along with the environmental inputs in the form of sensor data are directed to the Crash Prediction Network, which performs its task of predicting the likelihood of a crash. Only if the likelihood is low, is the vehicle allowed to perform the desired actions, else the vehicle is pushed into Fail-safe mode which varies depending on the predicted severity of the crash. It is important to note that for the model to stay relevant to the environment, it needs to learn and improve even in the operational stage. Thus, similar to the training stage the difference between the actual output and predicted output are used to update the model.
Crash Prediction Network is based on Bayesian Deep
Learning (BDL). The reason being that other Deep
Learning methods in use currently are known to make hard
classifications based on what they see and what they perceive.
The disadvantage with this method becomes apparent in a
system such as an AV where multiple components come
together to form a complex whole, an error in one
component could have a snowball effect up the pipeline, leading to
catastrophic outputs in the later components. A way to get
over this problem is to use BDL
The proposed system has definite advantages. Most importantly, such a system does not just focus on futuristic autonomous vehicles, but, can even be used in current day Advanced Driving Assistance Systems (ADAS) as well, thereby allowing a smoother transition to Autonomous Vehicles in future. Secondly, the model can be seen as making an intuitively ‘informed decision’, by taking into consideration data from multiple sources. Additionally, such a system would also generalize and scale well to different scenarios that the vehicle might encounter. One of the major problems that would be encountered during the development of the model, however, is the consideration of handling input data received from different sources in varied formats. Next, redundancy needs to be inbuilt to compensate for sensor failures/malfunctions in such a way that failure of a sensor does not affect the accuracy of the system. Another major aspect, apropos this methodology that needs experimentation and validation is that of having one ML based component supervising another.
This work covered the different aspects of safety for intelligent components which employ machine learning techniques in order to enable the integration of artificial intelligence for autonomous driving. The focus was on the main concerns and challenges to ensure safety in highly critical applications which are based on machine learning methods, with special emphasis on neural networks. Traditional safety approaches are not sufficiently poised for such systems and therefore, there is a need for more concrete methods like monitoring techniques, such as the one proposed Crash Prediction Network, which guarantees an acceptable level of safety for the system functions. The team is in the process of implementing and evaluating the proposed approach.