This paper proposes a practical list of safety concerns and mitigation methods for visual deep learning algorithms. The growing success of deep learning algorithms in solving nonlinear and complex problems has recently attracted the attention of safety-critical applications. While the state-of-the-art methods achieve high performance in synthetic and real-case scenarios, it is impossible to verify/validate their reliability based on currently available safety standards. Recent works try to solve the issue by providing a list of safety concerns and mitigation methods in generic machine learning algorithms from the standards' perspective. However, these solutions are either vague, and non-practical when dealing with deep learning methods in real-case scenarios, or they are shallow and fail to address all potential safety concerns. This paper provides an in-depth look at the underlying cause of faults in a visual deep learning algorithm to find a practical and complete safety concern list with potential state-of-the-art mitigation strategies.
Deep learning is a powerful tool that solves mathematically challenging tasks with high dimensional inputs and multi-variable optimization requirements such as human reidentification, optical character recognition, and object detection. The learning process involves using heuristic and numerical methods, which are often hard to explain or interpret as the dimension grows (black-box behavior).
While state-of-the-art deep learning algorithms achieve high performance in various synthetic and real-life cases, there is no guarantee for the reliability requirements that safety-critical applications typically demand since available safety standards do not provide a suitable verification/validation method for deep learning models.
Recent works found another way of dealing with the problem. By explaining the potential safety concerns of a deep learning algorithm, it is possible to provide suitable mitigation methods around them. While the overall strategy sounds effective, most works fail to provide a practical list of safety concerns and mitigation methods. These lists are typically vague, impractical to implement, shallow, and incomplete.
This paper focuses on the underlying cause of faults in a visual deep learning algorithm to provide a list of safety concerns and potential state-of-the-art mitigation methods. The main contributions of this paper are: • Providing a practical, complete, and categorical list of possible faults with their underlying cause for different visual deep learning algorithm components. • Providing potential state-of-the-art mitigation methods to deal with the faults.
The rest of this paper is structured as follows. Section 2 covers related works. Next, Section 3 explains safety concerns related to a visual deep learning algorithm and provides existing mitigation methods to deal with them. Finally, Section 4 concludes the work.
2
A visual deep learning algorithm is prone to different types of faults. Recent papers focus on either solving specific faults or providing an overview of all system-related safety concerns. Here we discuss some of the most important contemporary works:
Zhang’s review of recent papers explains how violation of
critical assumptions in the training stage would lead to faults
and a non-robust system
Kla¨s suggests using uncertainty wrappers on deep
learning components to ensure the outcome is dependable
Wozniak, Schwalbe, and Willers suggest different
approaches to providing a safety concern list and mitigation
methods for developing a deep learning algorithm
(Wozniak et al.
Houben provides an extensive list of practical methods to
improve the safety of a deep learning algorithm
Other similar works, such as
The development of a visual deep learning algorithm has three major stages: (1) training, (2) evaluation, and (3) inference. This section presents the list of possible faults within each stage. 3.1
Visual data is one of the significant sources of information for deep learning algorithms. Extracting useful information from visual data is a complex task that makes it prone to faults.
A deep learning algorithm approximates the relationship between the input data and the objects in the real world by reducing the empirical risk on training data. Thus, having a proper training dataset is essential to reach the desired quality in the algorithm. A training dataset should be: • Complete: contain samples from the defined output space for the task. • Adequate: contain samples with identical distribution to real-world. • Ample: contain a sufficient amount of samples for convergence of the algorithm. • Clean: contain well-labeled samples.
Moreover, different model structures come with specific sets of benefits and weaknesses. Choosing the correct model, setting up a suitable loss function and optimization algorithm, and finding the perfect hyperparameters are essential to achieve the best performance.
SC 1 – Incomplete Dataset: Due to the natural complexity of the real world, there is always a much larger open space than the defined output space for the task. Even with defined boundaries for the output space, known unknowns ( e.g., outlier classes) and unknown unknowns (e.g., adversarial attacks) pose a significant issue for the algorithm by producing over-confident wrong predictions.
SC 2 – Inadequate Dataset: Due to the ever-changing nature of real-world conditions, the collected data for training will not have identical distribution with the real-world environment in the inference stage. Even a slight mismatch in the distribution can cause a significant drop in performance and result in poor generalization.
ally labeling a dataset increases exponentially with its size. While having a small clean validation dataset is feasible, larger datasets tend to have noisy labels. A deep learning algorithm can memorize this noise, leading to poor generalization and low performance.
SC 4 – Ill-Matched Architecture: Manually comparing different models and hyperparameters to find the best match for the task is time-consuming and costly. Moreover, it requires an expert in the field to provide an insight into the problem. An ill-matched architecture could result in unforeseen faults due to inherent weakness against specific situations that might exist.
MM 1 – Learning with Unseen Data: Modern deep
learning tools could be utilized to force the boundaries of
the training dataset even further. Out-of-distribution
detectors can be used in the algorithm to detect unseen samples
in the inference stage and reject the over-confident results of
the algorithm. These methods introduce uncertainty metrics
to determine whether the algorithm should be trusted or not
Also, open-world recognition systems can be used to
extend the output space of the algorithm as it encounters
outlier samples in the inference stage. These methods
continue to learn new classes during the inference stage to
reduce the chance of over-confident wrong predictions
Moreover, the model could be trained to defend against adversarial attacks by including such patterns in the training dataset (Xu et al. 2020; Yuan et al. 2019).
Discussing MM 1: Out-of-distribution detectors typically result in lower accuracy, open-world recognition systems are slow and demanding, and adversarial attacks keep evolving and changing every day. The mentioned methods all have their limitation. A suitable backup plan would involve utilizing several models with various mitigation methods to create an ensemble to vote for the final result.
Modern deep learning tools could be utilized to reduce the
distribution mismatch between the training and inference
domain. Transfer learning and domain adaptation can be
used to fine-tune the algorithm online during the inference
stage. These methods help the model to adapt to new
environments quickly and achieve better generalization by using
a small batch of data in the inference stage
On the other hand, the algorithm can achieve higher
performance by utilizing multiple sources of information
for a single task (e.g., person identification with face, iris,
voice, and fingerprint). Multimodal learning methods
incorporate supplementary and complementary data from
multiple modalities to the performance of a single task
Discussing MM 2: Transfer learning and domain adaptation methods typically rely on having a decent starting point (trained network) and quality samples from the inference stage to fine-tune the model successfully. While the requirements are hard to achieve, it is not impossible. Moreover, multimodal methods have already been used with sensor fusion in autonomous vehicles (LIDAR, GPS, IMU, and so on), making them a strong candidate for use in deep learning systems. A suitable backup plan would involve storing the input data during the inference stage to re-evaluate and re-calibrate the algorithm by replacing parts of the older and non-useful training dataset in an iterative cycle.
Modern deep learning tools could be utilized to reduce
the effect of label noise or eliminate the need for a large
labeled dataset. Robust loss, sample selection, relabeling,
and weighted training are all potential solutions to deal
with noisy labels in the training dataset
On the other hand, data augmentation methods can be
used to create additional samples for the training dataset.
These methods typically involve rotating, scaling, shifting,
and flipping data
Moreover, the cost and time for manually labeling
datasets can be drastically reduced by using iterative
labeling methods
Finally, semi-supervised and unsupervised training
techniques can be used to decrease the dependency on a clean
training dataset
Discussing MM 3: Recent works prove that the label noise is instance-dependent, as shown in Figure 1. This discovery means most state-of-the-art methods in dealing with label noise need revision on how to mitigate the effects of label noise. Recent works happen to focus on this topic and provide effective solutions. While these solutions do not have mathematical proof, they perform decently on public benchmarks.
Meanwhile, the research around synthesized data indicates that it may not represent the real world in every situation due to the limitations of simulation environments and lack of involved experts in the process. Moreover, the existing public datasets might not suit the specific task or have other inconsistencies, such as low-quality images and noisy labels.
A suitable backup plan would involve developing a more realistic simulation environment while including the physical knowledge about the task in the training process.
deep learning tools could be utilized to select the optimum
model and hyperparameter for a given task. Automated
hyperparameter optimization
Discussing MM 4: Relying on search algorithms requires high computational power and proper comparison tools. While it will cost money and time to do it, the solution is not impossible or impractical in most safety-critical applications. 3.2
Evaluation of a trained deep learning algorithm requires prior knowledge about the task. A testing dataset should include samples from all scenarios, no matter how rare, to ensure the safety of the algorithm. Also, proper performance metrics should be selected during the tests to obtain comparable outputs.
Moreover, formal verification/validation methods depend on having an interpretable algorithm, which contrasts deep learning.
SC 5 – Incompatible Metrics and Benchmarks: The most common performance metric in deep learning algorithms is accuracy. However, other metrics might hold more value in safety-critical applications as the importance of false-positive and false-negative grow exponentially in this field. Moreover, gathering a proper dataset to use as a benchmark has similar challenges to the training dataset.
Benchmarks: By including a weighted cost for each type
of fault in the performance metric, the algorithm can be
evaluated according to safety requirements
On the other hand, a list of all hazardous scenarios can be
prepared for every task for inclusion in the testing dataset
by performing a risk analysis on the task
Discussing MM 5: While formulating a new cost function requires expert knowledge, it is within the scope of expectations in a safety-critical application. Various combination of weighted metrics can be utilized and compared to find the most suitable one for the task. However, a bad decision could result in a non-converging algorithm, thus there is a necessity for mathematical proof about the convergence of the algorithm.
Moreover, the competitive nature of industry typically prevents them from sharing any suitable benchmarks or cost functions publicly, which means each company has to spend time and resources on developing their own system. A suitable backup plan would involve third-party associations funded by multiple companies to handle the problem for the benefit of all members.
SC 6 – Black-Box Behavior: The large volume of parameters and non-linear functions in deep learning algorithms result in an uninterpretable system. With no clear relation between the input and output of this black-box system and the impossible task of testing the entire input domain, it is hard to verify/validate deep learning algorithms based on safety standards.
MM 6 – Opening the Black-Box: Representation
learning enables the deep learning algorithm to discover the
relation between input data and output in a presentable way
by showing the process of feature selection
Another way to gain such insight is to present a map of
pixel relevance for the algorithm. These heat maps illustrate
the importance of each pixel when calculating the output
Discussing MM 6: This specific problem could be one of the most important ones with the least proper solutions as of yet. While it is possible to gain some insight into the operation of deep learning algorithms, the information cannot be used in any form to verify/validate the algorithm based on traditional standards. A suitable backup plan would involve using safety case arguments and other similar approaches to bypass the need for verification/validation for now. 3.3
In a typical case, a similar sensor used for collecting offline data provides the online data for the implemented algorithm. On top of it, other hardware components are required for the algorithm to work correctly. These components can be summarized as: • A camera to capture the input image. • A communication channel to transfer the captured image. • A processing unit to host the deep learning algorithm. • A power supply to keep the system running.
SC 7 – Defective Hardware: The first concern in deep learning algorithms is providing the necessary hardware mentioned above. Hardware faults can have a wide range of effects on the algorithm based on the faulty component, an example being the results of a faulty camera on the captured image, as shown in Figure 2. An implementation of the algorithm might run into problems based on the defective hardware component: • Camera faults that might result in various disturbances in the input image, such as pixel corruption or image distortion. • Communication channel faults that might result in data corruption or data loss. • Processing unit faults that might result in wrong calculations, lagging, or freezing of the algorithm. • Power supply faults might result in breaking other hardware components or total system shutdown.
MM 7.1 – Following Functional Safety Standards: The
mentioned hardware components are not unique to deep
learning algorithms and have been used for decades in
safety-critical applications. As a result, the current
functional safety standards such as ISO 26262
Moreover, other precautions such as using redundant
hardware, proper noise shielding, and data fusion techniques
have already proved helpful in safety-critical applications
Discussing MM 7.1: Assuming the hardware is chosen based on the proper functional safety standards, it should operate without significant safety concerns. However, this mitigation method does not guarantee the complete removal of any disturbance or corruption of data. Environmental factors such as lousy illumination, movement, and obscured objects can affect input image quality without causing a hardware failure, as seen in Figure 3. While some of these problems might not be recognizable by a human annotator, the deep learning algorithm could run into faults based on the type and severity of corruption. Moreover, less severe levels of hardware failure might cause noise variations on the input data. A suitable backup plan would involve utilizing another mitigation approach described as follows.
MM 7.2 – Using Image Processing Techniques: Since
the exact relation between the input image and the output
of the deep learning algorithm is not known, it is
recommended to have clean input data to reduce the change of
unwanted outcomes. The current state-of-the-art image
processing techniques such as denoising
Discussing MM 7.2: Image processing techniques are only valid when it’s known that the image is corrupted. Otherwise, such functions can negatively affect a clean image during the operation (e.g., removing/fading edges, brightening the image without necessity, etc.). Applying a filter without knowing the type of corruption is almost as dangerous as not utilizing any technique. So, it is safe to assume that some form of corruption is inevitable. A suitable backup plan would involve using the rejection option as described before to reduce the amount of overconfident wrong outputs.
4
The research around using deep learning algorithms in safety-critical applications is growing rapidly, with the current state-of-the-art answers partially fulfilling the requirements of old standards. However, the nature of the problem demands to move away from the traditional broad-spectrum method of standardization as it is not suitable for deep learning algorithms. There is a high demand for task-specific standards to be developed. Until such standards are developed, the research community focuses on alternative approaches and empirical analysis to provide practical solutions on specific cases.
This paper provides a practical list of safety concerns for a visual deep learning algorithm by explaining the underlying cause of faults and providing current state-of-the-art solutions to mitigate them. By presenting the limitations of existing mitigation methods, the need for further study is expressed. We hope this paper offers an insight to those who want to utilize deep learning algorithms in their applications or those who want to develop proper standard or safety case arguments for such systems.
This research is done as part of a Ph.D. study co-funded by Tampere University and Forum for Intelligent Machines ry (FIMA).