This research highlights the negative impact of ignoring uncertainty on DNN decision-making and Reliability. Proposed combined preprocessing and post-processing methods enhance DNN accuracy and Reliability in time-series binary classification for 5G UAV security dataset, employing ML algorithms and confidence values. Several metrics are used to evaluate the proposed hybrid algorithms. The study emphasizes the XGB classifier's unreliability and suggests the proposed methods' potential superiority over the DNN softmax layer. Furthermore, improved uncertainty calibration based on the Reliability Score metric minimizes the diference between Mean Confidence and Accuracy, enhancing accuracy and Reliability.
Deep Neural Networks (DNNs) have seen extensive deployment due to their recent achievements
in several fields. Prediction distributions generated by such models increasingly make decisions
in the telecommunications and security sectors [
For example, 6G telecommunication systems will incorporate Machine Learning (ML)
mechanisms such as DNNs into their standards [
The authors in [
Understanding questions of risk, uncertainty, and trust in a model’s output becomes
increasingly important when augmented techniques are used at the original data preprocessing stage.
The authors in [8] suggest that prepossessing and post-processing techniques can improve
inputs and class DNNs. The authors in [
Inspired by the possibility of choosing a tolerable degree of uncertainty and increasing the reliability of DNN outputs used in 5G UAV security, this study presents several new combined prepossessing and post-processing techniques that increase the overall accuracy and reliability of binary classification deep networks by adjusting the uncertainty. We assess these methods using seven key performance metrics related to errors in calibration and in confidence values. Then, we utilize the Reliability Score (RS) that measures the diference between the Mean Accuracy (MA) and Mean Confidence (MC) to measure the degree of uncertainty. Finally, we evaluate the proposed algorithms’ impact on the DNN’s performance compared to the baseline DNN with no algorithms applied and the DNN added to the eXtreme Gradient Boosting (XGB) classifier. The XGB classifier is selected because of its superior accuracy in comparison to five other classifiers we test with our data [9].
Our dataset contains data from the Received Signal Strength Indicator (RSSI) and the Signal to Interference-plus-Noise Ratio (SINR) measurements collected when an authenticated UAV is connected in the small cell through the 5G communication system, and there are power attacks from other UAVs in the network. There are other terrestrial users connected to the network. The measured parameters in the authenticated UAV change as the interference from the other devices increases or decreases. More details on the dataset construction and one possible application for the dataset is available in [10] and in [11].
We apply this method on the probabilistic outputs of the DNN for all the augmented samples. Each output is in the one hot encoding form for binary classification. For example, [, 1 − ] in which is a number between zero and one.
In Method 2, we convert outputs from a probabilistic to an integer form and apply a majority voting algorithm to them.
Method 3 calculates the confidence of each output. The output with the maximum confidence value is selected as the final result.
We use well-known metrics proposed by [
Accuracy per Confidence. This metric is used in its visual form to analyze the calibration and uncertainty of the DNN model.
Mean Confidence and Mean Accuracy. These two metrics are the total weighted average of confidence and accuracy for the number of samples per each confidence interval.
Reliability Score. We define the diference between the MC and MA values by another metric which is denominated the Reliability Score.
Expected and Maximum Calibration Errors. At each confidence interval, the accuracy deviation away from the confidence interval center is considered as the error per each interval. The Expected Calibration Error is defined as the weighted error and the Maximum Calibration Error describes the maximum error per all intervals.
Negative LogLikelihood Loss (NLL). This metric is known as cross-entropy loss and is used as a loss function for DNNs [12]. It is also utilized as a metric to measure the quality of the probabilistic model [13].
Brier Score Loss (BSL). This metric is defined by the square error of the predicted probability vector and ground truth values in one hot encoding form.
In this section, we present the simulations results. We compare the results of the DNN using each of the five prospective methods to the results of the DNN with no method and the DNN with the XGB. We choose XGB because it is the best performing publicly available classifier applied to our dataset in terms of accuracy [11]. We use the Accuracy vs the Confidence Intervals Central Values and we evaluate the performance of each algorithm using the seven metrics previously mentioned in subsection 2.2.
It is expected to have ML mechanisms in 5G and 6G UAV communication systems. Therefore, it is fundamental to understand the uncertainties of the deep networks used in those systems and how reliable they are. In this study, we proposed five combined methods to increase accuracy and reliability concomitantly in binary classification deep networks applied to UAV security scenarios. By analyzing seven reliability metrics and the accuracy per confidence, Method 1 combined with Method 3 presented the best overall performance that satisfied most of the metrics by achieving the top three in each one. This algorithm reached an ECE of 2.19 and was closer to all ideal levels’ values.
Method 3 was the second-best performing algorithm in terms of reliability. With Method 2 + XGB, we showed that a lower performing ML algorithm can be combined with one of the proposed methods to increase the total DNN accuracy, but in terms of the reliability, this might not be a good option.
Finally, four of the five methods presented were able to increase accuracy, but not all of them increased the reliability. As a result, network engineers and developers must take extra precaution when proposing DNN architectures and analyze them in terms of accuracy and reliability.
This research received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Project Number 813391. Also, this work was partially supported by Fundação para a Ciência e a Tecnologia and Instituto de Telecomunicações under Project UIDB/50008/2020. [8] D. Hafner, D. Tran, T. P. Lillicrap, A. Irpan, J. Davidson, Noise contrastive priors for functional uncertainty, in: Conference on Uncertainty in Artificial Intelligence, 2018. [9] J. Viana, H. Farkhari, P. Sebastiao, L. M. Campos, K. Koutlia, B. Bojovic, S. Lagen, R. Dinis, Deep attention recognition for attack identification in 5g uav scenarios: Novel architecture and end-to-end evaluation, 2023. arXiv:2303.12947. [10] J. Viana, H. Farkhari, P. Sebastiao, S. Lagen, K. Koutlia, B. Bojovic, R. Dinis, A synthetic dataset for 5g uav attacks based on observable network parameters, 2022. arXiv:2211.09706. [11] J. Viana, H. Farkhari, L. M. Campos, P. Sebastião, K. Koutlia, S. Lagén, L. Bernardo, R. Dinis, A convolutional attention based deep learning solution for 5g uav network attack recognition over fading channels and interference, in: 2022 IEEE 96th Vehicular Technology Conference (VTC2022-Fall), 2022, pp. 1–5. doi:10.1109/VTC2022-Fall57202.2022. 10012726. [12] T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning: Data Mining,
Inference, and Prediction, Springer series in statistics, Springer, 2009. [13] M. P. Naeini, G. F. Cooper, M. Hauskrecht, Obtaining well calibrated probabilities using bayesian binning, in: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, AAAI Press, 2015, p. 2901–2907.