Deep learning has been at the core of the autonomous driving field development, due to the neural networks' success in finding patterns in raw data and turning them into accurate predictions. Moreover, recent neuro-symbolic works have shown that incorporating the available background knowledge about the problem at hand in the loss function via t-norms can further improve the deep learning models' performance. However, t-norm-based losses may have very high memory requirements and, thus, they may be impossible to apply in complex application domains like autonomous driving. In this paper, we show how it is possible to define memory-eficient t-norm-based losses, allowing for exploiting t-norms for the task of event detection in autonomous driving. We conduct an extensive experimental analysis on the ROAD-R dataset and show (i) that our proposal can be implemented and run on GPUs with less than 25 GiB of available memory, while standard t-norm-based losses are estimated to require more than 100 GiB, far exceeding the amount of memory normally available, (ii) that t-norm-based losses improve performance, especially when limited labelled data are available, and (iii) that t-norm-based losses can further improve performance when exploited on both labelled and unlabelled data.
Deep learning has been at the core of the autonomous driving field development [
In this paper, we show how it is possible to define memory-eficient t-norm-based losses, allowing for exploiting t-norms for the task of event detection in autonomous driving. We conduct an extensive experimental analysis on the ROAD-R dataset [14] and show that our proposal can be implemented and run on GPUs with less than 25 GiB of available memory, while standard t-norm-based losses are estimated to require more than 100 GiB, far exceeding the amount of memory normally available. Then, we train our state-of-the-art event detection model using diferent percentages of labelled training data (i.e., 10%, 20%, 50%, 75%, and 100%), and we show that while t-norm-based losses can improve the performance of the models in all cases, they are particularly helpful when data are scarce. Indeed, our models yield an improvement of up to 1.85% and 3.95% when using, 10% and 20% of the labelled training data, respectively. Finally, we investigate the behaviour of the t-norm-based loss when only 10% annotated data are available, along with 10% unlabelled data, and find that applying the t-norm-based loss on both labelled and unlabelled data after a warm-up training phase leads to further improvements, i.e., up to 2.75% w.r.t. the fully supervised baseline.
pairs (, )
where: An event detection problem is a pair ( , ) , where is a finite set of labels, and is set of 1. ∈ ℝ 3× ×
is the tensor associated with each frame in the video. (resp., ) represents the width (resp., height) of each frame, while 3 is the number of channels used in the RGB encoding, 2. is the ground truth of comprising a set of pairs (, ℒ ) , where ∈ ℝ 4 represents the coordinates of a bounding box, i.e., a rectangle marking the position of an agent in the frame, while ℒ represents the set of labels associated with .
X 234 ⋯ 78 168⋮
⋯⋱ 3 ⋮ H ⋮ 32 ⋱ ⋯ ⋮ 0 46 ⋮15⋯2 ⋱⋯195⋮21 34 ⋯ 76
W ([498.1,264.1,791.1,621.2], {MedVeh, Brake, Stop, VehLane}) ([801.4,401.9,866.9,476.2], {LarVeh, MovTow, IncomLane}) b
ℒ confidence of the model regarding which labels can be associated with ̂ . Given an output (,̂ )̂̂ , a prediction is then defined as the pair
(,̂ ℒ ̂), where ℒ ̂ is the set of labels associated with ̂ ; a label is associated with ̂ if ̂ ≥ , where is a user-defined threshold. To predict the bounding boxes in each frame, standard of-the-shelf event detection models use anchor boxes, which are predefined boxes of varying sizes at diferent locations within the frame. For each frame, the model defines anchor boxes whose positions are fixed, and then the position of each predicted bounding box is computed as the ofset from one anchor box.
An event detection problem with propositional logic constraints ( , Π) consists of an event detection problem and a finite set of constraints Π, expressed over the set of labels in . We assume w.l.o.g. that the constraints are given as a set of clauses, each of the form: 1 ∨ 2 ∨ ⋯ ∨ , (1) where every is a literal, i.e., is either a label ∈ or its negation, ¬ , for ∈ {1, … , } . Intuitively, (1) expresses the fact that the model should always predict at least one of the literals in the clause, i.e., in { 1, ..., }. We assume that in any clause, a label occurs either positively or negatively at most once. We say that a label occurs positively (resp., negatively) in the clause (1) if there is a literal in (1) such that = (resp., = ¬ ), and that occurs in (1) if occurs either positively or negatively in (1).
Inspired by [
2. the negation using strong negation, which, given ∈ [
Let ( , Π) be an event detection problem with propositional logic constraints. We can then express Π using two matrices + and −, both of size |Π| × | | , such that + = 1 (resp., − = 1), if the -th label appears positively (resp., negatively) in the th constraint, and + = 0 (resp., − = 0), otherwise. We call + (resp., −) the positive (resp., negative) constraints matrix. Let be an event detection model for Π using anchor boxes for each frame. For each input frame, outputs the prediction matrix of size × | | . We call the prediction matrix. Given and , our goal is to compute the degree of satisfaction of each constraint for each output, which can be compactly expressed as a matrix of size × |Π| . Ultimately, we want to use to compute the frame-wise logic-based regularisation term in the loss, and we do this by defining: ( ) = 1 − Standard Approach. Let ̂ be the tensor obtained by stacking |Π| times the matrix along its first dimension. Let ̂ + and ̂ − be the tensors obtained by stacking times the matrices + and − along their second dimension. We then obtain three tensors all of size × |Π| × | | . We can then choose the desired t-conorm and compute the matrix goal as:
+ − − = t-conorm([ (⊙̂ ̂ ) + (̂ − ⊙̂ ̂ )]d,im = 3), where ⊙ is the Hadamard product, and given a generic tensor of size ×× , t-conorm(, dim = 3)returns a matrix of size × whose element at position (, ) is equal to the value of the t-conorm computed over the third dimension, e.g., if we choose the Gödel t-conorm, we obtain t-conorm(, dim = 3) =max ( 1 , 2 , … , ).
Example 3.1. Let ( , Π) be an event detection problem with propositional logic constraints such that = { Car, Moving, Stopped} and Π = {¬Moving ∨ Car, ¬Moving ∨ ¬Stopped}. Then, our positive and negative constraints matrices, assuming labels are numbered as listed in , would be: + = [01 00 00] − = [00 11 01] .
Let be a model for ( , Π) using 3 anchor boxes. Given a prediction matrix as below, and supposing we use the Gödel t-conorm, is equal to:
The problem with this approach is that it requires working with dense 3-dimensional matrices, inducing a large computational overhead and making the method unfeasible, especially for application domains like autonomous driving. For example, given |Π| = 200 constraints, | | = 50 labels, and a model generating = 55 K anchor boxes per frame (a common number for event detection problems) and taking as input sequences of 10 frames at a time, storing a single matrix of size × |Π| × | | requires about 20 GiB (550000 × 200 × 50 × 4 bytes). Moreover, the standard approach works with 5 matrices of this size to compute the t-norm loss in the forward pass and then backpropagate through it. Excluding any other memory allocation needed for storing the input and intermediate outputs, just computing the loss and backpropagating through it would take 100 GiB, exceeding the memory space limits of even the largest GPU available today (i.e., the NVIDIA A100 Tensor Core GPU having 80 GiB RAM [16]). Notice that this computation is done for a single frame, however, normally deep learning models for event detection are trained using batches of 4/8 elements, each comprising 4 to 32 frames in sequence. It is thus impossible to use the above standard dense representation to train event detection models. Sparse Matrix Representation Approach. Our solution mostly relies on the intuition that in practice most of the constraints are written over a subset of the available labels , and that this subset is usually much smaller than . For example, in our experimental analysis, we will see that although there are 41 labels available in ROAD-R, the longest constraint is written over From left to right: (i) t-norm definitions, (ii) respective t-conorm definitions, and (iii-iv) operations to update on the grounds of the chosen t-conorm. Given two matrices, max (resp., min) represent the element-wise operation taking the maximum (resp., minimum) between two elements at the same position in the two input matrices. To simplify the notation, in the last two columns, we used 1 to refer to the matrix of ones of appropriate size.
Gödel Łukasiewicz Product
T-norm min(, ) max( + − 1, 0) ⋅
T-conorm max(, ) min( + , 1) 1 − (1 − )(1 − )
Operation to update +
max(
min( +, ⋅ |⊤+|) + + ⋅
|⊤+|, 1) 1 − (1 − +) ⊙ (1 − ⋅ |⊤+|)
Operation to update −
max( −, 1 − ⋅ |⊤−|) min( − + 1 − ⋅ |⊤−|, 1) 1 − (1 − −) ⊙ ( ⋅ |⊤−|) just 15 labels. As a result, + and − contain mostly zeros. Hence, we designed a method to capture the logic-based loss that makes use of this sparsity property and ultimately avoids the high computational costs induced by the 3D matrices, operating only on 2D matrices.
positively. Analogously, we define − = { − ∶ ∈ }
Given Π, we associate with each constraint an index, and then we define the set of sequences + = { + ∶ ∈ } , where + is the sequence of indices of the constraints in which occurs , where − is the sequence of indexes where 0×|Π| is a matrix of zeros of size × |Π| , and then, for each label ∈ of the constraints in which occurs negatively. Once we know which constraints each label occurs in, we can instantiate the goal matrix to the identity element of the disjunction, iterate through the labels in , and for each label ∈ update, according to the values in , all the columns of associated with constraints where occurs. More specifically, we set = 0 0×|Π| , , we determine: + ⟵ t-conorm( +, ⋅ |⊤+|) − ⟵ t-conorm( −, 1 − ⋅ |⊤−|), where (i) + (resp., −) selects the columns of associated with constraints where occurs positively (resp., negatively), (ii) corresponds to the column of associated with the label , (iii) indicates the unit column vector with elements, and (iv) t-conorm returns the pairwise t-conorm, i.e., given two matrices , of the same size, t-conorm(, ) = t-conorm( , ). Finally, given , we compute
( ) as defined in Equation 2.
Example 3.2 (Example 3.1, cont’d). Let ( , Π) be the problem in Example 3.1 and assume that we associate with the constraint (¬Moving ∨ Car)index 0, and with (¬Moving ∨ ¬Stopped) index 1. Let denote the empty sequence. Then, the sequences associated with each label are: Car = (0), Moving = , S+topped = , C−ar = , M−oving = (0, 1), S topped = (1). + + −
Suppose that we use the Gödel t-conorm, then after having initialized = 0 03×2, we start updating it from the label Car: 0 0 0.1 C+ar = max ([0] , [0.9]) = [0.9] and thus =
0.1 [0.9 0 update it according to M−oving (as M+oving = ):
− Since Car = , we do not need to further update for Car. We then consider the label Moving and Finally, we consider the label Stopped, and perform the last update:
We tested our t-norm loss on the task of event detection for autonomous driving, where the goal is to assign to each detected bounding box in each video a subset of labels—including one agent label, and a subset of the action and location labels. To this end, we used the recently introduced dataset for autonomous driving, ROAD-R [14], which extends the ROAD dataset [17] with 243 manually annotated constraints, provided in disjunctive normal form, as shown in Table 5 from Appendix A. The dataset contains 22 videos, each ∼8 minutes long, annotated with tubelets/tubes that link a sequence of bounding boxes in time. Each bounding box is annotated with a subset of the 41 labels available (listed in Table 4 from Appendix A). We used the available training partition for training the models and, for reproducibility purposes, we reported our results on the validation dataset, as the test set is not publicly available. For our experiments, we used the 3D-RetinaNet [17] detector with a ResNet50 [18] backbone combined with a Random Connectivity Gated Recurrent Unit (RCGRU) [19] for temporal feature learning, which we chose based on its high performance in [14]. We set a weight of 10 for the t-norm loss term and use sequences of 8 frames as input.
Memory assessment. To assess the eficiency of our method for computing the t-norm loss w.r.t. how much GPU memory is allocated during training the models, we compared it to the standard implementation of the t-norm-based loss. We used a Titan RTX GPU with 24 GiB of RAM for training models for 50 iterations on ROAD-R, while using diferent numbers of constraints. Note that, while most of the constraints in ROAD-R contain only two labels, to allow for a fair comparison with the standard implementation, we selected constraints with a diferent numbers of labels. For reference, in each iteration, the number of anchors was of about 67K. Figure 2 shows that our method significantly reduced the memory costs, making it possible to use t-norm-based losses on our dataset, where the number of constraints and data points per batch are both large. Using the standard implementation supports at most 40 constraints; this is 203 constraints less than the ones in ROAD-R. Results. We first investigated how our memory-eficient t-norm-based loss performs in a fully-supervised scenario. To this end, we tested our method with three diferent t-norm losses (i.e., Gödel, Łukasiewicz, and Product) using 10%, 20%, 50%, 75% and 100% of the available annotated ROAD-R data, and training for 110, 70, 45, 30, and 30 epochs, respectively. We used a learning rate of 0.0041 for all models, but for 100% labelled data, we dropped it at epochs 18 and 25 by a factor of 10, as in [14]. We always computed the t-norm-based losses w.r.t. all of the 243 constraints from ROAD-R. For evaluation, we used the frame-wise mean average precision (f-mAP) metric, computed by taking the mean average precision at a fixed intersection-overunion threshold of 0.5 over each frame for each class and then averaging these per-class scores, and reported the result at the best epoch.
Table 2 summarises the results, from which we first observe that t-norm-based losses always improve the baseline performance, except when using 100% labelled data, where only the Łukasiewicz t-norm outperforms the baseline—this being in line with the result on 100% labelled data from our previous work1 [14]. We also notice that unlike the Gödel and Łukasiewicz t-norms, in most cases, the Product t-norm brings negligible improvements, if any. Lastly, as expected, integrating background knowledge (via t-norm loss) in the neural models helps more when little data are available. Indeed, our models yield an improvement of up to 1.85% and 3.95% when using, 10% and 20% of the labelled training data, respectively.
The last observation led us to investigate
whether in our setting also holds the known re- Table 3: The best- and worst-performing models
sult that background knowledge helps when un- across fully-supervised models and models using
unlalabelled data are available (see, e.g., [
Neuro-symbolic works have proposed ways to embed available background knowledge by either
embedding it into the topology and/or into the loss. In the former category, we find works
that build a constrained layer on top of a neural network, such as Coherent-by-Construction
Network (CCN) [
The other main line of work comprises methods that relax the constraints and integrate
them into the neural networks’ loss, directly relating to our method. Early work on semantic
based regularisation (SBR) [24] on kernel machines led to the development of ways to map the
constraints into the neural networks’ loss according to the t-norm operations [
In this paper, we formalise an approach for computing a memory-eficient t-norm-based loss to equip neural networks with background knowledge logical constraints. We show that, unlike standard implementations of t-norm-based losses, our method can be applied in resourceintensive scenarios, such as event detection for autonomous driving. On the ROAD-R dataset, we test our t-norm-based loss on diferent amounts of labelled data showing that the t-norms indeed help in boosting the performance of state-of-the-art models, and we also present an efective way to use the t-norms in presence of unlabelled data.
For future work, we plan on conducting a study into how the loss can help in a semi-supervised scenario, with varying amounts of unlabelled data added during the training and using the warmup phase that we found helpful here for reducing the runtime strain brought by integrating background knowledge into neural networks. Another research direction could be a study on the efect of using diferent intervals of warm-up, before introducing the t-norm loss and applying it on the unlabelled data.
Mihaela C. Stoian is supported by the EPSRC under the grant EP/T517811/1. This work was also supported by the Alan Turing Institute under the EPSRC grant EP/N510129/1, by the AXA Research Fund, by the EPSRC grant EP/R013667/1, and by the EU TAILOR grant. We also acknowledge the use of the EPSRC-funded Tier 2 facility JADE (EP/P020275/1) and GPU computing support by Scan Computers International Ltd. [13] M. Diligenti, M. Gori, C. Sacca, Semantic-based regularization for learning and inference,
Art. Intell. 244 (2017). [14] E. Giunchiglia, M. C. Stoian, S. Khan, F. Cuzzolin, T. Lukasiewicz, ROAD-R: The autonomous driving dataset with logical requirements, Machine Learning (2023). [15] G. Metcalfe, Fundamentals of fuzzy logics, https://www.logic.at/tbilisi05/Metcalfenotes.pdf, 2005. [16] NVIDIA, NVIDIA AI A100, https://www.nvidia.com/en-gb/data-center/a100/, 2023. Accessed: 17/03/2023. [17] G. Singh, S. Akrigg, M. D. Maio, V. Fontana, R. J. Alitappeh, S. Khan, S. Saha, K. Jeddisaravi, F. Yousefi, J. Culley, T. Nicholson, J. Omokeowa, S. Grazioso, A. Bradley, G. D. Gironimo, F. Cuzzolin, ROAD: The Road Event Awareness Dataset for Autonomous Driving, IEEE TPAMI 45 (2023). [18] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proc. of
CVPR, 2016. [19] Y. Hua, Z. Zhao, Z. Liu, X. Chen, R. Li, H. Zhang, Trafic prediction based on random connectivity in deep learning with long short-term memory, in: Proc. of VTC-Fall, 2018. [20] R. Stewart, S. Ermon, Label-free supervision of neural networks with physics and domain knowledge, in: Proc. of AAAI, 2017. [21] N. Hoernle, R. Karampatsis, V. Belle, K. Gal, MultiplexNet: Towards fully satisfied logical constraints in neural networks, in: Proc. of AAAI, 2022. [22] P. Dragone, S. Teso, A. Passerini, Neuro-symbolic constraint programming for structured prediction, in: Proc. of IJCLR-NeSy, 2021. [23] A. Daniele, E. Krieken, L. Serafini, F. Harmelen, Refining neural network predictions using background knowledge, Machine Learning (2023). [24] M. Diligenti, M. Gori, M. Maggini, L. Rigutini, Bridging logic and kernel machines,
Machine Learning 86 (2012). [25] L. Serafini, A. d’Avila Garcez, Logic tensor networks: Deep learning and logical reasoning from data and knowledge, in: Proc. of NeSy-HLAI, 2016. [26] S. Badreddine, A. d’Avila Garcez, L. Serafini, M. Spranger, Logic tensor networks, Artif.
Intell. 303 (2022). [27] E. van Krieken, E. Acar, F. van Harmelen, Analyzing diferentiable fuzzy implications, in:
Proc. of KR, 2020. [28] E. van Krieken, E. Acar, F. van Harmelen, Analyzing diferentiable fuzzy logic operators,
Artif. Intell. 302 (2022). [29] K. Ahmed, E. Wang, K. Chang, G. V. den Broeck, Neuro-symbolic entropy regularization, in: Proc. of UAI, 2022. [30] Y. Grandvalet, Y. Bengio, Semi-supervised learning by entropy minimization, in: Proc. of
NeurIPS, 2004. [31] Z. Li, Z. Liu, Y. Yao, J. Xu, T. Chen, X. Ma, J. Lü, Learning with logical constraints but without shortcut satisfaction, in: Proc. of ICLR, 2023. [32] Z. Hu, X. Ma, Z. Liu, E. Hovy, E. Xing, Harnessing deep neural networks with logic rules, in: Proc. of ACL, 2016. [33] Z. Hu, Z. Yang, R. Salakhutdinov, E. Xing, Deep neural networks with massive learned knowledge, in: Proc. of EMNLP, 2016. [34] E. Giunchiglia, M. C. Stoian, T. Lukasiewicz, Deep Learning with Logical Constraints, in:
Proc. of IJCAI, 2022.
ID
Agent Descriptions in natural language
A motorbike cannot be a bus