An Adaptive Semantic Stream Reasoning Framework for Deep Neural Networks Danh Le-Phuoca , Thomas Eiterb a Technical University Berlin b Technical University Vienna Abstract Driven by deep neural networks (DNN), the recent development of computer vision makes visual sensors such as stereo cameras and Lidars ubiquitous in autonomous cars, robotics and traffic monitoring. However, due to operational constraints, a processing pipeline like object tracking has to hard-wire an engineered set of DNN models to a fixed processing logic. To overcome this, we propose a novel semantic reasoning approach that uses stream reasoning programs for in-cooperating DNN models with commonsense and domain knowledge using Answer Set Programming (ASP). This approach enables us to realize a reasoning framework that can adaptively reconfigure the reasoning plan in each execution step of incoming stream data. Keywords Semantic Reasoning, Neural-symoblic, Stream Reasoning, Stream Processing 1. Motivation ple, the recent report of Uber’s accident in Arizona [4] says "...The ADS detected the pedestrian 5.6 seconds be- The recent development of computer vision (CV) driven fore impact. Although the ADS continued to track the by deep neural networks (DNN) makes visual sensors pedestrian until the crash, it never accurately classified such as stereo cameras and Lidars ubiquitous in au- her as a pedestrian or predicted her path. By the time the tonomous cars, robotics and traffic monitoring. In par- ADS determined that a collision was imminent (1 second ticular, many DNN models for object detection [1] and before impact), the situation exceeded the response spec- tracking [2] are available. However making reliably ifications of the ADS braking system...". This accident working in a real-life online processing pipeline such could have not happened if the ADS could reconfig- as an Automated Driving System (ADS) or a traffic ure the object tracking pipeline on the fly, e.g chang- surveillance query engine is still very challenging. For ing DNN models or using alternative sensor sources to example, [2] reports that the most accurate DNN-driven improve the accuracy on detection and tracking. multi-object tracking (MOT) pipelines can process only This motivates us to propose an approach of com- 4-5 frames/second. To make such a system work on- bining stream reasoning with probabilistic inference line e.g. for ADS, where processing delay must be less to continuously configure such processing pipelines than 100ms [3], one has to hard-wire a fixed sets of based in semantic information representing common- DNN models with some sacrifices on accuracy and ro- sense and domain knowledge. The use of semantic in- bustness as the design constraints of an ADS limit how formation together with DNNs has proved to be useful much hardware can be put into a system [3]. For in- and led to better accuracy in image understanding [5] stance, an additional 400 W power consumption trans- and in object tracking [6]. Similar to ours, these ap- lates to a 3.23% reduction in miles per gallon for a 2017 proaches use declarative approaches to represent the Audi A4 sedan or similarly, the additional power con- processing pipelines of visual data. However, none of sumption will reduce the total driving range of electric them have considered how to deal with the aforemen- vehicles. tioned operational constraints in the context of stream Such a design-time trade-off often leads to unpre- processing. Our approach represents such constraints dictable fatal errors in a real deployment. For exam- in an extension of Answer Set Programming (ASP). This extension is proposed by leveraging LARS formu- Proceedings of the CIKM 2020 Workshops, October 19-20, Galway, las [7] for expressing stream reasoning programmes to Ireland. incorporate uncertainty of probabilistic inference op- email: danh.lephuoc@tu-berlin.de (D. Le-Phuoc); thomas.eiter@tuwien.ac.at (T. Eiter) erations under weighted rules similar to LP𝑀𝐿𝑁 [8], orcid: 0000-0003-2480-9261 (D. Le-Phuoc); 0000-0001-6003-6345 called semantic reasoning rules. As a result, we will (T. Eiter) be able to dynamically express a visual sensor fusion Β© 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). pipeline, e.g. MOT over multiple cameras, by seman- CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) tic reasoning rules to fuse probabilistic inference op- b 2 b 4 b 6 erations with ASP-based solving processes. Moreover, the expressive power of our approach also enables us b 1 b 3 b 5 to express operational constraints together with opti- det(yl,b ,car), trk(39,b ), … 1 det(yl,b ,car), trk(39,b ), 2 … 3 det(yl,b ,car), trk(41,b ), 4 … 5 6 misation goals as a probabilistic planning programme 1 2 3 t similar to 𝑝𝐡𝐢 + [9] so that our reasoning framework can reconfigure the reasoning plan adaptively in each Figure 1: A Semantic Visual Stream Snapshot execution step of incoming stream data. reads are missing), based on commonsense principles. 2. Formalization of Semantic We thus use the LARS framework [7] to represent a reasoning programme over our semantic stream data Stream Reasoning with DNN which can evaluated using an ASP solver. Models The LARS framework provides formulas with Boolean connectives and temporal operators @𝜏 𝛼, ░𝛼, and β—Šπ›Ό Semantically, a multi-sensor fusion data pipeline will to evaluate a formula 𝛼 at a time point 𝜏 (resp. ev- consume the data that is observed by a Sensor as a ery, some time point) in the current stream 𝑆; win- stream of observations (represented as an Observation) dow operators ⊞ 𝛼 take for evaluating 𝛼 a data snap- 𝑀 following standardized W3C/OGC Semantic Sensor Net- shot (substream) 𝑆 from 𝑆 by applying a function 𝑀 on β€² work Ontology (SSN) [10] . For instance, the example 𝑆. For example, the formula ⊞ β—Šπ‘‘π‘’π‘‘(𝑦𝑙, 𝐡, π‘π‘Žπ‘Ÿ) states +5 in Figure 1 shows 3 image frames are observed by a that at evaluation time 𝜏 , 𝑑𝑒𝑑(𝑦𝑙, 𝐡, π‘π‘Žπ‘Ÿ) holds at some traffic camera. These observations will then be fed into time point 𝜏 in the window [𝜏 βˆ’ 5, .., 𝜏 ] selected by β€² a probabilistic inference process such as a DNN model 𝑀 = +5; in our representation, 𝑑𝑒𝑑(𝑦𝑙, 𝐡, π‘π‘Žπ‘Ÿ) is the or a CV algorithm (represented as a Procedure) to pro- matching condition for "a car was detected in bound- vide derived stream elements which then are repre- ing box 𝐡 by the YOLO detector". The formula @𝜏 𝛼 senting Sampling instances. In this example, we have is aligned with a fluent 𝐹 resp. an event 𝐸 in Event 𝑑𝑒𝑑(𝑦𝑙, 𝑏1 , π‘π‘Žπ‘Ÿ) representing for the output bounding Calculus (EC) [12], such that @𝜏 𝐹 ≑ β„Žπ‘œπ‘™π‘‘π‘ π΄π‘‘(𝐹 , 𝜏 ) and box 𝑏1 from the YOLO detector π‘¦π‘™βˆˆπ·π‘‡ where 𝐷𝑇 (short @𝜏 𝐸 ≑ β„Žπ‘Žπ‘π‘π‘’π‘›π‘ (𝐸, 𝜏 ). This will help us to employ for Detector) represents for the set of detectors sup- ASP-based EC axioms for common sense reasoning rules ported. Similarly, π‘‘π‘Ÿπ‘˜(39, 𝑏2 ) represents for tracking as proposed in [6]. bounding box 𝑏2 generated by a tracking algorithm (a To deal with uncertainty, we extend LARS with weighted Tracker) which associates 𝑏2 with the tracklet 39 via rules under LP𝑀𝐿𝑁 semantics as in [8]. In LP𝑀𝐿𝑁 , the popular object tracking algorithm SORT [11]. facts generated from feature extractors (DNN models The symbol FeatureOfInterest (FoI) is used to repre- or OpenCV algorithms) as well as abduction rules on sent the domain of physical objects which are subjects top can be annotated with uncertainty information given for the sensor observations, e.g. tracking objects and by a weight, which allows reasoning under certain lev- field of views (FoV) of the camera. The relationship be- els of uncertainty. tween a Result generated by probabilistic inference al- For our concerns, a semantic reasoning program Ξ  is gorithms (e.g. YOLO detection model or Kalman filter a set of weighted rules π‘Ÿ of the form algorithm) to such object is represented by the predi- cate isSampleOf (denoted as iSO). As such algorithms πœ”βˆΆπ›Ό ←𝛽 (1) have output with uncertainty, we will use an abduc- where 𝛼, 𝛽 are LARS formulas and πœ” ∈ ℝ βˆͺ {π‘₯} is the tion reasoning process to search for explainable evi- weight of the rule. If πœ” = π‘₯, then π‘Ÿ is a hard rule, oth- dences for iSO via rules driven commonsense and do- erwise a soft rule; by Πℍ and Ξ π•Š we denote the sets main knowledge similar to [6]. of hard and soft rules of Ξ , respectively. The seman- To formalise the reasoning process with such a se- tics of Ξ  is given by the answer streams [7] 𝑆 of the mantic representation of stream data, we need a tem- LARS program Π𝑆 obtained from Ξ  by dropping the poral model that allows us to reason about the prop- weights and each π‘Ÿ where 𝑆 violates 𝛼 ← 𝛽; each erties and features of objects from streams of sensor such 𝑆 gets a probability π‘ƒπ‘ŸΞ  (𝑆) assigned calculated observations. This model must account for the laws of from the weights of the rules retained for Π𝑆 ; for more the physical world movement and in particular be able information, we refer to [8]. In Section 3 we will ad- to fill gaps of incomplete information (e.g., if we do dress how to translate restricted programs Ξ  into ASP not see objects appearing in observations, or camera programs to be fed into an ASP solver, and how the Reasonser solve weights πœ” can be learned from training data. genSnapshot Stream Processor To demonstrate how to build a semantic reasoning ASP Solver program, we will emulate DeepSORT tracking algo- KB rithm [13] via soft rules that can search for supporting lars2asp DNN models evidences to re-identify objects associated with track- Planner genPlanPf lets created by Kalman filter above, by using visual ap- solve pearance associations. DeepSORT extends SORT with a DNN that is trained to discriminate targeted objects (e.g. pedestrians or vehicles) on a labelled re-identificationFigure 2: Stream Reasoning Framework dataset. Hence, we will search for pairs of bounding boxes of 3. Dynamic Reasoning two similar tracklets w.r.t. visual appearance. Due to a large search space of possible matches, we will limit Framework the search space by filtering the candidates based on their temporal and spatial properties. Therefore, we To the realize our reasoning approach in Secion 2, we use rules with windows to reason about two discon- proposes a dynamic reasoning framework illustrated nected tracklets that have two bounding boxes matched in Figure 2. The key components Reasoner and Planner within a time window of 𝛿 𝑀 time points which are of the framework are built on top an ASP Solver and aligned with the DeepSORT’s gallery of associated ap- a Stream Processor which are pluggable and generic pearance descriptors for each track. Based on this gallery modules. The control logic of the framework is gov- of previous tracked boxes, the cosine distance is com- erned by Algorithm 1. puted to compare appearance information that are par- While any ASP Solver supporting weak constraints ticularly useful to recover identities after long term oc- can be used in our framework, existing stream proces- clusions, when motion is less discriminative. Hence, sors such as relational or graph stream processing en- for merging two adjacent tracklets that have visual ap- gines need to be extended with some prerequisite fea- pearance matches, we use the parametrized soft rule tures to connect with the rest of the framework. For (15) below. The pair of parameters (𝛿 , 𝑣𝑀 ) has to be 𝑀 𝑗 instance, in our under-development prototype, we ex- specified for each reasoning step via the probabilistic tend CQELS [14] to enable DNN inference on GPUs as planning component of our dynamic reasoning frame- built-in functions for CQELS-QL, the graph-based con- work in Section 3; 𝑣𝑀 is one of the available visual 𝑗 tinuous query language of CQELS. Via CQELS-QL, the matching models that represent the association met- auxiliary predicates such as π‘π‘œπ‘™, 𝑛𝑒π‘₯𝑑 and 𝑑𝑖𝑠𝑑 in Sec- rics to discriminate comparing bounding boxes. tion 2 are expressed as continuous queries in order to delegate the processing to the stream processor. This πœ”1𝑗 ∢ 𝑖𝑆𝑂(𝐡1 , 𝑂) ← @𝜏 π‘‘π‘Ÿπ‘˜(𝑇1 , 𝐡1 ), π‘π‘œπ‘™(𝐡1 , 𝑇2 ), π‘‘π‘Ÿπ‘˜π‘™π‘’π‘‘(𝑇2 , 𝑂), mechanism also helps us to avoid grounding overhead 𝑀 in continuous solving via materialised views similar to @πœπ‘’ 𝑒𝑛𝑑𝑠(𝑇2 ), @πœπ‘’ ⊞+𝛿 β—Šπ‘‘π‘Ÿπ‘˜(𝑇2 , 𝐡2 ), the over-grounding approach in [15]. In particular, we 𝑗 𝜏 < πœπ‘’ + 3, π‘£π‘€π‘Žπ‘‘π‘β„Ž(𝑣𝑀 , 𝐡1 , 𝐡2 ) (2) leverage continuous multiway-joins with windows of Similarly, we can define rules to trigger the object CQELS to delegate the processing of LARS formulas matching search based on visual appearances of two that do not occur in rule heads. tracklets from two adjacent cameras. We use Moreover, the visual stream data with different for- @𝜏 𝑑𝑖𝑠𝑑(𝐡1 , 𝐢, 𝑑 𝑑 ) to specify the time difference 𝑑 𝑑 from mats (e.g. RGB videos and Lidar PointCloud) together the candidate camera 𝐢 at time point 𝜏 to start the with the knowledge base (KB) have to be normalized search for the matches via the auxiliary predicate 𝑑𝑖𝑠𝑑. to the data model supported by the stream processor. Also, 𝐢 is filtered by the auxiliary predicate 𝑛𝑒π‘₯𝑑 stat- For example, ontologies and metadata and extracted ing 𝐢 is adjacent to the camera where 𝐡1 was gener- symbols as outputs of DNN inference processes are ated. 𝑙𝑒𝑓 𝑑𝐹 π‘œπ‘‰ (𝑂, 𝐢) represents for "object 𝑂 left the represented as temporal graphs of CQELS. With these FoV of camera 𝐢". features, the stream processor is able to generate data snapshots and planning profiles in ASP readable for- πœ”2𝑗 ∢ 𝑖𝑆𝑂(𝐡1 , 𝑂) ← @𝜏 π‘‘π‘Ÿπ‘˜(𝑇1 , 𝐡1 ), 𝑛𝑒π‘₯𝑑(𝐡1 , 𝐢), (3) mat via two methods 𝑔𝑒𝑛𝑃 π‘™π‘Žπ‘›π‘ƒπ‘“ and π‘”π‘’π‘›π‘†π‘›π‘Žπ‘π‘ β„Žπ‘œπ‘‘ re- spectively. 𝑀 @𝜏 𝑑𝑖𝑠𝑑(𝐡1 , 𝐢, 𝑑 𝑑 ), @𝜏 +𝑑 𝑑 ⊞+𝛿 β—Šπ‘™π‘’π‘“ 𝑑𝐹 π‘œπ‘‰ (𝑂, 𝐢), π‘‘π‘Ÿπ‘˜π‘™π‘’π‘‘(𝑇2 , 𝑂), π‘‘π‘Ÿπ‘˜(𝑇2 , 𝐡2 ), π‘£π‘€π‘Žπ‘‘π‘β„Ž(𝑣𝑀 𝑗 , 𝐡1 , 𝐡2 ) The Planner calls the method 𝑔𝑒𝑛𝑃 π‘™π‘Žπ‘›π‘ƒπ‘“ to prepare the input for the first reasoning step of each time point Algorithm 1 Semantic_Reasoning(S, 𝑑) LARS formulas to an ASP program using the method Input: Semantic Stream 𝑆, new observation 𝕆, time π‘™π‘Žπ‘Ÿπ‘ 2π‘Žπ‘ π‘, whose optimal models (answers sets) 𝐼 cor- 𝑝 point 𝑑 respond to the solutions of (5). Output: Optimal answer set 𝐼 βˆ— Μ‚ argmax π‘ƒπ‘ŸΞ Μ‚π‘ (Π𝐺 |𝑆, 𝜏 , Π𝑝 ) (5) 1: 𝑖 ← 0, Ξ π•Š ← βˆ… 𝐼 𝑝 βˆΆΞ Μ‚π‘ (𝜏 )⊩𝐼 𝑝 2: for {πœ” ∢ 𝛼 ← 𝛽} ∈ Ξ π•Š do 3: 𝑖 ←𝑖+1 With a chosen plan embedded in some 𝐼 𝑝 , the Reasoner 4: Ξ Μ‚π•Š ← Ξ Μ‚π•Š βˆͺ {π‘’π‘›π‘ π‘Žπ‘‘(𝑖) ← 𝛽, π‘›π‘œπ‘‘ 𝛼} calls the method π‘”π‘’π‘›π‘†π‘›π‘Žπ‘π‘ β„Žπ‘œπ‘‘ to generate the input 5: Ξ Μ‚π•Š ← Ξ Μ‚π•Š βˆͺ {𝛼 ← 𝛽, π‘›π‘œπ‘‘ π‘’π‘›π‘ π‘Žπ‘‘(𝑖)} data of the reasoning program for the next step in line (11) to carry out the second reasoning step from line 6: Ξ Μ‚π•Š ← Ξ Μ‚π•Š βˆͺ {∢∼ π‘’π‘›π‘ π‘Žπ‘‘(𝑖) [πœ”@0]} (12) to (14) to generate the output of the whole pipeline 7: end for Μ‚ as the optimal model 𝐼 βˆ— . To specify the weights of the 8: Π𝑝 ← 𝑔𝑒𝑛𝑃 π‘™π‘Žπ‘›π‘ƒπ‘“ (𝑆, 𝕆, Ξ π•Š , 𝑑 βˆ’ 1) Μ‚ soft rules, we use the weight learning approach of [17] 9: Π̂𝑝 ← π‘™π‘Žπ‘Ÿπ‘ 2π‘Žπ‘ π‘(Ξ π•Š βˆͺΠℍ βˆͺΠ𝑝 , 𝑑) which fits the weights using the training data via gra- 10: 𝐼 𝑝 ← Solve(Π̂𝑝 ) dient ascent. Training is done offline but uses Algo- 11: Π𝐷 ← π‘”π‘’π‘›π‘†π‘›π‘Žπ‘π‘ β„Žπ‘œπ‘‘(𝑆, 𝐼 𝑝 , 𝑑) rithm 1 to compute an optimal stable model in each 12: Ξ Μ‚ ← π‘™π‘Žπ‘Ÿπ‘ 2π‘Žπ‘ π‘(Ξ Μ‚π•Š βˆͺΠℍ βˆͺΠ𝐷 , 𝑑) step of updating weights in the gradient method. 13: 𝐼 βˆ— ← Solve(Ξ  Μ‚) 14: return 𝐼 βˆ— 4. Conclusion 𝜏 which finds the optimal reasoning plan to achieve a This position paper presented a novel semantic rea- certain goal Π𝐺 under an operational constraint Π𝑝 , soning approach that enables probabilistic planning following a probabilistic planning approach from [9] to adaptively optimize the sensor fusion pipeline un- in lines (8) to (10) of Algorithm 1. The reasoning prob- der operational constraints expressed in ASP. The ap- lem formalised in following is to find a configuration proach is realised with a dynamic reasoning mecha- (𝑑𝑑 𝑖 , 𝑣𝑀 𝑗 , 𝛿 𝑀 ) to result the highest probability of our nism that can integrate the uncertainy of DNN infer- tracking goal. For example, we can specify the goal of ence with semantic information, e.g common sense and being able to track the objects that were tracked in the domain knowledge in conjunction with runtime infor- previous time point 𝜏 βˆ’ 1 as mation as inputs for operational constraints. We are currently implementing an open sourced pro- Π𝐺 = β‹€π‘‚βˆˆ@𝜏 βˆ’1 π‘‘π‘Ÿπ‘˜π‘™π‘’π‘‘(𝑂,_) @𝜏 π‘‘π‘Ÿπ‘˜π‘™π‘’π‘‘(𝑂, _) totype of the proposed reasoning framework in Java Similarly, an operational constraint Π𝑝 can be expressed to exploit the code bases of CQELS and Ticker. We ASP rules. For instance, the example rule below rep- use the Java native interface to wrap C/C++ libaries of resents the constraint to limit executable plans Clingo 5.4.0 as the ASP Solver and NVidia CUDA 10.2 π‘π‘™π‘Žπ‘›(𝐷, 𝑉 , 𝛿) at time point 𝜏 based on the estimations as the DNN Inference engine. The solving and infer- the execution time of a candidate detection model 𝐷 ence tasks are coordinated in an asynchronous multi- and visual a matching model 𝑉 together a candidate threading fashion to exploit the massive parallel capa- window parameter 𝛿 𝑀 . The auxiliary predicates 𝑒𝑠𝑑 bilities of CPUs and GPUs. and 𝑛𝑂𝑏𝑗 provide the time estimation for correspond- ing DNN operations and the number of objects tracked in camera 𝐢. Acknowledgments This work was funded in part by the German Ministry ∢∼ @𝜏 𝑒𝑠𝑑(𝐷, 𝜏𝐷 ), @𝜏 𝑒𝑠𝑑(𝑉 , πœπ‘‰ ), @𝜏 π‘π‘™π‘Žπ‘›(𝐷, 𝑉 , 𝛿), (4) @𝜏 βˆ’1 𝑛𝑂𝑏𝑗(𝐢, 𝑁 ), 𝜏𝐷 + 𝑁 βˆ— 𝛿 𝑀 βˆ— πœπ‘‰ > π‘šπ‘Žπ‘₯𝑇 π‘–π‘šπ‘’ for Education and Research as BIFOLD - Berlin Insti- tute for the Foundations of Learning and Data (refs From Ξ  , lines (8) to (10) carry out solving the reason- 01IS18025A and 01IS18037A) and supported by the Aus- 𝑝 ing problem formalised by formula (4). To generate the trian Federal Ministry of Transport, Innovation and LARS program from the soft rules Ξ π•Š , the algorithm Technology (BMVIT) under the program ICT of the rewrites Ξ π•Š into LARS formulas with weak constraints Future (FFG-PNr.: 861263, project DynaCon). as Ξ Μ‚π•Š in lines (1) to (7) by extending the similar algo- rithm for LP𝑀𝐿𝑁 in [8]. Then, we use the incremental ASP encoding algorithm of Ticker [16] for rewriting References [13] N. Wojke, A. Bewley, D. Paulus, Simple online and realtime tracking with a deep association [1] L. Liu, W. Ouyang, X. Wang, P. Fieguth, J. Chen, metric, in: ICIP, 2017. X. Liu, M. PietikΓ€inen, Deep learning for generic [14] D. Le-Phuoc, M. Dao-Tran, J. X. Parreira, object detection: A survey, IJCV (2019). M. Hauswirth, A native and adaptive approach [2] G. Ciaparrone, F. L. SΓ‘nchez, S. Tabik, L. Troiano, for unified processing of linked streams and R. Tagliaferri, F. Herrera, Deep learning in video linked data, in: ISWC, 2011, pp. 370–388. multi-object tracking: A survey, Neurocom- [15] F. Calimeri, G. Ianni, F. Pacenza, S. Perri, J. Zan- puting (2019). URL: http://www.sciencedirect. gari, Incremental answer set programming with com/science/article/pii/S0925231219315966. overgrounding, TPLP 19 (2019). doi:https://doi.org/10.1016/j.neucom. [16] H. Beck, T. Eiter, C. Folie, Ticker: 2019.11.023. A system for incremental asp-based [3] S.-C. Lin, Y. Zhang, C.-H. Hsu, M. Skach, M. E. stream reasoning, TPLP (2017). URL: Haque, L. Tang, J. Mars, The architectural impli- https://doi.org/10.1017/S1471068417000370. cations of autonomous driving: Constraints and doi:10.1017/S1471068417000370. acceleration, in: ASPLOS ’18, 2018. [17] J. Lee, Y. Wang, Weight learning in a probabilistic [4] NTSB, Collision between vehicle controlled extension of answer set programs, in: KR, 2018, by developmental automated driving sys- pp. 22–31. tem and pedestrian in Tempe, Arizona, https://www.ntsb.gov/news/events/Documents/ 2019-HWY18MH010-BMG-abstract.pdf, 2019. Accessed: 2020-01-15. [5] S. Aditya, Y. Yang, C. Baral, Integrating knowl- edge and reasoning in image understanding, in: IJCAI, 2019. URL: https://doi.org/10.24963/ijcai. 2019/873. doi:10.24963/ijcai.2019/873. [6] J. Suchan, M. Bhatt, S. Varadarajan, Out of sight but not out of mind: An answer set pro- gramming based online abduction framework for visual sensemaking in autonomous driv- ing, in: IJCAI-19, 2019. URL: https://doi.org/ 10.24963/ijcai.2019/260. doi:10.24963/ijcai. 2019/260. [7] H. Beck, M. Dao-Tran, T. Eiter, LARS: A logic-based framework for analytic reasoning over streams, Artif. Intell. 261 (2018) 16–70. URL: https://doi.org/10.1016/j.artint.2018.04.003. doi:10.1016/j.artint.2018.04.003. [8] J. Lee, Z. Yang, LPMLN, Weak Constraints, and P-log, in: AAAI, 2017. [9] J. Lee, Y. Wang, A probabilistic extension of ac- tion language BC+ , TPLP 18 (2018) 607–622. URL: https://doi.org/10.1017/S1471068418000303. doi:10.1017/S1471068418000303. [10] K. Janowicz, A. Haller, S. J. D. Cox, D. L. Phuoc, M. LefranΓ§ois, SOSA: A lightweight ontology for sensors, observations, samples, and actuators, J. Web Semant. 56 (2019) 1–10. [11] A. Bewley, Z. Ge, L. Ott, F. Ramos, B. Upcroft, Simple online and realtime tracking, in: ICIP, 2016, pp. 3464–3468. [12] E. T. Mueller, Commonsense Reasoning: An Event Calculus Based Approach, 2 ed., 2015.