Context Aware Software Stacks for Mobility Composive.ai Overview Naci M. Dai Deniz Memis Burak Saglam Eteration Eteration Eteration Istanbul, Turkey Istanbul, Turkey Istanbul, Turkey naci.dai@eteration.com deniz.memis@eteration.com burak.saglam@eteration.com Abstract—Humans are very good at adapting to and driving in There are still many challenges to this learning process as very different conditions and environments. Complex AI based huge learning data leads to many difficult to explain biases software such as Autonomous vehicles (AV) are biased by the and edge cases where the algorithms come up with unexpected training data and they do not do well in different contexts. Environment perception and context awareness can be used to results. Deep neural networks learn from the data to cleanly adapt and deploy complex networks of AI systems that are separate the dataspace for the intrinsic dimensionality (nicely expected to perform safely on many different environments and demonstrated by Colah’s Neural Networks, Manifolds and edge cases. Typical Autonomous Drive (AD) software stacks Topology [1]), and require careful engineering and modelling provide localization, object detection and tracking using an to do this efficiently. These systems have very little or no array of sensor data. Additional sources of information such as V2X connectivity, passenger behavior, person-to-device mapping, contextual capability, or ability to abstract knowledge or urban context, degradation of traffic etc. complements the sensor reason about the drive [2]. DARPA recently announced the data that can be used to describe a context. This context is used Next AI challenge, and is focusing its investments on a third to predict and adapt the system of software and applications that wave of AI that brings forth machines that understand and are deployed to the vehicle. reason in context [3] (See Fig.1.). Context Aware Software Stack for Mobility (CASSM) de- The Contextual Adaptation means the systems construct scribes a model based approach for definition of a software stack for complex systems such as those found in autonomous vehicles. explanatory models for classes of real world phenomena and Machine Learning is used to detect a context and adapt the the explanations and the context can be used to adapt both software stack, which can then be used to identify and deploy the learning models and in our case the overall AD software services to autonomous vehicles. CASSM will be an open source stack. in-vehicle delivery of context-driven services and applications. This paper describes an architecture to define context-aware personalized & dynamically provisioned capabilities, applications & services to autonomous vehicles. Index Terms—Autonomous Vehicles, Context Awareness, Adaptive Software Stacks, Machine Learning I. I NTRODUCTION Experienced human drivers are good at adapting to different driving conditions and environments. They can instinctively transfer their earlier learning to the current driving context. The advent of big data and availability of computing allowed statistical deep learning methods to power innovation in many applications including AD. AD software are complex systems Fig. 1. DARPA - The Next Wave of AI (Figure Adapted from John of various interconnected AI and deterministic modules. These Launchbury) software do very well in the context that they are designed and trained for, but perform poorly in other environments. Humans are very successful at understanding the context Starting with Darpa challenges, statistical learning methods such as parking, urban driving, or driving fast on a highway, have been used in implementing very successful AD software. abstracting experiences and transferring ideas and reacting The quality of these solutions depend on their statistical appropriately. However, for AD software the design, capabili- models and the big data that is used to train them. These ties, sensors and actions taken for these contexts can differ systems are very good at learning, perceiving and adapting significantly. Even urban driving in a well organized city to the environment. These cars can successfully, and safely versus a city with narrow streets (i.e. Istanbul) will require drive on highways and in the cities, interacting with the training with completely different datasets, probably using environment, the traffic, other cars and pedestrians. different models. Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). We can define the context as the set of all the inputs from the C. Context Awareness sensors, and additional inputs from the environment, people’s Numerous context-aware systems have been presented in behavior, even if they are not used in the AD stack. The action the literature, they albeit differ in the way context related of driving takes place and all these inputs interact with the data is acquired and utilized; the main commonalities of these system and change its behavior. It involves the driver, the systems’ structure could be listed as below: vehicle physics, its parts, brakes, steering, mechanical com- • Acquiring information. ponents, the weather condition, the destination etc. Context- • Manipulating the information (filtering, compressing, awareness means that a service, although given the same transforming) to contextually represent the system. request parameters, is perceived differently with respect to a • Interpreting the context. given context [4]. In autonomous driving context-awareness • Distributing information based on the contextual interpre- means, (1) Semantics and modelling the context for which tation. AD software is trained and designed for and (2) Using the information about the context to adapt how AD perceives, In the AV domain, these systems have been utilized in a predicts, and acts. variety of applications such as ”route planning and learn- ing [7]–[9], real time perception of traffic congestion [10], II. C ONTEXTUAL AI AND C ONTEXT AWARENESS FOR collision avoidance [11] and finding vacant car parks [12]”. AUTONOMOUS V EHICLES Considering these use cases that provide a service to the user A. Contextual AI and above mentioned system characteristics there are two key Contextual AI is focused on providing a service or infor- aspects relevant to context-awareness: First is the knowledge mation to the user that is relevant to the acquired (explicit acquired about the context of the service to be provided and or implicit) context information [5]. This approach allows the second is to adjust the service to be contextually aware contextually-aware systems to provide tailor-made services considering the information within the context. and solutions under varying circumstances. There are mainly The first absolute benefit of context-awareness for AV, as four pillars to Contextual AI, based on these the system must is true for all autonomous systems, is adaptation [13]. Adap- be [5]: tation implies modifying a service that will, given the same • Intelligible: Ability to explain what it knows, how it parameters, yield a relevant distinguished output depending knows and what it is doing. on the context and the objective. This may include different • Adaptive: Ability to meet user’s needs and expectations realizations such as refining or tracking other information, under varying environmental circumstances. invoking or inhibiting other services. The other and relatively • Customizable: Ability to be controlled fully by the user. more sophisticated benefit compared to other autonomous • Context-aware: Ability to perceive at a human level. systems is proactivity, currently provided reactive services work upon the invocation of the user (a pull type service), on Even for the most successful AD software, its capabilities the other hand proactive services could deliver the information are still bounded by its compatibility to the current context without any explicit action required by the user. Furthermore, and the constraints it was trained for, making such systems these proactive services could keep delivering based on the heavily dependent on human intervention when it is faced with predicted outcome of future context [13]. individual cases where the inferences and plan fail. B. Explainable AI III. T HE M ODEL D RIVEN A DAPTIVE S OFTWARE S TACK Like all other safety and sensitive applications of AI, Simon described means-end analysis with the notion of Explainability of AD software is strongly related to context- adaptation, which is finding the difference between an existing awareness. AD software travelled successfully for billions of state and a desired state and then finding the correlating kilometers but when they fail, the reasons are not easily inter- process that will erase the difference [14]. pretable by a human. Interpreting and explaining is important Our work focuses on software systems that are amenable to gain trust into the prediction, and to identify potential data to model-driven methods. It is possible to dynamically adapt selection biases or artefacts. We need to know when an AD these models to better address a given context. A large system is operating within the limits of its training. These percentage of AD software is based on the Robot Operating limits identify contexts that the AD is designed to safely and System (ROS). ROS has the concept of computational nodes reliably operate, and prevent dangerous inferences and outputs. that can be connected with the data flowing between them. For example, if the perception system cannot classify the intent Many AD implementations involve sensor data, and higher of a cyclist, being able to explain why the input data led level objects that are produced by sensor-fusion, prediction, to an incorrect calculation will help in the adaptation of the planning and control tasks that are implemented with ROS. underlying learning models. The explanations could be related This is typically done with a code-first, development heavy to the structure or models of the system, of single decisions or practice. cases, or generalize to the complete overall behavior. Samek et. RobMoSys attempts to address this difficulty by providing al [6]. provide an extensive review of methods and applications standards for composable models and Software for Robotics of some of the most recent advances in AI explanations. Systems. We are exploring RobMoSys model-driven approach to define models for the proven ROS based open-source typical software stack like the ones used by Autoware (See AD frameworks such as Autoware [15], Baidu Apollo [16]. Fig.4.). Refactoring existing systems as model-based software will provide fast and effective feedback for the functionality and also will prove the platform independence. The metamodel- driven approach, that composes the building blocks (or a combination of them) into various systems in a purposeful way, the context detection models and algorithms (ROS nodes) will be in use. According to context, these nodes will be used to create a context-level metamodel and will enable other nodes to be deployed and to be used. The contextual adaptation of an AD software stack would change the active stack to better address the constraints and trade-offs of the context. Each stack model is mapped to a context that reflects the conditions of its training data. It is Fig. 4. Autoware AD Software Stack. conceivable that one can define an “uber” model that would satisfy most contexts and there will be no need for adaptation, In their work, they improve inferences by using the notions but this is probably not achievable with today’s understanding. of adapting the composer model and policy preferences. Al- Instead, we propose a pragmatic context-based adaptation ternatively, MROS [18] leverages the RobMoSys [19] model- approach that will identify a context based on the inputs of the based approach at runtime, to provide a solution for ROS2 system and dynamically change the software stack (a model- systems, based on architectural self-adaptation driven by on- based computation graph) to satisfy the requirements in the tology reasoning on the architecture models. new context (See Fig.2. & Fig.3.). Of course this approach is not limited to AD. The ap- plications in the mobility space exist in many different do- mains. Connected cars and smart cities make use of cloud based services and innovative human computer interactions. The same context-based adaptation techniques can be applied to model-based software applications in these domains. For example, the service orchestrations based on BPMN-like flows or component-based user interfaces that are amenable to view- flow like structures would also benefit from this solution. Composive.ai, will deliver our components and tools for building adaptive software stacks as open-source software (See Fig. 2. Metamodel extraction from existing systems. Fig.5.). Fig. 5. Composiv.ai Adaptive Software Stacks. Fig. 3. Adaptation using metamodel-driven feedback. Similar approaches produced promising results for other C ONCLUSION problems. Odena et.al [17] use reinforcement learning to State of the art AD software makes extensive use of statis- change model behavior at test-time. They use a composer tical learning methods with proven results. These applications model, which consists of a set of modules and a controller are typically limited by the context, size and quality of the network. The composer network is a graph of deterministic and datasets that were used to train them. Contextual adaptivity stochastic nodes made of neural networks that are organized and reasoning is the natural next step for AI methods to get into ‘metalayers’. The composer network is very similar to the closer to human like capabilities. In our project, Composive - Context Aware Software Stacks for Mobility, is a solution for adapting model-driven complex autonomous systems (such as AD). Composive will solve the adaptation problem by being able to define a context and detecting the variations within the defined context from a repository of pre-existing models using a metamodel-driven approach, subsequently it will adapt and deploy other context-driven services that are compatible with the future contextual model. IV. ACKNOWLEDGMENTS This work is being partially funded by The Scientific and Technological Research Council of Turkey, TEYDEB 1509 grant. R EFERENCES [1] http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ [2] Launcbury, John, A DARPA Perspective on Artificial Intelligence, 2017 https://youtu.be/-O01G3tSYpU [3] AI Next Campaign, https://www.darpa.mil/work-with-us/ai-next- campaign [4] Pichler, Mario et al. ”Context-awareness and artificial intelligence”. ÖGAI Journal 23. (2004): 4-. [5] Contextual AI: The Next Frontier of Artificial Intelligence,https://blog.adobe.com/en/2019/04/09/contextual-ai-the- next-frontier-of-artificial-intelligence.htmlgs.ytw2bv [6] W Samek, G Montavon, S Lapuschkin, C Anders, KR Müller, Explain- ing Deep Neural Networks and Beyond: A Review of Methods and Applications Proceedings of the IEEE, 109(3):247-278, 2021 [7] M. Younes, A. Boukerche, and G. Rom’an-Alonso, “An intelligent path recommendation protocol (ICOD) for VANETs,” Comput. Netw., vol. 64, pp. 225–242, May 2014. [8] M. B. Younes and A. Boukerche, “A performance evaluation of a con- textaware path recommendation protocol for vehicular ad-hoc networks,” in Proc. IEEE Global Telecommun. Conf. (GLOBECOM), Dec. 2013, [9] M. Masikos, K. Demestichas, E. Adamopoulou, and M. Theologou, “Energy-efficient routing based on vehicular consumption predictions of a mesoscopic learning model,” Appl. Soft Comput., vol. 28, pp. 114–124, Mar. 2015. [10] J. Liu et al., “High-efficiency urban-traffic management in context-aware computing and 5G communication,” IEEE Commun. Mag., vol. 55, no. 1, pp. 34–40, Jan. 2017. [11] B. Kihei, J. A. Copeland, and Y. Chang, “Automotive Doppler sensing: The Doppler profile with machine learning in vehicle-to-vehicle net- works for road safety,” in Proc. IEEE Workshop Signal Process. Adv. Wireless Commun. (SPAWC), Jul. 2017, pp. 1–5. [12] A. Alhammad, F. Siewe, and A. Al-Bayatti, “An infostation-based context-aware on-street parking system,” in Proc. Int. Conf. Comput. Syst. Ind. Inform. (ICCSII), Dec. 2012, pp. 1–6. [13] Pichler, Mario Bodenhofer, Ulrich Schwinger, W.. (2014). Context- awareness and artificial intelligence. ÖGAI Journal. 23. 4-. [14] Sciences of the Artificial, Herbert Simon, 1996, MIT Press [15] Autoware Foundation - https://www.autoware.org/ [16] Apollo Auto - https://apollo.auto/ [17] Augustus Odena, Dieterich Lawson, Christopher Olah - Changing Model Behavior at Test-Time using Reinforcement Learning, Workshop track - ICLR 2017 https://arxiv.org/pdf/1702.07780.pdf [18] https://robmosys.eu/mros/ [19] https://robmosys.eu Composable Models and Software for Robotics Systems