Towards AI-based Solutions in the System Development Lifecycle Stephan Jüngling, Martin Peraic, Andreas Martin FHNW University of Applied Sciences and Arts Northwestern Switzerland, School of Business Peter Merian-Strasse 86, 4052 Basel, Switzerland stephan.juengling@fhnw.ch, martin.peraic@bvb.ch, andreas.martin@fhnw.ch Abstract trained models can be re-used as modules, and with the help Many teams across different industries and organizations of transfer learning be adapted to a variety of similar explicitly apply agile methodologies such as Scrum in their applications, AI components will slowly be deployed in system development lifecycle (SDLC). The choice of the many more business cases. In the case of semiconductor technology stack, the programming language, or the decision manufacturing, it was demonstrated that a portable image whether AI solutions could be incorporated into the system design either is given by corporate guidelines or is chosen by classifier could be embedded in offline edge devices to the project team based on their individual skill set. The paper detect defects on laser chips with an accuracy of 97% (Hou, describes the business case of implementing an AI-based Liu, Pan, & Hou, 2019). However, what would be the impact automatic passenger counting system for public on the software development process within companies, transportation, shows preliminary results of the prototype given the fact that AI will be increasingly used as part of using anonymous passenger recognition on the edge with the help of Google Coral devices. It shows how different their IT systems, products, and solutions? solutions could be integrated with the help of rule base systems and how AI-based solutions could be established in the SDLC as valid and cost-saving alternatives to Application Domains for AI-Based solutions traditionally programmed software components. Smith and Eckroth (2017) provide a comprehensive insight into lessons learned from building AI applications during Introduction the last three decades. In one of their key insights, they mention that the ease of use delivered by the human AI in general and deep learning, in particular, are amongst interface is the “license to operate”. This statement, which the current hot topics in computer science research, and has been made focusing on the client and user perspective, many universities create new bachelor and master programs most probably will hold true during the entire software in data science. The hype is also visualized by the Gartner development lifecycle (SDLC). The “new” discipline of hype cycle for AI (2019), where edge AI, deep neuronal data scientists needs to be incorporated into the SDLC. networks, and machine learning are at the peak of inflected Traditional skills of people that are involved in software expectations. Nevertheless, AI technologies are already design need to be extended with AI topics such as machine used and well established in a wide area of different business learning and knowledge engineering. domains. However, most of the time, these are isolated and In many practical situations, the choice of the best suitable specialized applications, where they are clearly the sole system development methodology is actively discussed and possible IT solution, e.g., machine translation or image explicitly decided at the project start. Arguments for processing for cancer detection. In such cases, ML different methodologies such as Scrum, Kanban, SAFE, or capabilities are mostly predominantly compared to human even traditional water methods are sought, and different skills. There are not yet many situations where AI-based suitable tools are evaluated and selected. However, all these components are compared to traditional software or methods still take the predominant traditional design and hardware components. But with the current state of the art implementation processes of software components into neuronal network models and new emerging infrastructure account. Also, with agile methods, some sort of around TensorFlow (Tensorflow Hub, n.d.), where pre- Copyright © 2020 held by the author(s). In A. Martin, K. Hinkelmann, H.- Stanford University, Palo Alto, California, USA, March 23-25, 2020. Use G. Fill, A. Gerber, D. Lenat, R. Stolle, F. van Harmelen (Eds.), Pro- permitted under Creative Commons License Attribution 4.0 International ceedings of the AAAI 2020 Spring Symposium on Combining Machine (CC BY 4.0). Learning and Knowledge Engineering in Practice (AAAI-MAKE 2020). requirements engineering, be it in the form of use cases or As such, existing systems do not provide real-time data, and user stories, is conducted. Maybe it is followed by test- the measurements must first be collected and processed for driven feature implementation using continuous integration subsequent use. In addition to the high acquisition costs, this as a core idea, and customers can, at any time, verify the also results in ongoing costs for storing and processing the current implementation state of the application and provide recorded data, which is costly, inflexible, and no longer immediate feedback. Given the availability of state-of-the- appropriate. New methods for passenger counting must be art AI-based video analysis algorithms and edge computing studied. In particular, it would be a good idea if existing capabilities, the idea of building an Al-based passenger systems in the vehicles, such as the cameras installed for counting system for public transit was reasonable and could security purposes, could be re-used. It is therefore suitable be conducted during a bachelor thesis in cooperation with to apply the latest achievements in the field of AI-based Basler Verkehrs-Betriebe, a leading public transport object recognition. Such an APC method could re-use provider in the area of Basel (Peraic, 2019). existing systems, collect data in real-time, and directly process the data without further manual operations. In order to demonstrate the feasibility of an AI-based counting Business Case of an AI-based Passenger system and to describe first benefits and limitations, the Counting System for Public Transit following research questions have to be answered: With almost 29000 kilometers, Switzerland has one of the • RQ1: Is AI-based object detection competitive against densest public transport networks in Europe (VÖV, 2017). traditional infrared-based measurement systems under realistic conditions? This public service is offered by numerous private transport companies throughout Switzerland. In order to make it • RQ2: Is it possible to perform offline object recognition easier for customers to use these services, a variety of on edge devices from a legal perspective? regional and national fare associations exist in different • RQ3: Can existing vehicle systems, such as surveillance geographical areas. They ensure that public transport cameras, be re-used? services can be accessed with a single subscription across • RQ4: What are the benefits of an AI-based automated Switzerland. Consequently, all public transport operators passenger counting system? are obliged to record passenger data of onboarding and alighting passengers. Based on these figures, the percentage Implemented Prototype of the subscription income of the respective fare association is calculated for the participating companies. The Basler Verkehrs-Betriebe strives to continuously Automatic Passenger Counting Systems (APC), improve its core business, the transport of passengers, in especially developed for public transport, are applied to order to offer the best possible service to its 350,000 daily collect such data. Since the early 1970s, various customers. More and more processes are being enhanced or manufacturers have been offering solutions for passenger even replaced by modern software solutions. The need for counting (Siebert & Ellenberger, 2019), and many different tailor-made and in-house developed applications is approaches and measuring methods are applied. Some increasing and consequently the demand for optimized operators rely on the measurement of boarding and alighting conditions for the development of software. Currently, this motion with the help of light barrier sensors in the infrared new corporate focus is taking place, thus no company-wide spectrum, and some systems measure weight changes in the SDLC for development projects could be applied for the boarding area based on associated spring movements of the prototype. This led to challenges before and during the vehicle suspension. All of these “classic” methods are project. Development and test environments first had to be complex systems that must be installed at every door of the created from scratch. Alongside development, the Hermes vehicle. This requires additional hardware components, project methodology practiced by public companies in which are expensive to purchase or to upgrade. This poses Switzerland could not be applied, as this methodology financial problems, especially for smaller public transport follows little to no agile approaches. However, an agile operators. Such companies use manual counting methods approach was crucial for the development of an AI-based instead, which are frequently based on insufficient customer APC. Without a flexible delimitation of the objectives, the surveys regarding driving behavior and do not provide continuous testing of the prototype versions under real reliable figures (Siebert & Ellenberger, 2019). conditions and the subsequent optimizations, a runnable Previous classic APC models are rarely deployed fleet- prototype could not have been implemented in such a short wide due to high costs. Instead, the systems are distributed timeframe. across all existing routes and randomly measure the number The AI-based APC was realized with Google Coral devices of passengers, which are later extrapolated to estimate the (Google Coral, n.d.), which are optimized for TensorFlow total number of passengers in the entire transport network. Lite (TFL, n.d.) AI models such as the mobile Single Shot Detector (SSD). The TFL mobile SSD object recognition is source for the AI-based prototype. Subsequently, the supported by the correlation tracking provided by DLIB measurements could be compared and evaluated using the (Rosebrock, n.d.). With this combination, the SSD model timestamps of both measurement systems. The AI-based recognizes all objects (position coordinates) and their object prototype correctly determined all passengers in 54% of the class (human, bicycle, dog, etc.) in predefined frame cases. The traditional infrared APC solution achieved a intervals. Subsequently, position and class type are stored result of 72% correct measurements. temporarily and are further tracked during the consecutive frames by the less computationally intensive correlation tracker. Thereby the Coral Edge device can be relieved by the reduced utilization of the object recognition model to keep resources free for additional operations. That capacity is used for the ID-based tracking of captured and stored objects. For this purpose, Centroid Tracking is being used. A primitive yet efficient tracking algorithm. This feature provides the ability to observe objects across multiple frames and measure motion into or out of the vehicle. The setup is specially designed for offline operations to Figure 2 - Arithmetic Mean of Success in Relation to People enable onboard computation. It allows bypassing possible involved (AI System) future data protection obstacles of object recognition in public space by avoiding the transmission of sensitive data RQ1: First results measured under real conditions to a remotely located datacenter. demonstrate convincingly that with a small number of passengers, the AI-based APC can compete with the infrared system in terms of accuracy. Figure 2 reveals how only a few counting errors could be observed during experiments with one or two passengers. As complexity increases, so does the error rate. The object tracking mostly causes this increase. With a growing number of passengers or increasing complexity of movements, the basic Centroid Tracker can hardly distinguish between objects. They are either swapped or mistakenly re-detected, causing erroneous measurements. Conversely, the infrared system proved to be acceptable for multiple passengers, as shown in Figure 3. Figure 1 - Displayed Detection and Tracking Information After the development of the prototype, a first comparison of the counting accuracy between the already deployed APC, the infrared measurement, and the newly developed AI-based system was conducted. In this respect, real test scenarios from day-to-day life were executed under laboratory conditions. The number of passengers and their measurement complexity was continuously increased. Extreme cases, such as simultaneous and dense boarding of a larger crowd, were considered. These test scenarios were Figure 3 - Arithmetic Mean of Success in Relation to People created by observations from past experiments and real-life involved (IR System) situations. This combination of knowledge engineering for the optimization of the measurement scenarios proved to be Further erroneous measurements are attributed to the AI- a great advantage for the development of realistic and based object recognition. In realistic circumstances, object meaningful test cases. recognition can be misled by background noise or In order to obtain reliable results and to exclude any disturbance of passing objects. Since for performance phenomena, each test case was performed three times. All reasons the AI-based recognition is not applied at every measurements were recorded with a webcam directed at the frame, objects can pass the image without being detected. door of the vehicle. This video feed was as well as the data Possibilities for improving the AI-based APC have already been explored. Improved tracking algorithms like Deep performed passenger counting. Significantly fewer SORT (Wojke, Bewley, & Paulus, 2017) or the utilization hardware components must be installed for the AI-based of recent RGB-video or Depth-video approaches could APC. This results in further cost savings in the procurement provide the required performance improvement. In and maintenance of the system. Ultimately, edge devices particular, the use of depth sensors, as applied by Sun et al. can be continuously updated with new and enhanced AI (2019), theoretically outperform infrared sensors regarding models. Other applications such as the automatic detection accuracy. However, when facing problems with ML of dangers in the vehicle such as violence or emergencies methods, this could be the point to improve the results by are just a few examples that support a switch to this new combination with rule-based systems as described in the technology. section about combining machine learning with knowledge engineering. SDLC with Embedded AI-Based Solutions RQ2: The legal use of high-resolution video cameras as a Given the assumption of continuous improvements in the sensor has already been discussed for autonomous driving. area of object detection with low-cost edge devices, the In particular, the storage of video data was defined as a initial prototype could be improved within the next couple breach of data protection (Kunnert, 2017) if the image data of months, but there will be additional issues, which must be is stored on the device or an external database (e.g., cloud considered for the organization. storage). The AI-based prototype is therefore designed to According to a recent survey in Japan (Ishikawa & process the image data in real-time – storage-free. Yoshioka, 2019) software engineering professionals report Anonymous counting results could subsequently be sent to difficulties in system engineering when ML-based service providers for further processing. Such an components are incorporated in the engineering process. architecture requires further security considerations. It is Many of the existing principles and best practices need to be therefore essential to prevent third parties from gaining enlarged with additional domain knowledge on how access to the edge devices. These could collect sensitive machine learning and knowledge engineering can be image data or manipulate counting results, which would incorporated into the software development lifecycle of cause manipulation of the entire service (Zhang, Chen, companies. Zhao, Cheng, & Hu, 2018). Worth mentioning is also the performance difference when RQ3: Existing security cameras can theoretically be SDLC methodologies known from software development leveraged for AI-based passenger counting. Both the are applied in data science projects, where little standardized resolution and the quality of the images are sufficient for process methodologies exist. By comparing the efficiency object recognition. Only the position and the associated of different data science student teams working with viewing angle of the cameras are decisive for re-use. New different SDLC methodologies, Saltz, Shamshurin, & vehicle procurements, though, could take this into account Crowston (2017) reported an improved performance and without additional efforts and costs. For already procured efficiency of CRISP-DM and agile Kanban, while agile vehicles, the camera would have to be repositioned in the Scrum was even less efficient than using no methodology at event of misplacement. all. RQ4: The edge device architecture offers the ability to Further differences need to be considered when existing transmit real-time count data. Furthermore, the data could code or models will be re-used. While object-oriented be automatically merged and analyzed, allowing real-time programming existing functionality can be extended using measurement of passenger flows using the fleet-wide well-known inheritance mechanisms, new concepts such as deployment of the AI-based APCs. This information transfer learning need to be researched in more depth-first. provides the control center or emergency services, with Adding additional functionality would not only mean valuable information in a variety of situations. For example, adding additional code but also retraining some of the in the case of a traffic accident, the number of people in the models and redeploy it to the edge devices. vehicle can be evaluated immediately and required Besides the additional skills required for the design and emergency resources can be notified accordingly. On the implementation of hybrid solutions, characteristics of AI- other hand, conventional APC systems are usually entirely based solutions will most probably also have an impact on isolated and require complex and sometimes labor-intensive the skill sets in the field of requirements engineering as well processes to evaluate the data. Sensor data must be as testing. As stated by Jüngling & Hofer (2019), AI converted into human-readable datasets and merged by components could even become active parts of business skilled professionals. By eliminating these time-consuming scenarios and represented as actors in UML use case and resource-intensive tasks, major financial savings can be diagrams. achieved. Re-using already installed hardware such as surveillance cameras is an additional advantage of AI- Overall, companies that have internal software development bases about the most successful transfer learning skills will need either additional staff with appropriate modifications of re-usable pre-trained modules. Such experience in implementing AI technologies or educate knowledge bases could be established not only for the given internal employees and enable them to gain experience in case of passenger counting. Such a concept could generally order to establish new best practices for SDLC with hybrid be applied to different business application scenarios. solutions. With more experience from practical situations, all different disciplines, from requirements engineering to testing and deployment will have to develop new insights Conclusion and Outlook into how these two different kinds of systems engineering Although the given potential of AI in general and image can be combined. It could well be, that it is best to split up recognition and object tracking in particular, practical AI- the entire development into two parallel or serial sub- based business applications are still rare. Openness for new projects, one for the development of the AI components, the solutions and gaining practical experience within companies other for the traditional software components, where the will be key. Given the different nature of both ML and KE SDLC of both components can be decoupled, having solutions compared to traditional SDLC, it seems important different life cycles, independent regression tests, automated for companies to develop and deploy prototypes of AI-based build pipelines and integration-tests in the end. solutions into production. It will be necessary to manage Alternatively, both disciplines could be tighter integrated, these components over the entire lifecycle and optimize the such as we have seen from certain database engineering current SDLC accordingly. It will be important to gain frameworks, where the persistence layer is automatically experience with low-risk type of applications such as a generated based on the given design of the business layer. passenger counting system first, before thinking about more advanced application scenarios such as self-driving trams or trains, which, compared to self-driving cars seem to be Combining Machine Learning with Knowledge feasible earlier due to a lesser degree of freedom. Engineering Since different ML-based AI components are used as References sensors on edge devices, rule-based systems could be handy to orchestrate their results. Given that multiple ML-based Gartner Hype Cycle for AI, 2019. (n.d.). Retrieved from sensors are installed, their results could be combined with https://www.gartner.com/smarterwithgartner/top-trends-on-the- gartner-hype-cycle-for-artificial-intelligence-2019/ results from traditional sensors. Rule-based systems could Google Coral. (n.d.). Retrieved November 12, 2019, from be used to aggregate multiple inputs and decide about the https://coral.withgoogle.com/ most suitable estimates of a number of passengers. Similar Hou, D., Liu, T., Pan, Y. T., & Hou, J. (2019). AI on edge device to Grangström, Baum, & Reuter (2017), where they compile for laser chip defect detection. 2019 IEEE 9th Annual Computing and overview of different approaches combining various and Communication Workshop and Conference, CCWC 2019, sensor types for object tracking, or improvements reported 247–251. https://doi.org/10.1109/CCWC.2019.8666503 by Tian et. al (Tian, Luo, Wang, & Tang, 2015), where they Ishikawa, F., & Yoshioka, N. (2019). How Do Engineers Perceive improved pedestrian detection by adding semantics based Difficulties in Engineering of Machine-Learning Systems? - on additional pedestrians and scene attributes, sensors Questionnaire Survey. 2–9. https://doi.org/10.1109/cesser- ip.2019.00009 within the tram could be combined with sensors outside to determine the most appropriate result. Jüngling, S., & Hofer, A. (2019). Leverage White-Collar Workers with AI. Proceedings of the AAAI 2019 Spring Symposium on Considering the management of existing models that are Combining Machine Learning with Knowledge Engineering trained for particular purposes, knowledge bases (AAAI-MAKE 2019). Retrieved from http://ceur-ws.org/Vol- incorporating an ontology could be established describing 2350/paper2.pdf the context in which the models can be applied. Facts could Kunnert, G. (2017). “Autonomes Fahren” aus be collected from the results of different AI-based modules datenschutzrechtlicher Sicht (I. Eisenberger, K. Lachmayer, & G. in combination with the run-time-environment they are Eisenberger, Eds.). MANZ’sche Verlags- und Universitätsbuchhandlung. deployed, and rules can be learned, in which cases, which modules work best. Typical situations of runtime Peraic,M. (2019). “Feasibility Study: Al-based Passenger Counting System for Public Transit” Assistant. Bachelor configurations could be tracked, and the models could even thesis. Bachelor of Science in Business Information Technology, be updated or deployed to the edge devices based on School of Business, University of Applied Sciences and Arts decisions from a knowledge-based system. Northwestern Switzerland, Basel, Switzerland Also, for the general management of the SDLC and the Rosebrock, A. (n.d.). pyimagesearch. Retrieved November 14, design-time of ML-based components, best practice rules 2019, from https://www.pyimagesearch.com/2018/07/23/simple- could be established and stored in appropriate knowledge object-tracking-with-opencv/ Saltz, J., Shamshurin, I., & Crowston, K. (2017). Comparing Data Science Project Management Methodologies via a Controlled Experiment. Proceedings of the 50th Hawaii International Conference on System Sciences (2017), 1013–1022. https://doi.org/10.24251/hicss.2017.120 Siebert, M., & Ellenberger, D. (2019). Validation of automatic passenger counting: introducing the t-test-induced equivalence test. Transportation, (0123456789). https://doi.org/10.1007/s11116-019-09991-9 Smith, R. G., & Eckroth, J. (2017). Building AI applications: Yesterday, today, and tomorrow. AI Magazine, 38(1), 6–22. https://doi.org/10.1609/aimag.v38i1.2709 Sun, S., Akhtar, N., Song, H., Zhang, C., Li, J., & Mian, A. (2019). Benchmark Data and Method for Real-Time People Counting in Cluttered Scenes Using Depth Sensors. IEEE Transactions on Intelligent Transportation Systems, 20(10), 3599–3612. https://doi.org/10.1109/tits.2019.2911128 TensorFlow Hub. (n.d.). Retrieved from https://www.tensorflow.org/hub TensorFlow Lite. (n.d.). Retrieved from https://www.tensorflow.org/lite Tian, Y., Luo, P., Wang, X., & Tang, X. (2015). Pedestrian detection aided by deep learning semantic tasks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 07-12-June-2015, 5079–5087. https://doi.org/10.1109/CVPR.2015.7299143 VÖV. (2017). Facts & Figures Swiss Public Transport 2016/2017 (Vol. 41). Wojke, N., Bewley, A., & Paulus, D. (2018). Simple online and realtime tracking with a deep association metric. Proceedings - International Conference on Image Processing, ICIP, 2017- September, 3645–3649. https://doi.org/10.1109/ICIP.2017.8296962 Zhang, J., Chen, B., Zhao, Y., Cheng, X., & Hu, F. (2018). Data Security and Privacy-Preserving in Edge Computing Paradigm: Survey and Open Issues. IEEE Access, 6(Idc), 18209–18237. https://doi.org/10.1109/ACCESS.2018.2820162