=Paper=
{{Paper
|id=Vol-2600/paper2
|storemode=property
|title=Towards AI-based Solutions in the System Development Lifecycle
|pdfUrl=https://ceur-ws.org/Vol-2600/paper2.pdf
|volume=Vol-2600
|authors=Stephan Jüngling,Martin Peraic,Andreas Martin
|dblpUrl=https://dblp.org/rec/conf/aaaiss/JunglingPM20
}}
==Towards AI-based Solutions in the System Development Lifecycle==
https://ceur-ws.org/Vol-2600/paper2.pdf
Towards AI-based Solutions in the System Development Lifecycle
Stephan Jüngling, Martin Peraic, Andreas Martin
FHNW University of Applied Sciences and Arts Northwestern Switzerland, School of Business
Peter Merian-Strasse 86, 4052 Basel, Switzerland
stephan.juengling@fhnw.ch, martin.peraic@bvb.ch, andreas.martin@fhnw.ch
Abstract trained models can be re-used as modules, and with the help
Many teams across different industries and organizations of transfer learning be adapted to a variety of similar
explicitly apply agile methodologies such as Scrum in their applications, AI components will slowly be deployed in
system development lifecycle (SDLC). The choice of the many more business cases. In the case of semiconductor
technology stack, the programming language, or the decision
manufacturing, it was demonstrated that a portable image
whether AI solutions could be incorporated into the system
design either is given by corporate guidelines or is chosen by classifier could be embedded in offline edge devices to
the project team based on their individual skill set. The paper detect defects on laser chips with an accuracy of 97% (Hou,
describes the business case of implementing an AI-based Liu, Pan, & Hou, 2019). However, what would be the impact
automatic passenger counting system for public on the software development process within companies,
transportation, shows preliminary results of the prototype
given the fact that AI will be increasingly used as part of
using anonymous passenger recognition on the edge with the
help of Google Coral devices. It shows how different their IT systems, products, and solutions?
solutions could be integrated with the help of rule base
systems and how AI-based solutions could be established in
the SDLC as valid and cost-saving alternatives to Application Domains for AI-Based solutions
traditionally programmed software components.
Smith and Eckroth (2017) provide a comprehensive insight
into lessons learned from building AI applications during
Introduction the last three decades. In one of their key insights, they
mention that the ease of use delivered by the human
AI in general and deep learning, in particular, are amongst
interface is the “license to operate”. This statement, which
the current hot topics in computer science research, and
has been made focusing on the client and user perspective,
many universities create new bachelor and master programs
most probably will hold true during the entire software
in data science. The hype is also visualized by the Gartner
development lifecycle (SDLC). The “new” discipline of
hype cycle for AI (2019), where edge AI, deep neuronal
data scientists needs to be incorporated into the SDLC.
networks, and machine learning are at the peak of inflected
Traditional skills of people that are involved in software
expectations. Nevertheless, AI technologies are already
design need to be extended with AI topics such as machine
used and well established in a wide area of different business
learning and knowledge engineering.
domains. However, most of the time, these are isolated and
In many practical situations, the choice of the best suitable
specialized applications, where they are clearly the sole
system development methodology is actively discussed and
possible IT solution, e.g., machine translation or image
explicitly decided at the project start. Arguments for
processing for cancer detection. In such cases, ML
different methodologies such as Scrum, Kanban, SAFE, or
capabilities are mostly predominantly compared to human
even traditional water methods are sought, and different
skills. There are not yet many situations where AI-based
suitable tools are evaluated and selected. However, all these
components are compared to traditional software or
methods still take the predominant traditional design and
hardware components. But with the current state of the art
implementation processes of software components into
neuronal network models and new emerging infrastructure
account. Also, with agile methods, some sort of
around TensorFlow (Tensorflow Hub, n.d.), where pre-
Copyright © 2020 held by the author(s). In A. Martin, K. Hinkelmann, H.- Stanford University, Palo Alto, California, USA, March 23-25, 2020. Use
G. Fill, A. Gerber, D. Lenat, R. Stolle, F. van Harmelen (Eds.), Pro- permitted under Creative Commons License Attribution 4.0 International
ceedings of the AAAI 2020 Spring Symposium on Combining Machine (CC BY 4.0).
Learning and Knowledge Engineering in Practice (AAAI-MAKE 2020).
requirements engineering, be it in the form of use cases or As such, existing systems do not provide real-time data, and
user stories, is conducted. Maybe it is followed by test- the measurements must first be collected and processed for
driven feature implementation using continuous integration subsequent use. In addition to the high acquisition costs, this
as a core idea, and customers can, at any time, verify the also results in ongoing costs for storing and processing the
current implementation state of the application and provide recorded data, which is costly, inflexible, and no longer
immediate feedback. Given the availability of state-of-the- appropriate. New methods for passenger counting must be
art AI-based video analysis algorithms and edge computing studied. In particular, it would be a good idea if existing
capabilities, the idea of building an Al-based passenger systems in the vehicles, such as the cameras installed for
counting system for public transit was reasonable and could security purposes, could be re-used. It is therefore suitable
be conducted during a bachelor thesis in cooperation with to apply the latest achievements in the field of AI-based
Basler Verkehrs-Betriebe, a leading public transport object recognition. Such an APC method could re-use
provider in the area of Basel (Peraic, 2019). existing systems, collect data in real-time, and directly
process the data without further manual operations. In order
to demonstrate the feasibility of an AI-based counting
Business Case of an AI-based Passenger system and to describe first benefits and limitations, the
Counting System for Public Transit following research questions have to be answered:
With almost 29000 kilometers, Switzerland has one of the • RQ1: Is AI-based object detection competitive against
densest public transport networks in Europe (VÖV, 2017). traditional infrared-based measurement systems under
realistic conditions?
This public service is offered by numerous private transport
companies throughout Switzerland. In order to make it • RQ2: Is it possible to perform offline object recognition
easier for customers to use these services, a variety of on edge devices from a legal perspective?
regional and national fare associations exist in different • RQ3: Can existing vehicle systems, such as surveillance
geographical areas. They ensure that public transport cameras, be re-used?
services can be accessed with a single subscription across • RQ4: What are the benefits of an AI-based automated
Switzerland. Consequently, all public transport operators passenger counting system?
are obliged to record passenger data of onboarding and
alighting passengers. Based on these figures, the percentage
Implemented Prototype
of the subscription income of the respective fare association
is calculated for the participating companies. The Basler Verkehrs-Betriebe strives to continuously
Automatic Passenger Counting Systems (APC), improve its core business, the transport of passengers, in
especially developed for public transport, are applied to order to offer the best possible service to its 350,000 daily
collect such data. Since the early 1970s, various customers. More and more processes are being enhanced or
manufacturers have been offering solutions for passenger even replaced by modern software solutions. The need for
counting (Siebert & Ellenberger, 2019), and many different tailor-made and in-house developed applications is
approaches and measuring methods are applied. Some increasing and consequently the demand for optimized
operators rely on the measurement of boarding and alighting conditions for the development of software. Currently, this
motion with the help of light barrier sensors in the infrared new corporate focus is taking place, thus no company-wide
spectrum, and some systems measure weight changes in the SDLC for development projects could be applied for the
boarding area based on associated spring movements of the prototype. This led to challenges before and during the
vehicle suspension. All of these “classic” methods are project. Development and test environments first had to be
complex systems that must be installed at every door of the created from scratch. Alongside development, the Hermes
vehicle. This requires additional hardware components, project methodology practiced by public companies in
which are expensive to purchase or to upgrade. This poses Switzerland could not be applied, as this methodology
financial problems, especially for smaller public transport follows little to no agile approaches. However, an agile
operators. Such companies use manual counting methods approach was crucial for the development of an AI-based
instead, which are frequently based on insufficient customer APC. Without a flexible delimitation of the objectives, the
surveys regarding driving behavior and do not provide continuous testing of the prototype versions under real
reliable figures (Siebert & Ellenberger, 2019). conditions and the subsequent optimizations, a runnable
Previous classic APC models are rarely deployed fleet- prototype could not have been implemented in such a short
wide due to high costs. Instead, the systems are distributed timeframe.
across all existing routes and randomly measure the number The AI-based APC was realized with Google Coral devices
of passengers, which are later extrapolated to estimate the (Google Coral, n.d.), which are optimized for TensorFlow
total number of passengers in the entire transport network. Lite (TFL, n.d.) AI models such as the mobile Single Shot
Detector (SSD). The TFL mobile SSD object recognition is source for the AI-based prototype. Subsequently, the
supported by the correlation tracking provided by DLIB measurements could be compared and evaluated using the
(Rosebrock, n.d.). With this combination, the SSD model timestamps of both measurement systems. The AI-based
recognizes all objects (position coordinates) and their object prototype correctly determined all passengers in 54% of the
class (human, bicycle, dog, etc.) in predefined frame cases. The traditional infrared APC solution achieved a
intervals. Subsequently, position and class type are stored result of 72% correct measurements.
temporarily and are further tracked during the consecutive
frames by the less computationally intensive correlation
tracker. Thereby the Coral Edge device can be relieved by
the reduced utilization of the object recognition model to
keep resources free for additional operations. That capacity
is used for the ID-based tracking of captured and stored
objects. For this purpose, Centroid Tracking is being used.
A primitive yet efficient tracking algorithm. This feature
provides the ability to observe objects across multiple
frames and measure motion into or out of the vehicle.
The setup is specially designed for offline operations to Figure 2 - Arithmetic Mean of Success in Relation to People
enable onboard computation. It allows bypassing possible involved (AI System)
future data protection obstacles of object recognition in
public space by avoiding the transmission of sensitive data RQ1: First results measured under real conditions
to a remotely located datacenter. demonstrate convincingly that with a small number of
passengers, the AI-based APC can compete with the
infrared system in terms of accuracy. Figure 2 reveals how
only a few counting errors could be observed during
experiments with one or two passengers. As complexity
increases, so does the error rate. The object tracking mostly
causes this increase. With a growing number of passengers
or increasing complexity of movements, the basic Centroid
Tracker can hardly distinguish between objects. They are
either swapped or mistakenly re-detected, causing erroneous
measurements.
Conversely, the infrared system proved to be acceptable for
multiple passengers, as shown in Figure 3.
Figure 1 - Displayed Detection and Tracking Information
After the development of the prototype, a first comparison
of the counting accuracy between the already deployed
APC, the infrared measurement, and the newly developed
AI-based system was conducted. In this respect, real test
scenarios from day-to-day life were executed under
laboratory conditions. The number of passengers and their
measurement complexity was continuously increased.
Extreme cases, such as simultaneous and dense boarding of
a larger crowd, were considered. These test scenarios were Figure 3 - Arithmetic Mean of Success in Relation to People
created by observations from past experiments and real-life involved (IR System)
situations. This combination of knowledge engineering for
the optimization of the measurement scenarios proved to be Further erroneous measurements are attributed to the AI-
a great advantage for the development of realistic and based object recognition. In realistic circumstances, object
meaningful test cases. recognition can be misled by background noise or
In order to obtain reliable results and to exclude any disturbance of passing objects. Since for performance
phenomena, each test case was performed three times. All reasons the AI-based recognition is not applied at every
measurements were recorded with a webcam directed at the frame, objects can pass the image without being detected.
door of the vehicle. This video feed was as well as the data Possibilities for improving the AI-based APC have already
been explored. Improved tracking algorithms like Deep performed passenger counting. Significantly fewer
SORT (Wojke, Bewley, & Paulus, 2017) or the utilization hardware components must be installed for the AI-based
of recent RGB-video or Depth-video approaches could APC. This results in further cost savings in the procurement
provide the required performance improvement. In and maintenance of the system. Ultimately, edge devices
particular, the use of depth sensors, as applied by Sun et al. can be continuously updated with new and enhanced AI
(2019), theoretically outperform infrared sensors regarding models. Other applications such as the automatic detection
accuracy. However, when facing problems with ML of dangers in the vehicle such as violence or emergencies
methods, this could be the point to improve the results by are just a few examples that support a switch to this new
combination with rule-based systems as described in the technology.
section about combining machine learning with knowledge
engineering.
SDLC with Embedded AI-Based Solutions
RQ2: The legal use of high-resolution video cameras as a Given the assumption of continuous improvements in the
sensor has already been discussed for autonomous driving. area of object detection with low-cost edge devices, the
In particular, the storage of video data was defined as a initial prototype could be improved within the next couple
breach of data protection (Kunnert, 2017) if the image data of months, but there will be additional issues, which must be
is stored on the device or an external database (e.g., cloud considered for the organization.
storage). The AI-based prototype is therefore designed to According to a recent survey in Japan (Ishikawa &
process the image data in real-time – storage-free. Yoshioka, 2019) software engineering professionals report
Anonymous counting results could subsequently be sent to difficulties in system engineering when ML-based
service providers for further processing. Such an components are incorporated in the engineering process.
architecture requires further security considerations. It is Many of the existing principles and best practices need to be
therefore essential to prevent third parties from gaining enlarged with additional domain knowledge on how
access to the edge devices. These could collect sensitive machine learning and knowledge engineering can be
image data or manipulate counting results, which would incorporated into the software development lifecycle of
cause manipulation of the entire service (Zhang, Chen, companies.
Zhao, Cheng, & Hu, 2018). Worth mentioning is also the performance difference when
RQ3: Existing security cameras can theoretically be SDLC methodologies known from software development
leveraged for AI-based passenger counting. Both the are applied in data science projects, where little standardized
resolution and the quality of the images are sufficient for process methodologies exist. By comparing the efficiency
object recognition. Only the position and the associated of different data science student teams working with
viewing angle of the cameras are decisive for re-use. New different SDLC methodologies, Saltz, Shamshurin, &
vehicle procurements, though, could take this into account Crowston (2017) reported an improved performance and
without additional efforts and costs. For already procured efficiency of CRISP-DM and agile Kanban, while agile
vehicles, the camera would have to be repositioned in the Scrum was even less efficient than using no methodology at
event of misplacement. all.
RQ4: The edge device architecture offers the ability to Further differences need to be considered when existing
transmit real-time count data. Furthermore, the data could code or models will be re-used. While object-oriented
be automatically merged and analyzed, allowing real-time programming existing functionality can be extended using
measurement of passenger flows using the fleet-wide well-known inheritance mechanisms, new concepts such as
deployment of the AI-based APCs. This information transfer learning need to be researched in more depth-first.
provides the control center or emergency services, with Adding additional functionality would not only mean
valuable information in a variety of situations. For example, adding additional code but also retraining some of the
in the case of a traffic accident, the number of people in the models and redeploy it to the edge devices.
vehicle can be evaluated immediately and required Besides the additional skills required for the design and
emergency resources can be notified accordingly. On the implementation of hybrid solutions, characteristics of AI-
other hand, conventional APC systems are usually entirely based solutions will most probably also have an impact on
isolated and require complex and sometimes labor-intensive the skill sets in the field of requirements engineering as well
processes to evaluate the data. Sensor data must be as testing. As stated by Jüngling & Hofer (2019), AI
converted into human-readable datasets and merged by components could even become active parts of business
skilled professionals. By eliminating these time-consuming scenarios and represented as actors in UML use case
and resource-intensive tasks, major financial savings can be diagrams.
achieved. Re-using already installed hardware such as
surveillance cameras is an additional advantage of AI-
Overall, companies that have internal software development bases about the most successful transfer learning
skills will need either additional staff with appropriate modifications of re-usable pre-trained modules. Such
experience in implementing AI technologies or educate knowledge bases could be established not only for the given
internal employees and enable them to gain experience in case of passenger counting. Such a concept could generally
order to establish new best practices for SDLC with hybrid be applied to different business application scenarios.
solutions. With more experience from practical situations,
all different disciplines, from requirements engineering to
testing and deployment will have to develop new insights Conclusion and Outlook
into how these two different kinds of systems engineering Although the given potential of AI in general and image
can be combined. It could well be, that it is best to split up recognition and object tracking in particular, practical AI-
the entire development into two parallel or serial sub- based business applications are still rare. Openness for new
projects, one for the development of the AI components, the solutions and gaining practical experience within companies
other for the traditional software components, where the will be key. Given the different nature of both ML and KE
SDLC of both components can be decoupled, having solutions compared to traditional SDLC, it seems important
different life cycles, independent regression tests, automated for companies to develop and deploy prototypes of AI-based
build pipelines and integration-tests in the end. solutions into production. It will be necessary to manage
Alternatively, both disciplines could be tighter integrated, these components over the entire lifecycle and optimize the
such as we have seen from certain database engineering current SDLC accordingly. It will be important to gain
frameworks, where the persistence layer is automatically experience with low-risk type of applications such as a
generated based on the given design of the business layer. passenger counting system first, before thinking about more
advanced application scenarios such as self-driving trams or
trains, which, compared to self-driving cars seem to be
Combining Machine Learning with Knowledge
feasible earlier due to a lesser degree of freedom.
Engineering
Since different ML-based AI components are used as
References
sensors on edge devices, rule-based systems could be handy
to orchestrate their results. Given that multiple ML-based Gartner Hype Cycle for AI, 2019. (n.d.). Retrieved from
sensors are installed, their results could be combined with https://www.gartner.com/smarterwithgartner/top-trends-on-the-
gartner-hype-cycle-for-artificial-intelligence-2019/
results from traditional sensors. Rule-based systems could
Google Coral. (n.d.). Retrieved November 12, 2019, from
be used to aggregate multiple inputs and decide about the
https://coral.withgoogle.com/
most suitable estimates of a number of passengers. Similar
Hou, D., Liu, T., Pan, Y. T., & Hou, J. (2019). AI on edge device
to Grangström, Baum, & Reuter (2017), where they compile for laser chip defect detection. 2019 IEEE 9th Annual Computing
and overview of different approaches combining various and Communication Workshop and Conference, CCWC 2019,
sensor types for object tracking, or improvements reported 247–251. https://doi.org/10.1109/CCWC.2019.8666503
by Tian et. al (Tian, Luo, Wang, & Tang, 2015), where they Ishikawa, F., & Yoshioka, N. (2019). How Do Engineers Perceive
improved pedestrian detection by adding semantics based Difficulties in Engineering of Machine-Learning Systems? -
on additional pedestrians and scene attributes, sensors Questionnaire Survey. 2–9. https://doi.org/10.1109/cesser-
ip.2019.00009
within the tram could be combined with sensors outside to
determine the most appropriate result. Jüngling, S., & Hofer, A. (2019). Leverage White-Collar Workers
with AI. Proceedings of the AAAI 2019 Spring Symposium on
Considering the management of existing models that are Combining Machine Learning with Knowledge Engineering
trained for particular purposes, knowledge bases (AAAI-MAKE 2019). Retrieved from http://ceur-ws.org/Vol-
incorporating an ontology could be established describing 2350/paper2.pdf
the context in which the models can be applied. Facts could Kunnert, G. (2017). “Autonomes Fahren” aus
be collected from the results of different AI-based modules datenschutzrechtlicher Sicht (I. Eisenberger, K. Lachmayer, & G.
in combination with the run-time-environment they are Eisenberger, Eds.). MANZ’sche Verlags- und
Universitätsbuchhandlung.
deployed, and rules can be learned, in which cases, which
modules work best. Typical situations of runtime Peraic,M. (2019). “Feasibility Study: Al-based Passenger
Counting System for Public Transit” Assistant. Bachelor
configurations could be tracked, and the models could even
thesis. Bachelor of Science in Business Information Technology,
be updated or deployed to the edge devices based on
School of Business, University of Applied Sciences and Arts
decisions from a knowledge-based system. Northwestern Switzerland, Basel, Switzerland
Also, for the general management of the SDLC and the Rosebrock, A. (n.d.). pyimagesearch. Retrieved November 14,
design-time of ML-based components, best practice rules 2019, from https://www.pyimagesearch.com/2018/07/23/simple-
could be established and stored in appropriate knowledge object-tracking-with-opencv/
Saltz, J., Shamshurin, I., & Crowston, K. (2017). Comparing Data
Science Project Management Methodologies via a Controlled
Experiment. Proceedings of the 50th Hawaii International
Conference on System Sciences (2017), 1013–1022.
https://doi.org/10.24251/hicss.2017.120
Siebert, M., & Ellenberger, D. (2019). Validation of automatic
passenger counting: introducing the t-test-induced equivalence
test. Transportation, (0123456789).
https://doi.org/10.1007/s11116-019-09991-9
Smith, R. G., & Eckroth, J. (2017). Building AI applications:
Yesterday, today, and tomorrow. AI Magazine, 38(1), 6–22.
https://doi.org/10.1609/aimag.v38i1.2709
Sun, S., Akhtar, N., Song, H., Zhang, C., Li, J., & Mian, A. (2019).
Benchmark Data and Method for Real-Time People Counting in
Cluttered Scenes Using Depth Sensors. IEEE Transactions on
Intelligent Transportation Systems, 20(10), 3599–3612.
https://doi.org/10.1109/tits.2019.2911128
TensorFlow Hub. (n.d.). Retrieved from
https://www.tensorflow.org/hub
TensorFlow Lite. (n.d.). Retrieved from
https://www.tensorflow.org/lite
Tian, Y., Luo, P., Wang, X., & Tang, X. (2015). Pedestrian
detection aided by deep learning semantic tasks. Proceedings of the
IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, 07-12-June-2015, 5079–5087.
https://doi.org/10.1109/CVPR.2015.7299143
VÖV. (2017). Facts & Figures Swiss Public Transport 2016/2017
(Vol. 41).
Wojke, N., Bewley, A., & Paulus, D. (2018). Simple online and
realtime tracking with a deep association metric. Proceedings -
International Conference on Image Processing, ICIP, 2017-
September, 3645–3649.
https://doi.org/10.1109/ICIP.2017.8296962
Zhang, J., Chen, B., Zhao, Y., Cheng, X., & Hu, F. (2018). Data
Security and Privacy-Preserving in Edge Computing Paradigm:
Survey and Open Issues. IEEE Access, 6(Idc), 18209–18237.
https://doi.org/10.1109/ACCESS.2018.2820162