The idea of design patterns originated in the architecture domain, subsequently shaped the standardization and communication of object-oriented system design for IT architectures, and facilitated the description of best practices in business process design. Recently, the idea of design patterns not only stipulated an initial collection and classification of machine learning patterns, but has also been used to structure and document machine learning based systems from a traditional software engineering perspective. We promote the idea of using design patterns as a general means to visualize the design of hybrid AI systems and present how the strategy design pattern in particular can be used for a passenger counting system by switching the implementation strategies from a standard YOLOv5 based object recognition with Deep Sort tracking to a customized head-based YOLOv5 detection in combination with a customized Deep Sort tracking strategy. In our example, the newly presented human head detector and tracker could significantly improve the overall accuracy of passenger counting in dense and crowded situations. Furthermore, we show, how rule-based symbolic decisions can be allocated to an AbstractStrategy class, while the sub-symbolic machine learning task is delegated to the most appropriate person- or head-based ConcreteStrategy class during run-time.
Design patterns have been established as elements of reusable object oriented (OO) software
design [
Compared to the software development community, where Design Patterns (DP) are well
settled to communicate design ideas, to improve the overall system design or enhance the
maintainability and extendability of code, the use of DPs to communicate design ideas in
the field of machine learning is quite new and not yet well established. While the positive
impact of DP on the quality of software development is researched in a recent systematic
literature review from diferent perspectives [
Both pattern collection types provide domain specific guidance in form of best practices of
reusable problem solution pairs. However, the scope of the ML-DP is much broader and the
complexity much higher. The GoF-DP can easily be described with UML class and sequence
diagrams since these patterns focus on the collaboration of classes with appropriate interface
designs. Aspects of the software development lifecycle (SDLC), deployment of components,
aspects of performance or diferent data-types are out of scope of the GoF-DP, while the ML-DP
cover such aspects in order to provide some guidance for machine learning practitioners. In
the case of hybrid AI systems which combine sub-symbolic machine learning algorithms with
symbolic rule-based components, the complexity of the domain increases even further. To
visualize the interplay between data, symbolic and sub-symbolic components, Van Harmelen
and Ten Teye developed the concept of a boxology as a set of compositional DP defining their
own graphical syntax and semantics to describe all the possible solution designs of such hybrid
AI systems [
There are also many ways, how GoF-DP and Machine Learning (ML) can be combined.
Possible combinations range from applying ML algorithms to detect the GoF patterns in programming
code [
We like to go a little bit deeper into the implementation details of applying the strategy DP in the context of combining symbolic and sub-symbolic components for the allocation of responsibilities to the typical classes involved in this structural DP. As stated by Gamma et al., the intent of the strategy DP is to define a family of algorithms, encapsulate them and make them interchangeable. Strategy and Context interact in order to determine the the most adequate algorithm, or in our case of hybrid AI systems, the most adequate ML model. When calling the Strategy, the Context either passes all the necessary data as arguments to the Strategy for calling the predict() method, or the Context might pass itself as an argument to the Strategy (see figure 1) in order that the Strategy itself can determine which of the concrete strategies maps best to the current prediction task at hand.
Diferent ML models can be encapsulated in ConcreteStrategies and as such, can change the behavior of the Strategy which is seen by the Context. However,the first thing when implementing a new extended functionality in a software system is to decide where to allocate the additional code. In object oriented programming one basically has only two choices, either to use inheritance or delegation. In case of using inheritance, you extend one of the existing classes to reuse the existing functionality and add additional methods or overwrite or override existing methods. When implementing software systems with OO languages such as Java, C++ or Python, it is also a good practice to reduce the coupling in the overall class design as much as possible, by programming client code towards an interface and not towards an implementation. In the case of hybrid AI system design, the forces are somehow diferent. Knowledge Engineering (KE), ML, and OO programming are mostly disjoint disciplines and some of the current shortcomings come rather from to little than to tightly coupling. The idea of optimizing the overall system design in terms of coupling and decoupling of individual components remains the same. But in the domain of ML, it is not yet defined how to model their domain specific characteristics, where part of the models can be "reused and extended" with the help of transfer learning, which would be kind of similar to inheritance in OO class design. However, what we can visualize, is the aspect of encapsulating the diferent ML models in form of diferent ConcreteStrategies, where the Strategy DP can be used to modify the behavior of the ML model, as it is perceived by calling the interface of the Strategy of the Context class. However, in the long run, it would also be quite nice to show explicitly the reuse of pre-trained ML models, which can be adapted for the diferent business purposes. Nevertheless, we show in the following section, how the strategy DP and diferent ConcreteStrategies can be used to model and optimize the implementation of a passenger counting system.
While the initial experiments of using video based passenger counting systems as cost efective
alternative for passenger counting in public transportation systems has been investigated with
Basler Verkehrsbetriebe BVB [
A collaborative student project between SZTU School of Urban Transportation and Logistics and FHNW School of Business [11] focused on exploring potential approaches for solving this challenge. Initially, the task was defined as counting the persons within a defined view of a surveillance camera by augmenting a line. Any objects that traverse this line are counted. To achieve this contextual objective, a standard AI-based architecture for passenger detection and tracking was implemented. Based on the observations from various experiments with the available video data, possible solutions have to be designed, verified and tested within a single application in order to optimize the overall accuracy of the passenger counting system.
Having defined the methodology of this project, the individual approaches and their implementations shall henceforth be explained. To provide a starting point for this project, YOLOv5 [12] was chosen for detection and Deep Sort [13] for continuous object tracking. Both implementations were integrated with their standard configurations and models. No additional configurations were made for either YOLOv5 or Deep Sort. The objective of this standard implementation was to explore how capable the deployed AI architecture was to handle the passenger counting scenario. Subsequent experiments proved that many people are not detected by YOLOv5, particularly in large crowds. The selected angle of the surveillance camera causes the detection to ignore the bodies of people who are either located further back in dense crowds or are occluded behind objects. Based on the research of Hong et al. [14] and the initial experimental results, a decision to adapt the standard model of YOLOv5 was made. Rather than detecting entire humans, heads should be focused, as they appear to have a less higher probability to be occluded compared to bodies.
Approximately 1000 data samples were gathered and labeled from various recordings from diferent camera angles of the Shenzhen Metro. Subsequently, the YOLOv5 model was re-trained. Since the detector now passes bounding boxes of heads to the tracking, the feature extractor of Deep Sort must be adapted as well. To accelerate the training and embedding process, the concept and implementation described by Maiya [15] was adopted. Thereby a Siamese network [16] is used, which is ideally suited for feature mapping tasks. Another 5000 data samples were collected for the training of this network.
Once all the components had been merged together, it was time to run the experiments. For this purpose, various video sequences were selected from the available Shenzhen Metro data. Diferent sizes of passenger flows were taken into account and the number of people passing the augmented counting line was determined for each video. Using these video clips combined with the ground truth of the number of passengers, both the standard models and the adapted models could now be examined for their counting accuracy. The customized setup achieved a Mean Average Error (MAE) of 1.7714 and the standard setup a MAE of 6.4857. The results as shown in figure 3 impressively illustrate how the adapted AI setup can significantly measure the number of passengers with higher accuracy compared to the standard setup. Even under very complex movement patterns and large crowds, the adapted system was able to perform highly accurate counts.
Besides the higher counting accuracy, the customized setup also recorded significantly more tracks per frame. Unlike the standard setup, which only achieved between 6-10 tracks per frame, the customized setup as illustrated in figure 2 was able to register up to 80 tracks per frame. This increase in active tracks could be achieved as assumed by reducing occlusion efects. Nevertheless, negative features of the adapted version could be observed as well. Since the object size is considerably reduced by the shift from entire bodies to heads, the number of pixels also decreases the further back the person is located in the image. Thus, especially in the background where the pixel density is small, a re-identification of the objects frequently occurs. This problem can be solved relatively easily by excluding parts of the area, so that only objects located in a certain range in front of the counting line are tracked.
From a functional perspective, the implementation of the use case has been completed, but from a software engineering perspective, the code needs to be refactored in order to properly allocate responsibilities to classes and to improve the overall system design [17, 18]. In order to encapsulate the functionality of specific ML algorithms and to provide the flexibility of switching and comparing the diferent passenger counting approaches, the Strategy DP can be used, as illustrated in figure 4.
The Context class holds the attribute frame and counts values for each person passing the augmented line in one of the two possible directions (either the objects enter or leave the metro). Two separate strategies are used for the detection and tracking, which both can be implemented with the standardized as well as the customized version. In chronological sequence within the application, frame by frame, this sequence of detection strategy followed by one of the tracking strategies can be repeated. Both abstract classes DetectionStrategy and TrackingStrategy expose the interface from the ConcreteStrategies and hold a reference to the Context, which can be passed at the method call determineModel(). This allows us to dynamically evaluate the concrete strategy based on diferent input parameters, which could be implemented in form of a decision table [19], which in turn is a typical representative of a rule-based system. With the help of the Decision Model and Notation DMN, entire decision hierarchies might be compiled, such as shown in figure 5, where the specific implementation strategy is determined as combination of a SystemContext and a BusinessContext.
The SystemContext might refer to specific technical circumstances such as the current hardware type, camera resolution or number of GPUs available, while the Business Context might consider diferent situations such as people density or even diferent business purposes such as counting the number of persons or counting the number of persons wearing a mask, as it might be of interest in the current situation of COVID-19.
Although the development of hybrid AI-based software systems follow similar software development lifecycle stages and concepts as traditional IT systems, some of them are special due to the diferent nature of ML and KE. The ML lifecycle needs dedicated phases of data extraction, data preparation, model training evaluation and testing, until it can be deployed into operations. Not only the kind of reuse such as OO with established inheritance mechanisms, but also the maturity of these reusable concepts are diferent. Nevertheless, an increasing number of ML models are stored in standardized model repositories such as TensorFlow Hub [20], facilitating reuse in diferent BusinessContexts and SystemContexts. Hence, continuous delivery and automation pipelines can be built to enable the development and operation of large software systems, known as DevOps and MLOps, respectively.
Hybrid AI systems thus require a combination of disciplines from the selection and development of the model up to the final deployment. Figure 6 illustrates a possible solution path for this process. Once the model engineering is started, possible model reuse opportunities must be considered. Based on the requirements and the context, an existing model can be used either completely or as a starting point for transfer learning. This manually executed task facilitates the description of the abstract class using the Strategy DP. By using UML and standardized DP as proposed by this paper, a generally understandable blueprint can be developed, which can later be implemented in the system by diferent domain specialists. Subsequently, the necessary rules for the run-time strategy selection have to be defined. Based on the possible parameters of the context, a DRD diagram or a decision tree can be used as shown in Figure 5. To complete this first design phase within the lifecycle, all partial elements must now be tested for their compatibility and performance. Once the tests are successful, all symbolic and sub-symbolic elements can be deployed. Within the run-time environment, a monitoring can now be performed to determine whether the hybrid AI system is able to perform the given task. If the performance is either faulty or poor, or if the strategy requirement suddenly changes, adjustments can now be made dynamically to both the strategy and the decision model. Due to the standardization, these parts can be easily exchanged and deployed in a very short time.
Applied to the Shenzhen Metro use case, this lifecycle process was triggered by the need for improved models for passenger detection and tracking. In this process, the already available models YOLOv5 and Deep Sort were optimized to fit the head detection scenario. Furthermore, the implementation was optimized according to the Strategy DP as described in Figure 4. Using a decision table, various parameters from the context such as recognition targets, accuracy, speed, density, etc. can be used to identify a concrete strategy. After successful integration testing, the hybrid system could be deployed followed by real-time monitoring. In case of accuracy degradations due to changing contexts or scenarios, strategies can be switched and the system can adjust to the new situation thanks to the standardized interface and the recursive optimization process.
As shown in the implementation of the current application of passenger counting with the potential switch between diferent implementation strategies based on head or entire body detection, the structure of the strategy design pattern can easily be used to visualize the design idea of implementing diferent behavior of diferent machine learning algorithms in the run time environment. However, as described in the previous chapter about deployment possibilities, the complexity of the decision about the best choice of the deployment of the specific ML models can be huge and the number of facts and rules for proper selection by the DRD quite numerous. The overall lifecycle of the most adequate algorithm, which can be engineered, trained and validated based on the best practices of the ML-DP before it is deployed as strategy to the run time environment is quite challenging, and appropriate build pipelines from the design-time to the run-time environments, as they are known from the deployment of traditional software components remain to be standardized. Appropriate visualizations, which can be used for the abstraction of the diferent ML models are currently lacking. Could UML be extended by additional diagram types that could visualize the re-use and modifications of existing ML models? We raised this question already at the beginning when making the analogy of OO inheritance to transfer learning in ML. How could the diferent modifications to the architectures of existing ML models, which can be visualized as textual output in diferent ML models (e.g., the diferent layers of a Keras model with commands such as model.summary()) be visualized in kind of UML class diagrams? Up to our knowledge, appropriate solutions are currently not even proposed.
While design patterns are state of the art for many years in software engineering, investigations in machine learning design patterns only started recently. In order to build hybrid AI systems, where machine learning and knowledge engineering should be combined within a single system, the glue that enables this combination is still designed with traditional programming, and appropriate software architectures need to be built up. Therefore current best practice solution architectures which are built with the Gang of Four object oriented design patterns can be used to embed the appropriate machine learning and knowledge engineering components. We use strategy design pattern and show how rule based components can be embedded in the abstract strategy which in turn calls the most appropriate machine learning solution as a concrete strategy, solving the business problem at hand. In the use case of passenger counting from scenes with crowded people in the Shenzhen metro, we use YOLOv5 and Deep Sort as tracking and detection algorithms, we have shown that customized versions of both, which do the counting based on head tracking instead of entire person tracking Can be seen as diferent strategies showing superior performance in a given environment. We show how the rule based and machine learning components can be embedded in an overall IT system by using the idea of refactoring to patterns, which in turn can be visualized with UML. This strategy design pattern is only one of the 23 GoF design patterns, and for the remaining design patterns appropriate ways of how machine learning and knowledge-engineering components can be embedded and combined for the design of hybrid AI-systems remain to be researched. Additional investigations into how machine learning components that are reused in form of transfer learning, can be visualized in analogy to the successfully standardized inheritance mechanism from software engineering. Further investigations will be necessary in terms of mutual collaboration between the The diferent communities of software engineering, machine learning and knowledge engineering experts to foster common ways of finding optimal designs of hybrid systems. Other than in the single domain of software engineering, where the GoF design patterns originated from controlling typical defects of tight coupling of code, the same design patterns contribute to embed more coupling between currently quite heterogeneous fields of research. [9] S. Jüngling, M. Peraic, A. Martin, Towards AI-based Solutions in the System Development Lifecycle, Proceedings of the AAAI 2020 Spring Symposium on Combining Machine Learning and Knowledge Engineering in Practice (AAAI-MAKE 2020) - Volume I 2600 (2020). [10] UITP, World metro figures statistics brief, 2018. URL: https://www.uitp.org/publications/ world-metro-figures/. [11] G. S. Lu, Y. S. Peng, M. F. Peraic, Ai based passenger conting systems - approaches in basel ans shenzhen: Independent learning module msc fhnw school of business and sztu school of urban transportation and logistics, 2021. [12] J. Glenn, Yolov5 in pytorch, 2020. URL: https://github.com/ultralytics/yolov5. [13] N. Wojke, A. Bewley, Deep cosine metric learning for person re-identification, in: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2018, pp. 748–756. doi:10.1109/WACV.2018.00087. [14] Z. Hong, L. Zhang, P. Wang, Pedestrian detection based on yolo-d network, 2018, pp.
802–806. doi:10.1109/ICSESS.2018.8663790. [15] S. R. Maiya, Deep sort custom object tracking, 2019. URL: https://github.com/abhyantrika/ nanonets_object_tracking/. [16] G. Koch, R. Zemel, R. Salakhutdinov, et al., Siamese neural networks for one-shot image recognition, in: ICML deep learning workshop, volume 2, Lille, 2015. [17] M. Fowler, Refactoring - improving the design of existing code, in: Addison Wesley object technology series, 1999. [18] J. Kerievsky, Refactoring to patterns, in: XP/Agile Universe, 2004. [19] Decision model and notation dmn v.1.3, 2021. URL: https://www.omg.org/spec/DMN/1.3/
PDF. [20] Tensorflow hub, 2021. URL: https://www.tensorflow.org/hub/.