The effective lifecycle management of Artificial Intelligence (AI) solutions is crucial for ensuring their utility and reliability when moving from the sandbox into production scenarios. However, today, we are often facing the issue of AI models being developed, published and then abandoned, from a scientific point of view. This hinders the possibility for industry to adopt newest approaches as they are rarely developed with their lifecycle in mind. Unfortunately, this is also hindered by the unavailability of tools to support said lifecycle. As dynamic systems, AI solutions require continuous updates to remain relevant and accurate, which complicates verification, validation, and maintenance efforts.
In this paper, we want to focus on ontologies and knowledge graphs (KGs) as part of AI solutions and how their lifecycle management also remains an unsolved issue [1]. Today, ontologies and KGs have been adopted in many industries and they present a crucial part of hybrid AI systems, combining symbolic knowledge representation with diverse machine learning approaches. Hence, effective lifecycle management of KGs is crucial for ensuring their utility and reliability. This poses a significant challenge for companies, which do not have the funding to invest or develop their own approaches at maintaining such systems. This in turn keeps them from adopting hybrid AI in their production systems.
Therefore, we need to develop innovative tools and methodologies for the lifecycle management of hybrid AI solutions, and within this paper specifically KGs, enabling more responsible and effective use of these technologies in various applications. We propose this challenge to be addressed by borrowing from the domain of software lifecycle management and adapting those approaches. By making use of iterative and agile ways of working when developing hybrid AI solutions, we believe that we can find better ways for companies to deal with the lifecycles of AI solutions and in the process develop tools and methodologies. Two aspects play an important role here. An agile way of working ensures that a necessary culture, with the associated technology such as automated pipelines, is created that leads to continuous experimentation, facilitating both fast knowledge graph generation and frequent modification in order to innovate existing graphs [2,3,4]. But conversely, an approach based on knowledge graphs can also help to generate or improve the epics and user stories for those experiments [5,6].
Current hybrid AI systems utilizing KGs benefit from their ability to encode and reason over complex relationships. However, maintaining and updating KGs is challenging due to their dynamic nature and the need for continuous integration of new information.
With the uptake of semantic technologies to build KGs several approaches to define the KG construction process have arisen as analysed by [7]. Tamašauskaitė and Groth [7] performed a systematic literature analysis to identify a KG construction methodology: (1) identify data, (2) construct the KG ontology, (3) extract knowledge, (4) process knowledge, (5) construct the KG and ( 6) maintain the KG. They Their focus is on the construction of KGs from unstructured sources, mainly text, but the maintenance of the KG is a fairly neglected step in the overall process. Some other approaches include Radulovic et al. [8], Sequeda et al. [9], and Chessa et al. [10], however, also here the updating of the KG is not really considered in detail. Further, to our knowledge there is no available overall tool support for dealing with the KG lifecycle, as noted by Pernisch et al. [1]. Some first developments have been made, such as [11], but a thorough integration of such developments is still missing and needs to be addressed in the future.
Additionally, when addressing hybrid AI systems, the lifecycle of the KG needs to be integrated with the machine learning parts of the solution. For example, when using KGs in machine learning methods, a vector representation of the KGs is learned. Currently, there are only a few methods that can deal with updates to a KG in order to update such a representation. These are works by Song et al. [12], Cui et al. [13], or Daruna et al. [14]. However, these mostly focus on adding new information and do not regard deletions of information in their systems. Polleres et al. [15] pointed out these challenges in more detail in their survey and vision paper.
Further, existing lifecycle management approaches address artificial intelligence in general [16,17], methods such as MLOps and AIOps [18,19], and agile approaches for AI [20] they do not fully address the need for continuous updates and validation of KGs, leading to issues with compliance, accuracy, and usability but mostly hinders easy adaptation for companies. A limited number of studies address the use of KGs for software lifecycle purposes [21,22,23]. However, approaches aimed at using KGs to support lifecycle management of AI solutions in practical settings in companies do not seem to exist yet. That is why an investigation of this research gap and its support with practical innovations is necessary.
Our vision is to establish robust lifecycle management practices for hybrid AI, as well as develop methodologies and tools which support it, which integrate continuous updates, validation, and monitoring.
By leveraging both symbolic and non-symbolic AI approaches, the management of hybrid AI systems can be improved. Symbolic AI (ontologies and KGs) provides structure and reasoning capabilities, while non-symbolic AI handles large-scale data processing and adaptation. These aspects need to be carefully integrated with each other, and at the same time, its advantages should be leveraged. Iterative and agile ways of working will provide a starting point from where onwards we can then adjust and improve the methodology to suit the lifecycle of hybrid AI solutions better.
To develop a lifecycle management framework, we envision working closely with industry by applying iterative and agile approaches. We suggest using an iterative approach in which we would regularly evaluate the project results and adapt to the stakeholders' needs, just as is common for software solutions. By using this approach we will enable agile adaptation to the focus of the project maintaining the interest of stakeholders. In turn, we can revisit the applied methodology and adjust to the needs of the project itself, which will in the longer run result in better lifecycle management. Therefore, we want to create tools for continuous monitoring, analysis, and updating of hybrid AI solutions, ensuring they remain accurate and also, even though challenging, compliant with evolving regulations and standards.
Looking at results that have been achieved on a broad scale in the past decade in the field of a lifecyclewide scope on information and communications technology (ICT) applications, such as continuous integration and the associated DevOps (or BizDevOps, or DevSecOps) methods and technologies [24,25,26,27,28,29], there are still many lessons to be learned in the world of AI systems (MLOps/AIOps). Although AI systems certainly show similarities with ICT systems (they both involve code), there are important differences that necessitate a specifically AI-oriented approach. The differences between AI-based cyber-physical systems and traditional systems impact health prognostics (fit-for-purpose) and management mainly due to the AI-based systems' foundation in information flows, their novel system architectures that become necessary to enable system-internal awareness to ensure that decision-making is based on accurate information, and their unique abilities, as systems that learn and thus change their behaviour invalidates known health and performance indicators, often in unpredictable ways, and lifecycle management actions like updates, upgrades, or maintenance might no longer fit systems that adapted to their operational context [30].
Moreover, recent studies find that AI-based systems consequently pose a challenge within main Systems Engineering phases: systems that learn to change their behaviour during operations invalidate verification and validation efforts; updates or upgrades during system lifecycle management might no longer fit systems that adapted to changes in their operational context. That is why we suggest a focus on researching and developing innovation tools that support both AI makers and AI users in both the THINK phase prior to creating AI applications and the USE phase. The focus is on both Innovative AI solutions that support that lifecycle ('what': key technologies) and innovative methods and ways of working ('how': key methodologies).
We propose to tackle the presented problem and vision by establishing a collaboration, which call a lab, between industry, especially small to middles-sized businesses, and academic partners. We envision to start this work along four tracks, two focusing on the THINK phase and two on the USE phase of the three-phase model depicted in Figure 1. In Track 1, we will start with a project 'AI application mapping', which will conduct research into the development of an innovative toolset to generate data-driven personalised advice for companies for the most suitable AI application, based on added value, innovation capacity, and return on investment [31]. Using this advisory tool, a company can take a well-founded and evidence-based first step in deploying AI in daily operations. Track 2 starts with an exploratory study into the aspects of reuse of existing AI applications as developed in more than fifty labs associated with the Innovation Center for Artificial Intelligence (ICAI) 1 and the experiences on this subject of members of a industry branch association, e.g. International Council on Systems Engineering (INCOSE)2 , a not-for-profit membership organization for system engineers. This exploratory study thus lays the foundation for future research projects in Track 2.
The initial study with which Track 3 starts will investigate innovative ways to monitor and analyse AI systems in an operational setting to give companies better insight into the degree of fit for those systems. It is expected that, based on these innovative solutions, companies in the technical industry sector will have tools to carry out responsible and effective maintenance of existing AI systems by the end of the duration of the lab (planned for five years). In the fourth track, the lab starts with a study into developing an innovative tool with which customers can design continuous experimentation in an operational setting through A/B testing of their AI system. This innovation helps companies to organize continuous innovation and renewal of systems responsibly and effectively.
Effective lifecycle management of AI systems is essential for maintaining their reliability and utility. By integrating symbolic and non-symbolic AI approaches, we can develop innovative tools and methodologies to address current challenges. With this work, we see a very high societal/industry-oriented impact. Being able to provide tools and methodologies will especially help small and middle-sized businesses (SMEs) to safely and reliably deploy hybrid AI solutions more effectively. As large corporations already make use of AI solutions, it is hard for SMEs to keep up with the innovation because of the lack of funds. Therefore, we want to encourage the development and adaptation of lifecycle management practices as whole also in the research communities to continue bridging the gap between research and industry.