The development of AI-based systems entails a manifold of doubled-hard challenges. They are mainly due, on one side, to the technical debt of involved engineering disciplines (systems, safety, security), their inherent complexity, their yetto-solve concerns, and, on the other side, to the emergent risks of AI autonomy, the trade-offs between AI heuristics vs. required determinism, and, overall, the difficulty to define, characterize, assess and prove that AI-based systems are sufficiently safe and trustworthy. Despite the vast amount of research contributions and the undeniable progress in many fields over the last decades, a gap still exists between experimental and certifiable AIs. The present paper aims at bridging this gap “by design”. Considering engineering paradigms as a basis to specify, relate and infer knowledge, a new paradigm is proposed to achieve AI certification. The proposed paradigm recognizes existing AI approaches, namely connectionist, symbolic, and hybrid, and proffers to leverage their essential traits captured as knowledge. A conceptual meta-body is thus obtained respectively containing categories for Data-, Knowledge- and Hybrid- driven. Since it is observed that research strays from Knowledge-driven and it rather strives for Data-driven approaches, our paradigm calls for empowering Knowledge Engineering relying upon Hybrid-driven approaches to improve their coupling and benefit from their complementarity.
Safety can be defined as “freedom from risk which is not tolerable” (ISO). This definition implies that a safe system is one in which scenarios with non-tolerable consequences have a sufficiently low probability, or frequency, of occurring. Thus, Safety critical systems must be dependable during all their life-cycle, supporting evolution without incurring prohibitive costs. It becomes mandatory that an AIbased Critical System (AICS) does what it has been specified to do (correctness). In the near term, the goal of deploying AI on critical systems motivates research to handle accountability, reliability, suitability, timeliness, etc. Moreover, AICS need to be resilient, safe and (cyber)-secure. Complex mechanisms have to be integrated to ensure both responsibility and accountability of AICS and their outcomes. As any critical system, an AICS needs to be verified, validated, qualified and even certified, following, a suitable development methodology. Explainability is also an issue because AI systems and their decisions need to be explained to integration and maintenance teams as well as end-users in an understandable manner. Auditability for assessing algorithms, data, knowledge, design and integration processes is also a key property that has to be handled. The Fig. 1 highlights various non-functional requirements that have to be verified and demonstrated for a sound AI-based component deployment within an AICS. All such requirements need a sound and trustworthy AI engineering methodology with efficient supporting tools, while addressing various levels of granularity. On one hand they must encompass specific algorithmic domains engineering, including associated data, models and knowledge representations. On the other hand, they must guarantee architecture design correctness up to the complete system engineering cycle.
By the taxonomy of disciplines involved (Systems, Knowledge, Algorithms, Safety, and Security Engineering), their inherent complexity, the yet-to-solve concerns within each of them (technical debt), and the fact that AICS lay down in the intersection of those disciplines, then AICS development becomes a doubled-hard challenge. Indeed, from an engineering perspective, questions like, Which are the fundamental notions and features to characterize AICS? Which development languages and methods can sufficiently comprehend and interrelate those notions? How can knowledge be structured to suitably elicit, fulfill and verify requirements? and last, yet not the least, Whether the bundle of criteria (ranging from data-sensitivity to explainability) can be harmonized and how?
This position paper introduces the need of a conceptual paradigm which aims to provide a basis upon which answers to previous questions can be elicited. To handle the complexity of the subject, several choices are made. First, since autonomy is a distinctive feature of AICS, and it is intrinsic to safety, the latter is placed as top-priority criterion. Then it is used to guide design, conduct analyses and align the rest of criteria: all in all, we target Safety-Critical AICS (AISCS). Secondly, the proposed paradigm assumes that fundamental notions, as such, should appear in there, irrespective of the development methodology or framework. Last yet not the least, the paradigm also assumes that challenges in AISCS development can be addressed via a body of knowledge amenable to (1) emulate whatever humanbeings perceive as intelligence and (2) integrate related concerns and in particular safety. Overall, this paper is dedicated to provide a first specification of a conceptual paradigm as a basis for safe AI-based systems development, in order to design a sound and tooled AI engineering methodology that encompasses with objective of trustworthy AI algorithm engineering, data engineering, knowledge engineering and AI system engineering.
In this Section, we shortly describe the background taken as reference for the description of our paradigm and how it is leveraged to constitute the expected foreground.
After conducting a brief survey of approaches for AI
development
The proposed paradigm leverages the background described in previous Subsection in the following manner. First, symbolic, connectionist and hybrid are methods and techniques with distinctive features which are applied to achieve the intended AI functionality. In that respect, once a technique is selected, the engineering choices are mostly oriented to explore and decide HOW and WHEN to apply. Given the referred challenges for AISCS development, and in order to extend the space of engineering choices, it is proposed to consider such techniques as instances of a more abstract meta-structure: an AI-body of knowledge. Such body is meant to include structured information amenable to dissert about WHAT and WHY, in a first place, and in addition HOW and WHEN. Thus, since the connectionist approaches treat and depend upon data sets, then the AI-body of knowledge includes the category of approaches driven by data, i.e. a Data-driven AI. Similarly, the symbolic approaches rely upon rules allowing reasoning on terms and notions to infer further knowledge, then the AI-body contains a category of approaches driven by knowledge, i.e. a Knowledge-driven AI. A mix of Data- and Knowledge-Driven yields the third category named Hybrid-driven AI.
From the literature survey, the research seems to stray
from symbolic AI methods and instead leverage
learningbased artificial neural networks. However, some
underlining issues of current data-driven approaches such as
robustness, fairness, explainability, maintainability, etc. are
leading some to call for a return of knowledge-based or
some reconciliation of the two main paradigms through
Hybrid AI
The conceptual paradigm aims to address several stakes of
AISCS. Some of the most salient are listed below.
• Quantitative safety metrics and methods. Metrics based
upon failure rates have been proven effective to achieve
suitable levels of safety. However, new metrics and
methods to measure errors and misbehaviours of AI modules
are yet to be defined and incorporated.
• Qualitative safety metrics and methods. Human
interpretability calls for qualitative metrics. Indeed,
explainability, risks, and even safety of AI are notions that
require qualitative interpretation (meaning) in order to be
assessed.
• Data modeling and quality. Data modeling is proposed
as a mean for assessment of data features. The influence
of data traits (e.g., data diversity
The conceptual paradigm aims to be a basis of a framework for representation and integration of previous aspects as well as for inference and assessment. The building process consists in empowering Knowledge Engineering through a better coupling of Data- and Knowledge-driven approaches. The rest of the paper is dedicated to describe the salient aspects and constituents of the proposal.
Introduced in 1956, AI is a computer sciences discipline concerned with the theory and development of artificial systems able to perform cognitive activities such as reasoning, knowledge representation, planning, learning, natural language processing, perception and decision. AI includes a wide range of technologies which can be divided into two broad categories: (1) Data-driven AI which includes neural networks, statistical learning, evolutionary computing...; and (2) Knowledge-based AI which focuses on the development of ontology and semantics graphs, knowledge-based systems and reasoning... However, each AI paradigm only focuses on portions of the information and decision chain, leading to solutions that are thus not driven by the global “good decision” goal, making them globally inefficient.
The premises of data-driven AI and knowledge-based AI are
fundamentally different (see figure 2). The paradigm of
datadriven AI is based on brain-style learning such as neural
networks, whereas Knowledge-based AI approaches employ
model and knowledge reasoning.
• Often used in the context of pattern recognition,
classification, clustering or perception, data-driven AI such
as machine learning aims at capturing tacit knowledge
knowledge which is difficult or impractical to explicitly
or analytically define - through statistical approaches by
inferring the inherent structure of a set of examples (input
data) that can be used for mapping new data samples.
•
In Software Engineering, the distinction between a functional specification and the design/implementation of a system is often discussed as a separation of what and how. During the specification phase, what the system should do is established in interaction with the users. How the system functionality is realized is defined during design and implementation (e.g., which algorithmic solution can be applied). This separation does not work in the same way for a knowledgebased system (KBS) which is a computer system that represents and uses knowledge to carry out a task and inference procedures to solve problems that are difficult enough to require significant human expertise for their resolution. Thus, such system has two distinctive features: a knowledge base and an inference engine, where knowledge is then assumed to be given “declaratively“ by a set of Horn clauses, production rules, constraints or frames and where inference engines like unification, forward or backward resolution, and inheritance capture the dynamic part of deriving new information.
For instance, constraint programming (CP) is a
knowledge-based AI approach
Knowledge engineering (KE) is the process of understanding and then representing human knowledge in data structures, semantic models (conceptual diagram of the data as it relates to the real world) and heuristics. Expert systems, constraint programming, ontologies, ... are examples that form the basis of the representation and application of this knowledge. The basic assumption is that both knowledge and experience can be captured and archived in textual or rule-based form, using formalization methods.In its initial form, KE focused on transferring the expertise of a problemsolving human into a program that could take the same data and make the same conclusions. In the 1990s, the KE community shifted gradually to domain knowledge, in particular reusable representations in the form of ontologies. This evolution aimed at alleviating KE limitation to accurately reflect how humans make decisions and more specifically its failure to take into account intuition and “gut feeling”, known as “reasoning by analogy”. Nevertheless, designing knowledge-based AI component induces some general fundamental problems. 1. Knowledge discovery: how do we translate knowledge as it currently exists in textbooks, articles, databases, and human skills into abstract representations in a computer? 2. Knowledge representation: how do we represent human knowledge in terms of data structures that can be processed by a computer? How to determine the best representation for any given problem? 3. Knowledge reasoning: how do we use these abstract data structures to generate useful information in the context of a specific case? How to manipulate the knowledge to provide explanations to the user?
date the knowledge base? How to evaluate and validate knowledge-based systems?
While some knowledge is easy to obtain and understand, other knowledge may be difficult to obtain or interpret. In many situations, experts do not have any formal basis for problem solving or for explaining their reasoning process. So they tend to use “rules of thumb” (heuristics) developed on the basis of their experience to help them make decisions. Thus, Knowledge discovery is the process of collecting, extracting, transferring, accumulating, structuring, transforming and organizing (domain) knowledge (e.g., problem-solving expertise) from data and information or from the synthesis of prior knowledge. One of the most important key points of knowledge discovery is to ensure that correct and relevant knowledge is extracted and represented to the stakeholders and decision makers. No matter what kind of knowledge is collected, this process can be realized in a manual way and in an automatic way.
Even if knowledge discovery is today dominated by
machine learning (ML) approaches, the iterative execution of
the CRISP-DM1 methodology
Knowledge Representation and Reasoning (KRR) represents information from the real world for a computer to understand and then utilize this knowledge to solve complex real-life problems. KRR is not just about storing data in a database, it is the study of how what we know can at the same time be represented as comprehensibly as possible and reasoned with as effectively as possibly. One of the main issue is to find the best trade-off between these two concerns.
For
Developers building knowledge-based systems (KBS), usually create knowledge bases from scratch through a tedious and time-consuming process. First, they have to deal with the diversity and heterogeneity of knowledge representation formalisms and with modeling, taxonomical, and terminological mismatch of different knowledge items, even if they belong to the same application domain. Thus, while the data engineers focus on building the data pipes and data scientists focus on inference methods, knowledge engineers focus on modeling structural use cases and detailing concepts of expert knowledge. Knowledge engineering methods adapt to use cases of knowledge and can model for specific requirements and in many cases produce reusable formats. One of the main limitations of knowledge-based systems lie in the abstract nature of the considered knowledge, in acquiring and manipulating large volumes of information or data, and the limitations of cognitive and other scientific techniques.
Despite of the progress in KE and ontology engineering
in the last decade, obstacles remain. Modeling is still a
difficult task, as with the choice of the suitable
knowledgebased AI technology. Like every model, such a model is
only an approximation of the reality. The modeling
process is often cyclic. Expert Knowledge, notably via
Modeling bias, whereby a human manually designing a model
(or part of a model) does not take into account some
aspects of the environment in building the model, consciously
or unconsciously. New observations may lead to a
refinement, modification, or completion of the already built-up
model. On the other side, the model may guide the
further acquisition of knowledge. Therefore an evaluation of
the model with respect to reality is indispensable for the
creation of an adequate model. These limitations relate to the
so-called symbol grounding problem
Another key concern in knowledge based modeling is stability. How much variability is there between instances of the problem? How stable is the solution method to small changes? Is the problem very dynamic? What happens if (a small amount of) the data changes? Do solutions need to be robust to small changes? Many such questions need to be answered before we can be sure that a particular knowledge based AI technique is a suitable technology.
ML based AISCS engineering is often portrayed as the creation of a ML/DL model, and its deployment. In practice, however, the ML/DL model is only a small part of the overall system and significant additional functionality is required to ensure that the ML/DL model can operate in a reliable and predictable fashion with proper engineering of data pipelines, monitoring and logging, etc. To capture these aspects of AI engineering we defined the ML algorithm engineering pipeline (see fig. 3), where we distinguish between requirements driven development, outcome-driven development and AI-driven development. As the starting point, data must be available for training. Based on data engineering, there are various ways to collect and qualify data set and divide it to training, testing, and cross-validation sets. Engineering activities have to be encapsulated as a series of steps within the pipeline such as: • 1) Problem specification, including the Operational Design Domain (ODD), that is the description of the specific operating condition(s) in which a safety-critical function or system is designed to properly operate, including but not limited to environmental conditions and other domain constraints. These requirements describe the specific function that the ML items should implement as well as the safety, performance, and other requirements that the ML items should achieve. • 2) Data engineering, including data collection, preparation and data segregation propose some guidelines. A machine learning model requires large amounts of data, which help the model learning about system objectives and purpose. Before it can be used, data needs to be collected and usually also prepared. Data collection is the process of aggregating data from multiple sources. The collected data needs to be sizable, accessible, understandable, reliable, and usable. Data preparation, or data pre-processing, is the process of transforming raw data into usable information. • 3) ML Algorithm Design, after feeding training set to the ML algorithm, it can learn appropriate parameters and features. Once training is complete, the model will be refined by using the validation data-set. This may involve modifying or discarding variables and including a process of tweaking model-specific settings (hyperparameters) until an acceptable accuracy level is reached. • 4) Implementation, to develop ML components, we have to decide on the targeted hardware platform, the IDE (Integrated Development Environment) and the language for development. There are several choices available. Most of these would meet our requirements easily as all of them provide the implementation of AI algorithms discussed so far, but sometimes we have to take into account embedded constraints. • 5) Evaluation and verification, after an acceptable set of hyperparameters is found and the model accuracy is optimized, we can finally test our model. Testing uses our test dataset and is meant to verify/demonstrate that our models are correct and guarantee some required properties such as robustness and/or explainability. Based on the feedback, we may return to training the model to improve correctness, accuracy and robustness, then adjust output settings, or deploy the model as needed. • 6) Model Deployment in the overall system with respect to safety and cyber-security system requirements. Learning assurance case methods can be used.
Then, an ML algorithm has to be designed or selected on
existing ML library (such as Scikit Learn
Conclusion “Data-driven AI is the AI of the senses, and knowledgebased AI is the AI of meaning”2. This is why, in order to cover all cognitive capacities, the future lies in the hybridization of these two paradigms, which are often placed in opposition to AI. Indeed, the shortcomings of deep learning align with the strengths of knowledge-based AI, which 2David Sadek, VP Research Technologies and Innovation at Thales raises the possible benefits of hybridization. First, thanks to their declarative nature, symbolic representations can easily be reused in multiple tasks, which promotes data efficiency. Second, symbolic representations tend to be highlevel and abstract, which facilitates generalization. Lastly, because of their propositional nature, symbolic representations are amenable to human understanding. AI algorithms need relevant observations to be able to predict the outcome of future scenarios accurately, and thus, data-driven models alone may not be sufficient to ensure safety as usually we do not have exhaustive and fully relevant data. Nevertheless, as any critical system, an AISCS needs to have well defined development methods from its design to its deployment and qualification. This requires a complete tool chain ensuring trust at all stages, as: 1. Specification, knowledge and data management, 2. Algorithm and system architecture design, 3. AI functions characterization, verification and validation, 4. Deployment, particularly on embedded architecture, 5. Qualification, certification from a system point of view.
All that demands a sound and tooled AI engineering methodology that encompasses, with objective of trustworthy AI algorithm engineering, data engineering, knowledge engineering and AI system engineering by addressing the issues described above. Academic research already proposes solutions towards AI certification3, industry should take over now (see Fig. 4). At French national level major industrial players in the fields of Automotive, Aeronautics, Defense, Manufacturing and Energy (Air Liquide, Airbus, Atos, EDF, Naval-Group, Renault, Safran, SopraSteria, Thales, Total and Valeo) with the support of academic partners (CEA, INRIA, IRT Saint Exupe´ry and IRT SystemX) are collaborating together to address such issues through the French National Program “Confiance.ai” (https://www.confiance.ai/). Based on the specifications described above, this program aims to bridge the gap between AI Proof of Concepts and AI deployment within critical systems toward certification by providing an interoperable engineering workbench to support AI processes and practices through methods and tools during the overall lifecycle of the AI-based system.