Online monitoring is an architectural pattern well-known to safety engineers, but it had to be adapted to AI technologies. In this paper, an innovative multi-time scale online monitoring architecture is presented. The main idea is to combine several monitoring timescales - PresentTime Monitoring (PTM), Near-Past Monitoring (NPM), and Near-Future Monitoring (NFM) on different monitoring assets (inputs, internal states, and outputs of the AI model) to ensure a high anomaly detection rate by design of the online monitor.
In the industry, it is commonly established that
the main objective of online monitoring of AI
models (also called Run Time Assurance in [
To tackle the topic of monitoring AI models,
some works started to define a taxonomy of
anomalies that are specific to AI technologies [
The context of the online monitoring function is described in Figure 1 below.
The AI-based product consisting of one or several integrated AI models is depicted in the black oval. This product can be an item (i.e., a component), or a subsystem of an entire system. The generic term "product" is used in the following. The product receives at inference time operational data from sensors. Above the product, in a white oval, the online monitoring item receives both the external inputs and outputs of the product, as well as some information about the internal states of the product using pre-designed probes placed in the product's software code or hardware. At the top of Figure 1, an item in charge of continuously collecting all relevant operational data is usually required to feed a complementary offline monitoring function. The offline monitoring function may have several objectives according to the use case such as (but not limited to): (i) calculating offline metrics, (ii) fine tune some of the online monitor parameters, (iii) detecting data and concept drifts, and (iv) act as a hypervisor of the online monitor. At the bottom right-hand side of Figure 1, a controller is responsible for synthesizing the output produced by the product and the verdict of the monitor to compute, based on certain business logic (that is in general specific to the use case), the final output, which is so-called the “safe output”.
To illustrate the product to be monitored, consider the very simplified didactic example in Figure 2, which represents a linear physical phenomenon = ( ) = + to be approximated by an AI model ̂ = ̂( ), where tk is the system clock which also clocks the monitoring item (at each time tk the device acquires data to produce a verdict; 0 is the product start-up time). Let’s also assume that system requirements specify a set of properties to be satisfied by the product in operation. These properties are business-driven and thus specific to each use case. In general, these properties may be (but are not limited to) (i) functional properties related to the nominal expected behavior of the product like performance requirements, (ii) safety properties identified by safety risk analyses, (iii) security properties determined by security risk analyses, (iv) explainability properties coming from human factor analyses. To keep the logic of a simplified didactic example, consider that there is only one general property materialized by robustness bounds depicted by the green dashed segments in Figure 2. The area bounded by these two green dashed segments defines the validity domain ℧ of the product output ̂ . The very simplified general safety property2 can therefore be expressed as follows: ∀ ≥ 0, ̂( ) ∈ ℧ (1) Regarding the design of the product, since sufficient data were collected and are available to characterize the physical phenomenon to be modeled, it has been decided to use Machine Learning (ML) technology to design the product (e.g., using an artificial neural network). The ML model is obtained after several iterations of learning and is depicted by the blue curve in Figure 2. It is deliberately not perfect. Indeed, it is possible to observe several operating points of the ML model output ̂ fall outside the validity domain ℧ and do not satisfy the safety property (1). The online monitoring function aims to detect all operating points that violate the system properties – let’s call them by the generic term anomaly in the rest of this paper. To be efficient, the online monitor should ensure a sufficient anomaly detection rate, and this is precisely the ultimate goal of the multi-scale monitoring framework which is the main contribution of this paper.
The principle of multi-scale monitoring is described in Figure 3. It consists in combining several monitoring timescales: monitoring of the product at the present time, monitoring over a configurable time window in the near past, and 2 In section 5 (Application), an industrial use case is presented with more complex properties to be monitored. monitoring over a configurable time window in the near future. To illustrate the combination of these three different timescale monitoring functions, let us continue the discussion on the trivial example of Figure 4. At time tx, the ML model output ̂ overpasses the robustness boundaries, and it is expected that the Present Time Monitoring (PTM) will be able to detect such abnormal behavior. Between times tx and ty, it is possible to observe that ̂ starts to unexpectedly oscillate. It is an unintended behavior that could be a precursor of a failure of the ML model. Since this oscillation phenomenon should be observed and confirmed on several clock cycles, it is expected that the Near Past Monitoring (NPM) will be the appropriate monitoring timescale to detect such oscillation anomaly. At time tz, ̂ has an abrupt trend that will make it overpass the robustness boundaries at the next clock cycles. Here, the Near-Future Monitoring (NFM) is the most appropriate monitoring function to detect such potentially abnormal behavior since it is based on trend analysis.Through this didactic example, one can observe that an efficient combination of these three different monitoring timescales NPM, PTM, and NFM allows one to detect several classes of anomalies and to achieve this by designing a high online detection rate when the AI model is in production.
In the previous sections, a first design principle has been presented through the new multitimescale monitoring framework that aims at increasing by design the anomaly detection rate. However, there are many other design principles of online monitors that are important as well. Below is a synthesis of the main industrial design principles collected and formalized by major international industrial groups within the frame of the French research program Confiance.ai3. All these design principles are not detailed in this paper since each of them would require a full technical paper to be comprehensively presented. Design Principle 1: The monitoring function should by design ensure completeness of anomaly detection while minimizing false alarms Design Principle 2: The sophistication of the monitoring function should be proportionate to the criticality level of the AI function Design Principle 3: The monitoring function should be smart to manage complexity and performance issues Design Principe 4: The monitoring function should not have any safety adversarial common mode of failure with the monitored AI function Design Principle 5: The monitoring function itself should not has an unacceptable impact on the system safety and security (innocuity)
In the next section, an industrial use case from the automotive domain is presented to illustrate some of the concepts presented earlier.
The application used to present some results related to multi-timescale online monitoring is called the Renault Welding Use Case.
The industrial context is a plant producing mechanical components used for the ground connection of motor vehicles and the mechanical parts of interest in this use case are only the parts of the rear axle. During the manufacturing process shown at the top of Figure 5 (see Operational Platform (OP) #120), metal parts are welded together. The mechanical quality of the final component depends on the quality of the weld.
Until now, a systematic inspection of the weld is carried out by a specialized human operator on a screen like the one at the bottom of Figure 5 (see Display OP#120). The screen displays different 3 See www.confiance.ai involving more than 40 partners including large industrial groups such as: Airbus, Air Liquide, Atos, Naval Group, Renault Group, Safran, Sopra Steria, Thales, Valeo, and others (full list on the web site). photos of the same weld taken by different cameras from different angles. Based on its experience, the human operator classifies the weld as “compliant”, “not compliant” or “unknown” (see examples of welds in Figure 6). This last status "unknown" leads to a further deeper technical evaluation of the manufacturing process.
(a) Compliant Weld (b) Non-Compliant Weld
In practice, the overwhelming majority of welds are compliant (robotized welding), and from a human factor perspective, this situation is likely to decrease the attention of the operator in charge of quality control. To mitigate this risk, Renault launched a project to develop an AI-based system to assist the operator in charge of the quality control of welds as depicted in Figure 7. performs an automated preliminary conformity assessment of the weld quality. The WCM is developed using supervised ML technology based on labeled datasets containing historical data of compliant and non-compliant welds. The design details of the WCM are not important in this paper since it is considered a black box by the online monitor that only looks at its inputs and its outputs and not at its internal states as shown in Figure 7). The WCM provided by Renault reaches very good performance (measured with an f1 score) but only on a given domain, called ODD, that is characterized according to operational parameters. Based on a dedicated study of the ODD done with Renault representatives, two operational parameters have been considered in this study as the most impacting the performance of the ML Model (and therefore of the correctness weld conformity classification): (i) image brightness and (ii) image blur. Thus, the 2 properties to be monitored are expressed as follows: ∀ ≥ 0, ∈ ℧
ℎ ∀ ≥ 0, ∈ ℧ (2) (3) Where is the image received by the WCM at time tk and ℧ ℎ and ℧ are respectively a projection of the full WCM ODD on the two targeted operational parameters – i.e., image brightness and image blur. Besides, ℧ ℎ and ℧ are not calculated theoretically, they are determined based on test campaigns with augmented data as illustrated for the brightness ODD in Figure 8.
The AI-based product in Figure 7 is called Welding Classification Model (WCM) and it Once ℧Brightness and ℧Blur are determined, it is possible to develop dedicated monitoring functions to detect Out-Of-Distribution (OOD) input images since we can observe in Figure 8 that the f1 performance score of WCM drops sharply outside ℧Brightness (depicted with the white arrow at the top of Figure 8). And it is the same for the blur (not represented in the paper to avoid overloaded information). The rules-based design of the online PTM monitoring functions for brightness and blur OOD detection are detailed in Tables 1, 2, and 3. Examples of anomalies detected in the Welding use case are represented in Figure 10 (brightness anomalies) and Figure 9 (blur anomalies). The performance of the developed PTM monitoring functions has been evaluated by the LNE4 which is an independent partner of the Confiance.ai Program specializing in calibration, testing, and certification under the trusteeship of the French Ministry for the Economy and Finance with oversight for Industry.
LNE randomly selected 11,000 images for the evaluation set, and identified the following evaluation metrics: Analysis of the classification of the monitor compared to the noise for each image: True positive: the image has a medium noise or important noise, and the monitor raised an alarm True negative: the image has a slight noise, or no noise, and the monitor did not raise an alarm False positive: the image has a slight noise, or no noise and the monitor raised an alarm False negative: the image has a medium noise or important noise, and the monitor did not raise an alarm Using these four values, the precision, recall, and f-measure are computed Precision: total of true positives by the total of detected positives (true and false) Recall: the total of true positives by the total of real positives (true positives and false negatives) F-measure: harmonic mean of the precision and recall
The recall metric is very important in domains such as automotive quality controls where you want to minimize the chance of missing positive (i.e., missing to detect a non-compliant weld) by predicting false negatives (i.e., a non-compliant weld is predicted as a compliant one and there is no alarm sent by the monitor). These are typically 4 https://www.lne.fr/en 5 Vertical motion blur cases where missing a positive case has a much bigger safety impact than wrongly classifying something as positive.
These evaluation results show that the rulebased PTM OOD functions have a good or a very good recall (90% recall 100%). However, the precision results (and thus the f-measure scores) show that the number of false positive alarms is still high and needs to be reduced in a further version of the monitoring functions.
There is no result presented in this paper on NPM and NFM functions since the development of these monitoring functions are in progress within the Confiance.ai Program. The results of ongoing research on NPM and NFM will be published in future papers, as well as the results of the integrated multi-timescale monitor combining PTM, NPM, and NFM into a single monitoring item.
This work has been supported by the French government under the "France 2030” program, as part of the SystemX Technological Research Institute within the Confiance.ai Program (www.confiance.ai).
A special thanks to Guillaume BERNARD from LNE who contributed to the independent evaluation of the monitor, and to Dominique TACHET & Meriem LAFOU from Renault who provided valuable support on the Renault Welding use case.