Verification and validation of software and systems is the essential part of the development cycle in order to meet given quality criteria including functional and non-functional requirements. Testing and in particular its automation has been an active research area for decades providing many methods and tools for automating test case generation and execution. Due to the increasing use of AI in software and systems, the question arises whether it is possible to utilize available testing techniques in the context of AI-based systems. In this position paper, we elaborate on testing issues arising when using AI methods for systems, consider the case of different stages of AI, and start investigating on the usefulness of certain testing methods for testing AI. We focus especially on testing at the system level where we are interesting not only in assuring a system to be correctly implemented but also to meet given criteria like not contradicting moral rules, or being dependable. We state that some well-known testing techniques can still be applied providing being tailored to the specific needs.
Because of the growing importance of AI methodologies for current and future software and systems, there is a need for coming up with appropriate quality assurance measures. Such measures should come up with certain guarantees that the resulting products fulfill their requirements, e.g., provide the requested functionality and safety concerns. Providing guarantees seem to be essential in order to gain trust in AIbased system solutions. In particular, in autonomous driving to mention one more recent application area of AI, we have to establish a certification and homologation process that assures an autonomous vehicle to follow given regulations and other requirements.
Because of the fact that artifacts making use of AI technology are themselves systems, the question is whether it is possible re-use ordinary testing methodologies and to adapt them for providing means for certification and homologation. In particular, besides components like vision systems relying on machine learning, there are other components that do not rely on any AI methodology. In Figure 1 we give an overview of the architecture of such a system comprising the
AI component and other components implementing functionality like providing user interfaces or database access. In addition, such system rely on a computational stack where we also have to consider the operating system, firmware, and even the hardware for verification and validation purposes. As a consequence, we have to consider verification and validation of the whole system for quality assurance.
In a previous paper
We organize this paper as follows. In the next section, we discuss the system testing challenge in detail. We focus on different aspects of testing to be considered and refer to related literature. Afterwards, we present three approaches of systems testing that have been proven to find faults when testing systems using AI techniques. Finally, we summarize the obtained findings.
As depicted in Figure 1 systems comprising AI methodology
also rely on other components providing interfaces and
functionality, as well as runtime support including operation
systems, firmware, and hardware. As a consequence, we have to
consider testing as a holistic activity that has to take care of
all different parts of the whole system. In particular, we have
to clarify what to test and how to test. For example, a
logicbased reasoning system comprises a compiler for reading
in the logic rules and facts, and the reasoning part. Hence,
we have to test the compiler and the reasoning part first
separately and afterwards together in close interaction. The
compiler can be tested, for example using, fuzzing where
more or less randomly generated inputs are generated (see
e.g.,
If using fault injection, we are interested in how systems
react on inputs that occur in case of faults. When using
invariants like the order of rules, we do not test all aspects
of reasoning. Hence, in order to thoroughly test such
systems, we need to understand what to test in order to identify
shortcomings of underlying testing methods to be used.
Besides this and more specifically to AI methods, we have to
provide some measures that at least indicate the quality of
testing. For ordinary programs, coverage (e.g.,
Let us consider testing neural networks as an example. Neural networks are trained using a set of examples and evaluated afterwards. Evaluation is used for assuring that a network reaches a given quality of the prediction outcome. The set of examples used for training and evaluation have to be distinct. The question is now whether this evaluation is good enough for replacing further testing effort. The answer is no, because but not only of adversarial attacks (Su, Var1Note that besides code coverage there are other coverage definitions used like test input coverage, combinatorial coverage, etc. (a) (c) (b) (d) gas, and Sakurai 2019) that lead to misclassifications even in case of small input variations. Other reasons for misclassifications are the use of a training data set that is not covering all different examples, and other aspects like the distribution of examples. Furthermore, note that variations of the appearance of objects in the real world exists often. In Figure 2 we depict different images of the traffic sign ”do not enter” ranging from a bend to occlusions because of stickers attached. An autonomous car would require always to handle these case, and it is very unlikely that we really have all of such cases represented in the training data set. Moreover, if so, we still would have misclassifications occurring, requiring to assure that there is no unwanted effect on the behavior of the overall system.
There is plenty of literature regarding different testing
approaches for neural networks, e.g.,
In the case of neural network we may also ask whether
classical coverage or mutation score used in ordinary
software engineering can be used as quality measure when
testing a current neural network implementation.
Although, we may need to live with the challenge that we
cannot completely tests certain system parts and that there
is always a critical case where the AI part of a system may
deliver a wrong result, the further question is whether this
establishes a problem for the whole system. The answer in
this case is no, providing that the system itself is able to
detect this critical case and to react appropriately. For example
in autonomous driving, we may make use of more than one
sensor for obtaining information regarding objects around
the vehicle and use sensor fusion to obtain reliable
information. We only need to assure that the whole system interacts
with the environment in a way that is dependable and
fulfills our requirements including ethical or moral
considerations. Hence, identifying critical scenarios between the
system and its environment seems to be a crucial factor of
testing AI-based systems
Moreover, it seems also of importance to consider that critical scenarios often originate from different settings that have to occur at the same time. One issue, e.g., missing a certain traffic sign may not lead to an accident, but in combination with other issues would.
We summarize our discussion in the following position: Position 1 Testing aims at identifying interactions between the system under test and its environment leading to an unexpected behavior. When testing systems utilizing AI, we have to consider testing all parts of a system including the one with and the one without AI as well as their interactions. Evaluating performance characteristic of implemented AI methodology may not be sufficient for assuring meeting quality criteria.
Most of testing is performed during development of systems before deployment. In some cases certification (or even homologation), i.e., the formal confirmation that an application, product or system, meets its required characteristics, is needed. In case of AI technology we are interested that the system fulfills dependability goals like safety but maybe also given ethical or moral rules. For example, we want a conversational agent or a decision support system not to be racist or sexist. Furthermore, because of the fact that the system’s underlying software is updated regularly in order to cope with changes required because of bugs or improved functionality, there is a need of carrying out any certification regularly as well. For example, in autonomous driving we have to assure that a new software update is not going to lead to an unsafe system. However, regression testing may require a lot of effort or come with high costs, which may be reduced when automating testing.
Hence, automating at least part of certification may be a future requirement. But how can certification of AI be carried out? What we need is a process where we identify what we want to achieve, and how this can be checked (or tested)? How can we come up with certain parameters justifying that testing is appropriate? We shall also think about the methods for checking, their limitations, and how to assure that the methods can guarantee (with respect to a given certainty) that the system fulfills requested needs. However, in any case in order to bring AI technology into practice, we have to convince customers that the systems are not of harm. Certification that takes into account such customer’s considerations as well as regulations provide the right means for further supporting the delivery of AI technology into practical applications we are using on a daily basis.
It is worth noting that there are many initiatives like the
ethics guidelines for trustworthy AI
Position 2 There is a need for well-defined certification and homologation processes for AI-based systems that ideally can be carried out in an automated way. Such certification and homologation processes shall rely on existing guidelines considering all aspects of trustworthy AI.
When we want to carry out certification at least partially automated we may rely on testing. Hence, we have to state the question whether existing testing techniques can be used for confirming that an AI-based system fulfills regulation and other rules and expectations. This includes besides testing functionality the degree of fulfilling generally agreed ethical and moral rules. In the following section, we introduce three techniques that can (at least partially) serve this purpose.
As discussed there seems to be a need for testing the whole
system considering functional and non-functional
requirements including moral and ethical rules. For testing systems
at the system level black-box approaches are used that do
not consider the internal structure. Various methods with
corresponding tools have been proposed including
modelbased testing (MBT)
The focus on modeling the environment of the system in order to obtain test cases is somehow different to ordinary MBT where a model of the system is used for test case generation. Changing from modeling the system to modeling the environment is necessary for finding critical interactions between an AI-based system and its environment. Moreover, in this kind of testing we are not interested in showing that an implementation works accordingly to a model, but is capable of handling arbitrary interactions that may not be foreseen during development.
In contrast to MBT, CT has been developed to search for
critical interactions between configuration parameters and
inputs. It has been shown that CT can effectively detect
faults in many different kinds of software
The question is whether we can also apply CT for AI
testing? In
CT and also search-based testing applied to test autonomous and automated driving functions always has to fulfill the property that no crash with another car or even a pedestrian occurs. In this context closeness to a crash is often represented as the time to collision (TTC), where 0 means that a crash occurs. Usually, in many applications positive but small TTC values may also be considered as unwanted. When testing in the case of the automotive domain including autonomous driving, we can always rely on the TTC for judging whether a test case passes or fails.Hence, there the test oracle can be automated using the TTC, which is not always the case when testing AI. We, therefore, require other means for dealing with the oracle problem, i.e., providing a function that allows to distinguish passing executions of programs and systems from failing ones.
The objective behind metamorphic testing
Position 4 Metamorphic testing seems to be of use for implementing the test oracle problem of AI systems allowing to identifying contradictions with requirements, which may include ethical and moral considerations.
There are more system testing approaches that can be also adapted to fit the purpose of AI testing with the objective of assuring safety of AI-based systems and software. However, we have identified approaches where there is experimental evidence that they could be effectively used for testing AIbased systems. These approaches may also fit into certification and homologation processes. For this purpose, certain measures have to be developed that can be used for deciding when to stop testing in cases no failing test case could be obtained.
Moreover, the presented methods and techniques for testing AI-based systems have disadvantages. They are mainly focusing on quality assurance of the overall system and not its comprising parts. For example, the CT approach considers a model of the environment, which works as basis for obtaining the CT input model. The approach is testing whether certain interactions of the CT with its environment reveal a fault, and in case of automated or autonomous driving a crash, but does not consider any knowledge regarding the SUT’s internal structure or behavior. Finding out the root cause of any misbehavior within the SUT might be complicated. Moreover, we are not able to make use of quality assurance measures like code coverage or mutation score for the particular test suite. Furthermore, CT like MBT requires to concretize the abstract test cases computed using these testing methods. This concretization step cause additional effort and has to be done carefully in order to come up with good test cases that can be executed and most likely reveal a fault.
In case of metamorphic testing it is essential to define the metamorphic relations, which cause additional effort and influence the ability to work as a good test oracle. There are maybe metamorphic relations leading to test cases a SUT can easy fulfill only allowing to test a fraction of functionality. In such cases metamorphic testing would not lead to tests covering most of the functionality and, therefore, can be considered as incomplete. Search-based testing requires to implement a search procedure using a function allowing to estimate the quality of a current test, e.g., the ability of a test revealing a fault. Again this requires additional effort and costs. It is worth noting that in some cases random testing, i.e., generating test inputs using a random procedure, also provides fault revealing test cases requiring even less time than search-based testing at almost no additional costs.
In this position paper, we focused on providing an answer to the question whether there exists testing techniques that can be efficiently used for checking that a software or system comprising AI methodologies fulfills requirements including also moral and ethical rules, and regulations. We also discussed the involved challenges of testing where we identified also shortcomings that arise when only focusing on specific parts and not providing a holistic view. Finally, we introduced several testing methods that have been developed in the context of testing ordinary systems and elaborate on their usefulness in the context of AI-based systems. Searchbased testing, combinatorial testing, and metamorphic testing seem to be excellent candidate for this purpose and may also be of use for automating certification and homologation processes for AI applications.
However, further studies have to be carried out. For CT more experiments making use of other autonomous and automated functions have to be considered. Moreover, we require to come up with certain measures of guarantees for the computed test suites. Parameters of CT like the combinatorial strength maybe sufficient but in the context of AI-based systems there is no experimental evidence. For metamorphic testing we further need more use cases and experimental evaluations making use of AI-based systems. In the case of chatbots and also logic-based reasoning metamorphic testing has already been successfully applied. However, there is a need to show the usefulness of metamorphic testing also in other applications where AI technology is a central part.
The research was supported by ECSEL JU under the project H2020 826060 AI4DI - Artificial Intelligence for Digitising Industry. AI4DI is funded by the Austrian Federal Ministry of Transport, Innovation and Technology (BMVIT) under the program ”ICT of the Future” between May 2019 and April 2022. More information can be retrieved from https: //iktderzukunft.at/en/ .