More and more applications rely on Machine Learning (ML) techniques, e.g., to automate software engineering. Like other applications, they need to be tested and validated. Testing ML-based software difers from testing software which do not rely on AI and ML: non-determinism, lack of oracle, high dependence on training and evaluation data are hard points. The problem is even more complicated when humans are involved in the process. In this paper, we analyze the problem of evaluating OCE, a human-centered intelligent system based on reinforcement learning that automatically builds user-tailored software. We present a test environment composed of two tools which are based on the notions of “scenario” and “ideal assembly”: OCE Scenario Maker to edit scenarios and OCE Scenario Runner to automate and repeat their execution.
1. Introduction mechanism. It describes the principles of our solution and the tooling we have developed.
More and more applications rely on Machine Learning Sec. 2 summarizes the principles of Opportunistic Soft(ML) techniques, namely to automate software engineer- ware Composition and our needs for testing. Sec. 3 ing. As part of our project on Opportunistic Software analyzes the main issues of testing ML-based systems Composition, we are designing an intelligent solution in particular with humans “in the loop”, then focuses based on reinforcement learning (RL) to automatically on OCE. Sec. 4 briefly presents several tools that allow build applications in ambient contexts, and we are de- testers to carry out experiments on ML-based solutions, veloping several working prototypes of an Opportunistic then describes the principles of our solution, the protoComposition Engine (OCE). With OCE, the user is put in type tools we have developed and integrated with OCE, the loop and provides feedback on the built applications. and a demonstration. Sec. 5 concludes and discusses
For ML-based solutions, as with any software, develop- some open issues. ment teams need to address validation issues. Testing is a potential solution. However, testing human-centered MLbased systems is quite diferent from testing traditional 2. Background software, i.e., those which do not rely on ML techniques.
For our part, we need to assess the prototype versions 2.1. Opportunistic Software Composition of OCE in diferent use cases. Beyond the common problems posed by the evaluation of ML-based systems, our issues lie in the dynamics of the learning environment, including the human user. To answer, we have developed a tooling that supports the definition of test scenarios with simulated users, their execution and the evaluation of the results, which use is illustrated in a video1.
The purpose of this paper is to present our work on
testing and evaluating OCE, more specifically its learning
Today’s users live in ambient environments invaded by
connected objects. These environments are open,
complex and dynamic: at any time, objects may appear
and others may disappear. Besides, the user needs may
change depending on the situation. One of the issues
is how to manage the variability of such unpredictable
environments, and propose relevant services to the user
[
To tackle this issue, Opportunistic Software
Composition is a human-centered approach based on
reinforcement learning (RL) [
from other components. This makes software components easier to assemble, replace and reuse. Technically, they can be assembled by connecting their services, the assembly implementing an application.
The goal of opportunistic software composition is to provide the user with the right application at the right time, but without them having expressed needs and preferences due to the dynamics and unpredictability of the environment, which make their specification dificult. User-adapted applications are designed in bottom-up way and emerge from the environment, relying on automatically on-the-fly learnt knowledge about the user’s needs and preferences.
To realize opportunistic software composition, an Opportunistic Composition Engine (OCE) has been designed: every service is locally managed by an “agent” that learns by reinforcement the connection preferences according to the context and decides on its own connections. Building an application consists in three steps, called an OCE cycle, and involves the user:
ment teams to focus on testing and evaluation issues.
However, testing ML-based systems is quite diferent from testing traditional software, i.e., those that do not rely on AI and ML.
A first point is that learning is assessed indirectly.
When testers evaluate a ML-based solution, they seek to
evaluate the correctness of the learning mechanism. But
1. OCE probes the ambient environment to detect learning can hardly be isolated from the rest of the
applithe available components. cation, which is tested as a whole including the decision
2. According to their learnt knowledge on the user’s process. Therefore, the observed outputs are those of the
preferences, the agents decide on their connec- whole application, not those of the only learning process.
tions to others to collaboratively build up an Secondly, testers struggle to deal with randomness and
assembly plan that defines an application and non-determinism [
or is there a design or implementation error [
fore, to get significant results, experiments must be retion.
peated a certain number of times to reduce the efects of
User’s actions in the last step are used as feedback non-determinism.
for OCE. From this feedback, a reinforcement signal is Moreover, in many cases, testers can not predict the
computed from which OCE’s agents learn by reinforce- produced outputs for specified inputs: there are not
alment the user’s needs and preferences. As the cycles are ways oracles [
Readers who wish to know more about OCE and ICE input data and observe the diferences in the outputs [
Moreover, OCE is parameterizable, and the parameter the incremental building of knowledge. In this case, the values have to be adjusted. Among the parameters, some system learns over time and makes decisions depending are ML-specific (for instance the amount of exploration on both its current knowledge and the state of the envivs. exploitation), others are user-specific and must be ronment. Thus, the dependence on the environment and tailored to the individual: for instance, the coeficient its variability over time can lead to outdated or obsolete that defines the user’s preference for components or ap- learnt knowledge. plications not yet encountered.
Human-centered ML-based systems are interactive
systems. In the field of interactive systems, it is common
to carry out test campaigns involving real or potential
users to evaluate interaction criteria, such as usability
[
To alleviate these costs, designers often model
synthetic users, e.g., personas, and simulate their actions.
This raises some challenges:
• The potential introduction of biases [
In the case of interactive ML-based systems [
to carry out experiments on ML-based solutions. We can
cite the following:
• Arcade Learning Environment (ALE) [
The goal of a learning agent is to maximize the
scores obtained on each game. ALE thus proposes
evaluation metrics to compare and understand
the performances of learning agents on the
different games.
• OpenAI GYM [
For each experiment, Cogment generates logs to allow the tester to analyze the results.
OCE runs a distributed human-centered RL process where the human user is an integral part of the learning environment: as they accept, modify, or reject an assem- These tools ofer a wide range of features to perform bly proposed by OCE, they provide feedback that OCE experiments. The Cogment framework is fairly new but transforms into a reinforcement signal. This way, learnt seems to be the most attractive since it focuses on the knowledge is personalized and difers from one user to an- presence of humans in the learning process and fits well other. It also depends on the dynamics, whether it comes the human-centered reinforcement learning paradigm. from the ambient environment or from the changing However, generally speaking, the gap between what user’s needs and preferences depending on the current these tools provide and the needs for testing OCE seems situation and time. OCE decisions are then dependent on quite important and expensive to fill. Indeed, in order the knowledge gained. to test OCE using these tools, it would be necessary to
Thus, the evaluation of the OCE learning process faces make the predefined environments and agents interoperthe issues of both ML-based and interactive systems: in able with those of OCE in particular to use the available particular, non-determinism and the challenge of assess- benchmarks. Among the problems, some seem dificult ing the quality of OCE’s decisions on the one hand, de- or even impossible to address such as taking into account pendence on the user, their profile and the variability of the distribution nature of OCE learning. However, some their needs and preferences on the other hand. In ad- concepts could be useful for the evaluation of OCE, e.g., dition, the multiple forms of the ambient environment the ALE evaluation metrics. must be taken into account with the variety and number of components, as well as the dynamics.
each training and assessment cycle, both which components populate the environment and the ideal assembly.
For that, OCE Scenario Maker is a tool composed of: • A library of dummy software components. • A component creator that allows the tester to define dummy components by indicating their name and associated services, and add them to the library. • A cycle creator that allows the tester to specify a cycle by choosing the participating components and expressing the ideal assembly (i.e., giving a list of connections between components).
In practice, OCE Scenario Maker is a single-page Web application. A GUI (see Sec. 4.7) assists the tester to graphically manipulate the scenario elements (components and connections). Once defined, the test scenario description is saved in a JSON file. our own tooling to automate the testing of OCE learning process, which consists of a pair of tools: • OCE Scenario Maker, which supports the definition of tests, • OCE Scenario Runner, which allows to automate and repeat their execution.
narios with the ambient environment and the user’s preferences and their possible variations over time, (ii) a way to repeat tests any number of times, and (iii) scores that measure the quality of OCE decisions.
A benefit of these tools is that they allow to carry out test campaigns more easily and at a lower cost, without having to develop concrete components and involve real users.
the results that could be provided by OCE: each cycle is described both by a set of software components of the ambient environment and the user’s reaction to the proposal that OCE would make in this context.
To define the user’s reactions without knowing the results provided by OCE, we introduce the concept of ideal assembly: an ideal assembly is the one that the user would prefer in a given context. Ideal assemblies model the user preferences in the diferent situations.
A scenario includes training cycles followed by assessment cycles. In a training cycle, the ideal assembly allows OCE to learn the user’s need and preferences in the stated context. In an assessment cycle, the ideal assembly is used to verify that OCE has correctly learnt: to do so, a distance between this ideal assembly and the proposition of OCE is computed.
Defining a test consists thus in specifying a sequence of cycles with a variety of components, a more or less important number of them, and a more or less dynamic ambient environment. By specifying the ideal assembly for a cycle, the tester behaves as an oracle. Besides, specifying ideal assemblies allows to simulate diferent user profiles: users with more or less stable preferences, more or less open to new applications. . . Note that the selection of adequate training and assessment cycles (i.e., the design of test cases) is another challenge that we do not address in this paper.
In the following sections, we first present OCE Scenario Maker and OCE Scenario Runner. Then, we de- 4.6. Software architecture scribe the architecture of the testing environment which associates OCE Scenario Maker, OCE Scenario Runner, and OCE (the engine) to run and test the latter.
Maker, it has to be executed to carry out automatic testing. OCE Scenario Runner is a Java desktop application, which handles these experiments and their repetition as many times as required to reduce the impact of nondeterminism.
To start an experiment, several parameters must be set such as the test scenario JSON file produced by OCE Scenario Maker, OCE parameters, the version of OCE and the number of repetitions of the experiment. For each cycle, OCE Scenario Runner compares the assembly proposed by OCE and the ideal assembly and calculates a Jaccard Similarity index between the set of the connections proposed by OCE and the one of the ideal assembly; then, it computes an average score over all the cycles.
This provides a measure of the distance between OCE’s propositions and the ideal assemblies, and so indicates whether the OCE decisions make relevant applications emerge according to the learnt user’s preferences. This measurement only makes sense for the assessment cycles as the previous cycles are used to create knowledge. It therefore makes the tester able to analyze and understand how OCE behaves.
Figure 1 shows how OCE Scenario Maker and OCE Scenario Runner fit in OCE runtime environment to create the testing environment. In the production environment, OCE interacts with the ambient environment by probing available components. Then, according to its knowledge about the user’s preferences and needs, it plans an as- The video shows how the tester defines the componentsembly which is displayed on ICE. Last, through ICE, based ambient environment and the ideal assemblies cythe user gives a feedback to OCE that is converted into cle by cycle, then runs the resulting multi-cycle scenario reinforcement signals for OCE learning agents. and gets similarity scores to check the relevance of OCE’s
OCE’s modular architecture allows to seamlessly re- decisions. place ICE and the ambient environment with OCE Scenario Runner, thus allowing simulation and automation of the interactions with the user and the environment. 5. Conclusion When running a scenario, at each cycle, OCE retrieves the set of the available components provided by OCE Scenario Runner, then proposes an assembly. Finally, OCE Scenario Runner returns the ideal assembly that is used by OCE to create feedback for its agents. Therefore, OCE Scenario Runner plays both the role of the ambient environment and the role of the user, in accordance with the scenario description produced using OCE Scenario Maker.
The following video2 shows the case of a tester who wants to check OCE’s behavior in a simple use case where new components appear dynamically and the user preferences are stable. In this test scenario, the user is virtually surrounded by devices like switches and lamps, a switch can be connected to a single lamp, and OCE builds applications from them as they appear or disappear. Initially, three lamps (but only two of them work) and a switch are available in the user’s surrounding environment, and the user prefers to use the first lamp. After a while, the third lamp works again and the user prefers to use it. An other component appears but do not afect the user’s needs.
an intelligent solution for automated software composition. We have presented both a reflection about testing human-centered ML-based systems and a pair of tools that we have developed to automate the testing of our Opportunistic Composition Engine OCE. When defining scenarios using OCE Scenario Maker, the tester behaves as an oracle by designing diferent situations to simulate the environment dynamics and the user behavior. Then, experiments can be executed and repeated using OCE Scenario Runner. Repetition smoothes out the results altered by the presence of randomness in learning and decision.
Of course, questions arise concerning the design of the test sets. For now, we only propose tools that facilitate specification, execution, repetition and analyze of tests, but not a methodology: how to design test sets and how to assert their relevance? As it is highly dependent on the users, one way would be to involve them more in the design, e.g., build scenarios based on observations of their behavior or ask them about their preferences in diferent situations. Another way would be to improve usability of our tools in order to involve end-users in the implementation and the execution of tests. These are tracks to be explored further.
Although our work is still in progress, we hope that our approach can inspire other practitioners in the field to develop tools for testing intelligent human-centered automated solutions.