Intelligent agents using reinforcement learning offer interesting capabilities for optimizing processes, products, and services in various branches of industry. Developing such applications at an economically viable level, however, is still a major challenge: Regardless of the potential offered by reinforcement learning, many solutions suffer from a lack of explainability. Deploying an application that makes unexplainable decisions is a potential risk to a project's success. Despite the ongoing effort, most reinforcement learning solutions remain a black box. In this position paper, we motivate to use concepts derived from human knowledge to unveil the inner workings of an intelligent agent on a more meaningful level without having to limit the algorithms themselves.
Reinforcement learning, a method to create intelligent
agents, is a promising area of Machine Learning (ML) and
Artificial Intelligence (AI). In supervised and unsupervised
learning methods, algorithms gather knowledge from
existing data and search for patterns, which in turn are applied to
new data. Thus, algorithms are fed by existing experiences
collected in the past. With reinforcement learning, agents
typically gather their own experiences by training in
simulations. Successful actions are rewarded and the agents aim
to maximize their profits by improving their strategies.
Several recent examples show the potential: intelligent agents
learned to play Chess (Silver et al.
The core of reinforcement learning is an optimization
process driven by random exploration and increasing
exploitation of knowledge from gathered experiences. The agent
more or less blindly stumbles through the various
potential actions, probing for success and reaching for higher
rewards. Prominent pitfalls are that agents find highly
rewarded strategies unintended by developers
Explainability of AI algorithms – often referred to as Exaplainable AI or XAI – is a major research topic, but most approaches have a strong focus on analyzing algorithms and the way data is processed. Unfortunately, we have to argue that the current state of the art still lacks the necessary progress to mitigate the risks resulting from unexplainable erroneous agent behavior. However, in this position paper we argue that techniques well-known in software engineering can be used to achieve a first level of explainability on a more meaningful level.
In this paper, we contribute to the emerging field of
software engineering for intelligent agents by describing how
concept-based engineering
We will first review related work with regard to reinforcement learning and XAI. We then introduce our engineering approach and describe thereafter how this can be used to explain an agent’s decisions and behavior. Finally, we conclude the paper with an outlook on future work.
We will know discuss related work in the areas of machine learning. First, we will focus on the development of intelligent agents and reinforcement learning. We then review recent approaches for Explainable AI.
Reinforcement Learning enables intelligent software agents
to optimize their behavior based on past experience. We
assume that the agent lives in a world characterized by an
underlying Markov Decision Process (MDP) consisting of the
tuple (S; A; P; R; ). The set S contains all possible states
and the set A all possible actions that the agent can choose.
Each state s 2 S and a valid selected action a 2 A is
followed by a transition to a subsequent state s0 2 S. While the
transition can be deterministic, in reality, uncertainty factors
may occur making it more probabilistic. P (s; a; s0) thus
describes the probability of a transition to the next state. Any
transition from a state s to a subsequent state s0 by
executing an action a may result in an immediate value given by
the reward function R, which can be sparse in some cases,
meaning that not every new state necessarily has an
immediate value. In many real-world problems, different steps need
to be taken before a particular outcome can be achieved, and
in order to be able to make long-term decisions accordingly,
the expected rewards usually have to be considered over
several steps. One possibility is to sum up the expected rewards
within a k finite horizon, but the result E hPtk=01 rti is then
limited by k, so that potential rewards are not considered in
subsequent states. Since there is usually no fixed k, it is more
common to use a discount factor 0 1 to represent the
expected future reward E [Pt1=0 trt] with an infinite
horizon
The assumption in an MDP is that the agent can directly observe the state s. In practice, the state variables are often only partially or indirectly observable, thus a Partially Observable Markov Decision Process (POMDP) exists and the agent has to estimate the actual state from its observations. The result is a set B of possible estimates called beliefs. It is possible to reduce the POMDP to an MDP by considering the set of possible beliefs themselves as a set of possible states in which the agent can find itself. The result is an MDP with a continuous state space described by (B; A; P; R; ).
Given a complex MDP or POMDP, with reinforcement learning a value function is used that calculates the immediate value of a state s and quantifies its value by taking into account the discounted expected rewards in the future. The value function is defined by V (s) = maxa Ps0 P (s; a; s0) [R(s; a; s0) + V (s0)]. Based on this function, the Q-function can be derived, describing the expected future discounted reward, given a state s and additionally an already selected action a. The Q-function is defined by Q(s; a) = Ps0 P (s; a; s0) [R(s; a; s0) + V (s0)].
Q-Learning is a method of reinforcement learning that
learns to approximate the Q-function. This is a kind of
mapping problem of the state to the corresponding Q-value,
which can be learned with supervised learning. In theory,
any function approximator can be used, such as decision
trees
Q-Learning is potentially capable of dealing with very
large and continuous state spaces. Especially if neural
networks are used, all the possibilities of this technique can
be used. For example, the processing of image data by
applying convolutional layers or temporal data using LSTM
cells or similar methods is possible. However, the action
space must be discrete and is limited in size
However, these methods often suffer from large variance
in policy gradients (Grondman et al. 2012), making them
potentially unstable
Reinforcement learning strategies require the exploration of new ways to gain experience from which the best potential actions can be approximated. Without prior knowledge, only random exploration can take place. With increasing knowledge about the environment, the agent might exploit it further to find the best possible ways to reach the desired goal. Exploration and exploitation are therefore important elements of intelligent agents. The challenge is to find the right trade-off between exploration and exploitation.
One of the core things that makes reinforcement
learning work is the Bellmann Equation
Having a good function approximation allows to deduce from data that have not been seen before. In contrast to supervised learning, however, reinforcement learning does not know the real target function. Instead of the real values having to be specified and labeled, the target function is tried to be inferred from observations of the real environment. This makes reinforcement learning more open to new solutions that are less influenced by human biases, but has two shortcomings: dynamic training data and the lack of a deterministic transition function.
The set of training data is not static, it is dynamically influenced by the learning itself. Each update may lead to different explorations and thus to a possibly different distribution of the data. Learning from dynamically changing observations is usually much more unstable than learning from a fixed set. Furthermore, especially in realistic scenarios, no deterministic transition function between the states of the environment can be assumed. This is due to various possibilities of uncertainties, such as unobservable variables or external factors. Beyond self-contained toy examples, it must therefore be assumed that the same action in the same observed state does not necessarily lead to the same result. Large updates due to fewer observations therefore also potentially lead to instability. An advantage of reinforcement learning here is that if the right hyperparameters are found, potentially good policies can also be learned in stochastic and noisy environments. However, finding the hyperparameters that are ideal for exploration is another challenge and the success of a reinforcement learning solution is potentially unstable depending on the parameters used.
Another challenge is the assignment of rewards. To be
able to learn successfully, the agent needs negative or
positive feedback after an action and reinforcement learning
works best with continuous feedback
The development of intelligent agents through machine learning techniques such as reinforcement learning offers great potential for future applications. On the other hand, especially on the engineering side of entire applications, a major challenge is to control the potential variance of the algorithms and to ensure that the final result meets the requirements. The lack of controllability is given by a set of intrinsically rooted properties of the underlying algorithms discussed in the last section.
Given the possible future significance of the procedures, however, there are already a number of possible solutions within the community which have to be mentioned. It has been shown that one promising way to address these problems is to decompose the core problem into smaller problems. This can be done in the context of reinforcement learning in two different ways.
The first possibility is a more architectural approach:
Hierarchical Reinforcement Learning. The core idea here is to
divide complex tasks into subtasks, which can be learned
more easily and reassembled later in further steps. From
a software engineering point of view, Hierarchical
Reinforcement Learning is a way of Divide and Conquer
using reinforcement learning, which is often very intuitive.
Especially in the area of robotics but also in other areas
this methodology has shown success and helped to
successfully implement more complex tasks
The second option is Curriculum Learning. This is not
only limited to reinforcement learning but has its origin in
the supervised learning methods (Bengio et al. 2009). The
core idea here is a decomposition of the learning process
itself. A model is first trained in a simpler environment and
then in increasingly complex environments. Like
Hierarchical Reinforcement Learning, Curriculum Learning has been
successfully applied and proven in many cases
Machine learning processes are gaining importance in the
development of software applications and are thus moving
more and more from research to practice and industrially
used applications. One of the big challenges today, which
results especially from the real world, is the
comprehensibility of machine learning models and the resulting decisions.
Many methods, especially in the field of deep learning, are
often referred to as black-box. Although they are
transparent at a technical level, their enormous internal complexity
makes them virtually incomprehensible. The
comprehensibility and explainability, however, is sometimes necessary
for reasons of acceptance or possibly also for legal reasons.
This leads to an increasingly important field of research in
the field of Explainable AI, which is also called XAI. The
XAI problem for models that would be considered as
blackboxes can be divided into three subproblems: Model
explainability, outcome explainability, and model inspection
The problem of model explainability is about training
another model that imitates the original model and is itself
explainable. Decision trees are one of the essential
models, which are considered to have a certain
explainability because the learned rules are directly readable.
However, this must be limited to simple trees and does not
necessarily apply anymore to random forests. But even
simple decision trees cannot be interpreted if they are
sufficiently complex. Due to the principally explainable
nature of the model, solving the problem by approximating
the actual model with a simple decision tree is an
essential approach in the literature. Here there are successful
examples in current
The problem of outcome explainability deals with the
explainability of a specific output for a given input. The
overall explainability of the model is of secondary importance
and does not necessarily have to be given. The most
common approach is to calculate a mask over the actual input
that identifies the subset that is mainly responsible for the
calculated result. Most of the approaches work on images.
These have the advantage that humans are particularly well
able to visually interpret and deduce more complex data
in this way. One way to determine the masking is to do
this via backpropagation, which is called Layer-wise
Relevance Propagation
Explainable AI currently works on the algorithm level, but there are other levels that are helpful. If you try to understand the actions of another human being, you don’t usually do this on the technical level of neuron connections, but try to understand how certain concepts and goals led to a decision. Using the possibilities of Hierarchical Reinforcement Learning and Curriculum Learning, this level of understanding can also be made possible for intelligent agents without having to severely limit the complexity and capacity of the underlying learning algorithms. What we present here is an idea of how we can better combine knowledge engineering and state-of-the-art technologies of machine learning, especially reinforcement learning, to achieve comprehensibility not only at a more technical level but also at a more meaningful human level. To better define this idea, we first define the context in which machine learning applications typically come into play and are developed in the real world.
Machine learning is always a solution in software
development projects if a certain aspect is not economical or good
enough to implement based on rules
If machine learning is used to solve these problems, expert knowledge is often no longer part of the solution. This makes sense because knowledge in these cases is usually not explicit, but what is discarded is that tacit knowledge can also be externalized. The result is not knowledge that can be described at the rule level, but more abstract knowledge that is also called conceptual knowledge. Conceptual knowledge is often too abstract to be transformed into rules, however, it is a powerful tool that can be used as a basis for decomposition, but is often simply underestimated and not used. empathising articulating tacit
This is especially true when talking about ideas of complex long-term strategies. There are some successful application examples in the field of ontologies.
In knowledge management, a well-known model for the
description of different knowledge generations and
conversions is the SECI-model (figure 1) presented in
Knowledge can be available in tacit and explicit form.
In our experience, the boundary is fluid; stakeholders
often have different intermediate states of knowledge. Since
methods of machine learning and artificial intelligence in
software development projects make sense when knowledge
is not available in a sufficiently explicit form to convert it
into acceptable rules at economically affordable costs, we
assume at this point that knowledge in machine learning
projects is, to a large extend, available to the stakeholders
as tacit knowledge. Implicit or tacit knowledge is gained
through the process of socialization, i.e. through
observations, imitations, practice and participation in formal and
informal communities, such as business environments
Since the idea of concept-based engineering is quite abstract, we want to show in the following a more concrete example of how this can look like in practice and how it can be used to better understand the behavior of intelligent agents. The example scenario is the optimization of a warehouse. There is a sales curve for the product to be managed, which is shown in figure 2. Due to various exogenous influencing factors, there are variances in the measured sales (several different warehouses with the same product serve as the basis here). Each week a delivery can be ordered to increase the stock again. However, there is a possibility that the delivery will fail due to delays, weather, strikes or other problems, despite the order being placed. Finding the optimal order quantity can therefore not simply be calculated using the expected sales curve, but must take into account storage costs and the risks of delivery failure. In addition, there are data on public holidays and the inclusion of the product in advertising flyers, which allow a more accurate estimate than just using the curve itself.
starts with a short creative workshop based on
Furthermore, we can also capture statements about the environments of the concepts to be learned: 1. Optimal Environment: The Optimal Environment does not have any delivery failures. This environment is the simplest imaginable. 2. Real Environment: The real environment has delivery failures. The consideration of these should tend to increase order quantities and converge to the order quantities in an optimal environment if no failures are expected.
What should be emphasized here is that all this knowledge is simple knowledge that does not even need much experience in this field and can be formulated more or less by anyone who is more involved with it. In our experience, such abstract knowledge is relatively easy to find in many environments.
Knowledge, especially at the conceptual level, is often formulated very informally. In the second step, this must now be transferred to the technical space. The optimization of stock levels can be formulated as a Markov Decision Process. At each time t there is a condition which is described by the current stock, the date, as well as by further information like coming holidays and inclusion in advertising flyers. The action area is continuous and consists of the selection of the requested goods. Goods are sold every step of the way. The sales quantity follows a distribution based on historical data. With a given probability, the delivery is canceled and the order is not implemented and must be balanced by the goods in stock or the subsequent order (this essentially results in the temporary order problem).
The question is, how are the identified concepts and environmental variations reflected on a technical level? There are different possibilities here. The following is a nonexhaustive list that should to be filled in the future research: Modified Reward Functions: The goal can be changed by another reward function Different environmental parameters: Individual characteristics of the environments can, for example, be switched off Modified observations: Some observations can be supplemented or removed Modified actions: The action space can be simplified or extended ...
In the example used, we can do the mapping as follows: Concept of full demand fulfillment: Since the sales volume is generated by a probability distribution, there is no upper bound value here. However, the concept can be represented by an environment in which the reward function weighs storage costs very low and costs for the nonfulfillment of demands very high.
Concept of complete sale: The same applies here with reversed weighting.
Concept of optimal balance: Here the weighting in the reward function is carried out realistically. In addition, the observation is enriched by the decision of the other two concepts.
Optimal Environment: This is the original environment with a different parameter: the default probability for deliveries is zero.
Real Environment: This is the original environment.
The result of the mapping of concepts captured from the
business point of view to the technical level is a
decomposition on two levels, which is shown for the example used
here in figure 3. First in the form of concepts, which
represent stand-alone or hierarchical tasks and further in the
form of different environments, which represent the
training ground. The first is what is trained in the literature for
example by Hierarchical Reinforcement Learning. The
latter is referred to in the literature as Curriculum Learning,
where the complexity of training environments is
increasing. Using these methods, we can now train the identified
concepts in a technical way. The result, in this case, are six
artifacts: In each case, one concept (agent) trained on the
optimal environment and in each case, the continued version
trained on the realistic environment. Since the goal of this
position paper is not the concrete selection and
configuration of the algorithms, the exact execution is not considered
here and it is assumed that the training has been
successfully completed in order to focus on the explainability part.
Details on how to train agents and machine learning
models through Hierarchical Reinforcement Learning and
Curriculum Learning can be found here
The resulting artifacts in the form of hierarchical sub-agents and intermediate training results according to different environments or curriculum steps. The individual models are still black-boxes, but the big advantage is that we know exactly how they interact or should have evolved. This fact can now be used at this point to understand the decisions made and to explain what otherwise would not have been possible without further effort. To do this is not further complex. It is only important to have an understanding of what the technical intention behind the individual concepts and training steps was. That is why it is so important to create them as described by a knowledge-driven process. With the idea of how the concepts are related based on the existing conceptual knowledge and when they are expected to interact differently or in other ways, questions can now be posed to the model and can be answered. We now show this in our example in two cases. 1. Question 1: Does the agent estimate delivery failures as probable? There are two versions of the final agent due to the decomposition that was performed. First the version trained on the optimal environment and then the version trained on the real environment (Curriculum Learning). We also know from human knowledge that as the probability of delivery failure increases, the order quantity must be increased in order to continue to meet demand. As a result, the difference between the two versions is an optimal measure for this question: q1(xt) = optimalbalancerealenv (xt) optimalbalanceoptimalenv (xt), where in this case a higher value describes a higher expectation of a delivery failure. 2. How certain is the agent that his order is actually being sold? One of the most important concepts identified was to be able to estimate what is approximately the maximum order quantity that must be placed to meet any demand. If the agent is sure he can sell everything, the more he must tend to actually place this maximum order quantity. So here we can formulate the following score: q2(xt) = f ulldemandrealenv (xt) optimalbalancerealenv , where a smaller value describes a higher certainty.
By using the identified concepts, we are able to visualize and represent the decisions of the intelligent agent at any time. The result of the questions can be easily visualized. Since the exact value range of the respective results does not have to be known intuitively, we use a z-score to standardize the results with z(xt) = ply map the results to a color range and display them in an appropriate visualization (a really simple prototype can be seen in Figure 4). In contrast to most of the related work in the field of XAI, it is not necessary to explain the algorithms themselves and the capabilities of the algorithms themselves do not have to be restricted.
xt (xE)[2x] . This allowed us to sim
In this position paper, we discussed how we use a conceptbased engineering approach to explain the decisions and behavior of intelligent agents based on reinforcement learning. Decomposing the agent’s training into concepts based on tacit knowledge of domain experts simplifies development and reveals options to inject domain-specific knowledge into the agent. The defined and trained concepts can be used to reason over the agent’s behavior and its decisions as their interrelations and interactions are known. The motivated idea differs from most current approaches to explainable AI systems (XAI) in that it essentially does not operate on a purely technical level, but rather on the conceptual level known to humans. This makes this approach on the one hand more meaningful and on the other hand enables the unrestricted use of complex machine learning methods, such as reinforcement learning, since no special requirements are placed on the algorithms.
Since this is primarily a position paper, the work is still at an early stage. However, we hope to encourage the community to take further steps in this direction and see this approach as a good complement to the more technical solutions currently available for XAI.
For future work, several interesting tasks remain. First
attempts to merge typical software engineering and machine
learning development activities into combined process
models have been published recently (e.g. Hesenius et al. (
0 ep 2 t s 0 3 0 0 1 0 4 0 5 <<Concept>> Complete sale <<Concept>> Full demand fulfillment <<Concept>> Optimal balance Min Order
Max Order
Order trained on
extends (+delivery failures) <<Environment>>
Real <<Environment>>
Optimal through the overall development cycle. However, they tend to define necessary activities on a rather abstract level, we thus plan to develop a more specific approach to create intelligent agents. Furthermore, we aim to develop tools and artifacts that support teams in deriving the necessary knowledge to train algorithms and to identify helpful concepts.