Dialogue has long been used in human society to explain seemingly opaque concepts. In this paper we focus on how to better explain training models for neural networks, to entertain as well as inform. We present a multi-agent argumentation-based dialogue system to generate human understandable dialogue to explain backpropagation. The system incorporates a model of agent personality and introduces social elements between agents to produce characterful discussion. Natural language templates are used to render utterances in English.
Explainability in machine learning (ML) is not guaranteed by using an effective model.
In fact, often the highest performing models (e.g. Deep Learning) are the least
explainable. The issue of explainability in ML is pervasive, and is seen to affect the methods
used to train models as well as how they perform [
In this paper, we seek to improve the explainability of one such training algorithm
for neural networks, backpropagation, using dialogue as a means of explanation.
Dialogue and argumentation are commonly used in human society to inform, persuade and
deliberate over the course of action. To the same effect, dialogue can be used in
computational systems to resolve conflict between cooperating software agents [
We propose a multi-agent system incorporating the elements of a neural network
and backpropagation algorithm. The system includes a logical model of dialogue for
agents to explain their preferred steps in the algorithm and a Natural Language
Generation method for rendering these dialogues in a human understandable format. The
system can also be configured to support other dialogue topics and agent personalities
using a well defined interface. We define our own set of agents to give identity and
character to our dialogue participants. Moreover, we give these agents personality and
organise them in a social structure, with the aim of generating convincingly human
dialogue that entertains as well as informs. Our method presents the initial steps to a novel
approach to explaining an ML algorithm in an entertaining way.
Many current ML techniques, including neural networks, focus on constructing
models in their internal representations [
In developing explanations, Roscher et al. [
Two additional dimensions are important in our work to enhance user engagement
and support understanding of how a training algorithm works. Engagement with ML
explanations has been sought through different means, including gamification of ML
tasks (e.g., [
Finally, we deliver the rendered debates using a systematic approach for
producing natural language text from non-textual data by borrowing techniques from Natural
Language Generation (NLG) [
We observed that there exists a wealth of related work in the areas of machine learning explainability, argumentation for explainability and multi-agent dialogue systems. However, current work at the intersection of these fields, in particular using multi-agent dialogue systems to explain an ML algorithm in an entertaining way, is limited. In this research, we describe an initial system to address this gap. 3
Here we present our general approach to deliver a dialogue based explanation of the backpropagation algorithm in an entertaining and easy to access way. 3.1
A neural network is composed of a number of neurons connected via directed links
[
The goal of training a neural network is to set the weights and biases in order to minimise the error at the output layer. This error is a measure of how close the output is to the desired output, which is calculated using a loss function, L(w; b), where w and b are matrix representations of the weights and biases. For a multi-layer feedforward network, training can be achieved using backpropagation. At each step, the gradient of the loss function is computed with respect to the weights and biases. The negation of this gradient (the direction of steepest descent) is then used to update the weights and biases. This process is repeated until training converges on a minimal possible loss.
In this paper, we train a neural network for a classification task. Output targets are one-hot encoded, resulting in a target of 1 or 0 for each individual output neuron. For the purpose of later promoting discussion around whether these targets are met, we introduce the concept of tolerance, which is the minimal difference required between an output neuron’s activation and its target for the neuron to be deemed on target. 3.2
We construct our system around the idea of an agent society. Let us regard the natural hierarchy of a neural network’s elements: the network itself, its layers and their respective neurons. We map this structure onto the concept of a labouring agent society, with each element represented by a single agent (see Fig. 1). The neuron agents reside at the lowest level, working to change weights and biases, and are concerned mainly by the personal impact of these changes on themselves and their immediate neighbours. Above them, the layer and network agents concentrate on the bigger picture: training outcomes and errors, guiding the lower levels towards improving these results. The single network agent at the top is the most accountable, responsible for setting goals and communicating these to the layer agents. The layer agents form the ‘middle-management’ and must mediate the debate of the many neuron agents, whilst directing them to perform actions to meet the goals set by the network.
In backpropagation, the error in each output neuron is propagated back to connected hidden neurons according to the strength of the respective connections. The hidden neurons are therefore made responsible for some fraction of the output neuron’s error. This notion of responsibility can be incorporated into an agent’s personality. For example, their individual contribution to the overall training error might interact with their sense of moral or social responsibility to the other agents to influence their choices in a dialogue game. Agent roles and their social relationships are encoded within our application of the belief and personality framework of an agent.
With the aim to convey an explanation of the steps of the backpropagation algorithm, our dialogue can be structured as follows: 1. Begin with the network agent stating an overall goal (e.g., a desired classification). 2. Proceed with the network agent inquiring as to whether the goal has been met. 3. Participants then exchange relevant information to discover the answer to the inquiry. 4. If the goal has not been met, participants argue about what action to take until consensus is reached.
This dialogue incorporates two of the argumentation dialogue types proposed by
Walton and Krabbe [
Our approach to represent the dialogue uses the framework from Riley et al [
The following section will present how we instantiated this framework in our work. 4
We implemented our system in Python, on the basis of Nielsen’s code [
Our agent model encapsulates an agent’s epistemic knowledge, normative knowledge and personality dimensions. In our system we have five agent types: Network (NT), Layer (Li), Input Neuron (INi), Hidden Neuron (HNij) and Output Neuron (ONi). The symbols i; j indicate numerical indices that identify individual layer and neuron agents.
The network agent NT encapsulates the neural network, including the backpropagation algorithm. NT is also responsible for initialising and storing the other agent objects in the society, including the layers and neurons, as well as updating the values of their beliefs to reflect the current state of the network. For example, NT will update the weight, bias and activation beliefs of the neuron agents to ensure that they are consistent with the latest values. NT is also capable of updating its own beliefs between training iterations, such as the target classification of a training sample.
The layer agents Lis are responsible for deriving knowledge relevant to individual neuron agents from NT’s beliefs, such as activation targets for individual output neurons using a target class known to the network. The neuron agents Nis are responsible for harbouring most of the normative knowledge in the society. Nis know about the weights and biases in the network and what actions can be performed to modify them. 4.2
We model agent personalities as a vector of binary personality dimensions. These dimensions can be associated with certain modifications to an agent’s knowledge framework, altering what information they may share in an inquiry or what actions they may propose during deliberation. Thus, an agent’s personality is self-contained within the agent’s knowledge framework, allowing the dialogue protocols to remain generic.
Personality dimensions may also be used to alter the natural language templates used for certain facts. This allows different personalities to be superimposed onto the same sequence of moves via changes in natural language. An optional ‘intent’ with an asserted fact or action argument, represents the intention of the agent behind the assertion and indicate the template variant to be used by the natural language generator. 4.3
Agent Configuration. An agent configuration file allows an agent to store the knowledge base in a concise data-interchange format, JSON. Configuration files make the system accessible to less-technical users by providing a way to edit an agent’s knowledge base to craft different dialogues without the need to write any code. The agent configuration files include the required sections: facts, rules and AATS, below: { "facts": { "all": [...], "pa" : [...], "pb" : [...] }, "rules": { "all": [...], "!pa": [...] }, "AATS" : { "all": [...] }}
Knowledge definitions within these sections can be optionally associated with personality traits pi, as shown. These form a key-value pair, with the personality trait string as the key and array of knowledge definitions as the value. A prefix ! specifies that knowledge applies to all agents that do not display a particular personality trait.
n ! Epistemic knowledge. The belief system represents an agent’s epistemic knowledge and is formed by defeasible facts, literals denoted as , and defeasible rules 1 ^ ::: ^ 0 where 1; :::; n are the premises and 0 is the conclusion.
Facts. An agent maintains facts representing ground predicates listed in facts and including any predicates instantiated by the network agent. A fact 1 has a structure subject (X1 ; : : : ; Xn; Value; Intent ) where xi represents the agents which the fact relates to. These identifiers are important in our domain to be used in the fact’s natural language template to refer to agents by name. In the configuration file we represent these facts as $X1 ... $Xn subject(Value)#Intent strings with a different structure for ease of manipulation. The string $self represents the name of the agent the knowledge belongs to. Neuron specific references can also be used in a neuron configuration file, such as $otherNeuronInLayer and $nextLayerNeuron, which act as wildcards that can apply to multiple other neurons. The following fields are optional: Value specifies a numerical value for the fact and Intent associates an intention with a fact. Intent has no meaning in the context of the logical system and is only referred to during NLG to select the template used to render the fact in a sentence.
Facts instantiated by the network agent have named values specific to neuron agents: – X1 activation & X1 bias: neuron X1’s last activation and bias values. – X1 X2 weight: weight value of the link between neuron X1 and a neuron X2 in the proceeding layer. – X1 target: value of 0 or 1, representing output neuron X1’s target activation on the last training sample. – X1 tolerance: value in the range [0; 0:5) representing how close neuron X1’s activation value must be to the target for it to be deemed on target.
An example of fact is $self $ON1 weight indicating that the connection this weight is associated with is between the neuron agent ($self) and output neuron 1 ($ON1). Rules. Rules are represented as 1 ^ ::: ^ n ! 0. In addition, special rules act as enablers for other rules, which is fundamental in generating a coherent dialogue. We define specific derivation enabling rules of the form r1 : 1 ^ ::: ^ n ! such that r2 : ^ 1 ^ ::: ^ n ! but for simplicity we represent this with rule names 1 ^ ::: ^ n ! r2. This facilitates more finely grained control over the order of rule assertions in the inquiry dialogue. To prevent issues with non-terminating defeasible derivations, the definition of a nested rule is constrained to ensure that no fact acting as a rule’s premise can occur in its conclusion as part of another rule.
In the configuration file, rules 1 ^ ::: ^ n ! 0 are represented as strings of the form premises -> conclusion. In premises & and ! are used in place of ^ and :. A clause may consist of one or more numerical facts combined with Pythonic mathematical notation to construct a boolean sub-expression. For example, consider a rule string known to a neuron agent:
$self_targetIs1 & $self_activation < 0.5 -> $self_activationTooLow This specifies that for the claim to be valid, the neuron agent’s target activation must be 1 ($self targetIs1) and its actual activation ($self activation) must be a value less than 0.5. The claim is $self activationTooLow, which is true if the neuron agent’s activation fell below its target.
Normative knowledge. For deliberation dialogue, we need to bestow agents with
normative knowledge about the effects of actions. Following [
{"source": ["$self_activationTooLow"], "action": "increase_$self_bias", "destination": ["!$self_activationTooLow"]} 4.4
In order for agents to participate in the dialogue, they must be able to construct
arguments on the basis of their beliefs and AATS. Following [
Our dialogue is a combination of deliberation and inquiry. The deliberation protocol begins with an inquiry sub-dialogue whose topic is the set of facts that make up the deliberation goal. The aim of this inquiry is to collectively discover relevant information to the goal. After this, the agents move to deliberation where the goal is a target state for the physical system after the deliberated actions are performed.
In a dialogue, an agent x can make the moves: open hx; open; dialogue( ; ; )i; assert hx; assert; i; and close hx; close; dialogue( ; ; )i. The type of dialogue protocol is described by dialogue( ; ; ): if = inquiry, is a set of propositions whose truth values are to be found; if = deliberation, is a state representing the goal. is the set of participants, is either a set of a-arguments or b-arguments depending on which has not yet been asserted. A commitment store, CSx, stores an agent x’s exchanged information. The global commitment store, CSg, is the union of the commitment stores of all agents participating in the dialogue.
In the inquiry each agent asserts all its relevant beliefs that are not already present in CSg. Relevant means that either this is a fact, a premise of a rule in the global commitment store, or a rule of which conclusion is a relevant defeasible fact. For the remainder of the dialogue, each agent checks whether any of its asserted defeasible rule are fully supported by the information in CSg. For a supported rule, an agent will assert the rule conclusion as a b-argument if the conclusion is not already present in CSg. If the agent cannot assert anything new then they will instead move to close the dialogue.
In the deliberation dialogue, CSg is set to the sub-dialogue store after the inquiry is complete. If it is discovered during the inquiry that all the goal propositions are satisfied, then no action needs to be taken and the protocol can terminate early. Otherwise, the protocol proceeds with each agent asserting relevant a-arguments formed from their AATS. Each agent determines the start state q0x CSg using information in CSg, q0x is the largest possible subset of CSg in Qx. If any action transitions exist in its AATS from q0x to the end state, the goal, then the agent will assert an a-argument to propose the action. Algorithms 1-2 demonstrate these protocols.
In our implementation, CSg acts as a blackboard in the system, broadcasting agent assertions. To determine the agent speaking order, we leverage a token-ring algorithm in which a token is exchanged among the participants of the dialogue in some fixed order, starting from the network agent NT. However, this choice has a cost in generating natural language. For example, unlike direct messages, broadcast messages have no specific recipient associated with them. Therefore, when generating natural language from a broadcast message, we may have to infer who the intended recipient is using information in the message if we wish to address a specific agent. 4.5
We implement the rendering of dialogues into natural language using a template-based
NLG system [
Input: A list = (x1; x2; :::; xn) of agents An initiator xI 2 A list = ( 1; 2; :::; n) (the topic)
A global commitment store CSg Output: A list of moves moves fhxI ; open; dialogue(inq; ; )ig cnloospeasrtici0p,atnts0 j j while closes < no participants do xi [t]
RelevantBeliefs(xi; ; CSg) [ SupportedBArguments(xi; ; CSg)
CSg if j j > 0 then moves CSg closes else closes closes + 1 t (t + 1) (mod no participants) return moves
moves [ fhxi; assert; ig CSg [ 0
Input: A list = (x1; x2; :::; xn) of agents
An initiator xI 2
A list = ( 1; 2; :::; n) (the topic) Output: A list of moves CmSovges ; fhxI ; open; dialogue(del; ; )ig cnloospeasrtici0p,atnts0 j j moves moves [ Inquiry( ; xI ; ; CSg) while closes < no participants do xi [t]
SupportedAArguments(xi; ; CSg)
CSg if j j > 0 then moves moves [ fhxi; assert; ig CSg CSg [ closes 0 else closes closes + 1 t (t + 1) (mod no participants) return moves or more slots that can be replaced with interchangeable words or phrases to produce varied output. Additionally, identifiers in these slots are replaced by specific values, such as the numerical value of the associated fact or names from the agent list. Similar to the agent configuration files, these template strings can be defined in a separate JSON file, grouped via fact names. This file also includes the various strings that can be substituted in the place of the template slots. These non-terminal nodes are synonyms or variations of specific words that help producing a diverse, but equivalent, set of sentences.
In the inquiry dialogue, we generate sentences for assertions of defeasible facts, including those asserted as the claim of a b-argument, using templates for each fact. We do not immediately generate sentences for the assertion of defeasible rules, including those asserted as the support of a b-argument. If a rule is later used as a support, then sentences will be generated for the premise assertions, so we can avoid first generating a sentence for the rule to introduce these premises.
In the deliberation dialogue, we generate sentences for asserted a-arguments using templates for each action. Here, it is not necessary to generate sentences to describe either the AATS start state, which is a combination of facts already asserted in the dialogue, or the end state, which is the goal that was stated in the opening sentence.
For example, the natural language template for the weight fact X1 X2 weight and the action ‘increase’ are shown below. The fact includes substitutions for the names of the neuron agents that share the weight and the value of the weight.
weight: "the weight between X 1 and X 2 is VALUE" increase: "I’ll increase FACT" 5
In this section we present an application of our dialogue system. The deliberation goal is for a specific output neuron agent to meet its activation target on a given training sample. The neural network is trained on the Iris flower dataset from the UCI Machine Learning Repository1 using four input features to identify three classes. Epistemic knowledge. In the inquiry portion of the dialogue, agents will discover whether the output neuron agent is on target before deliberating. This target, an activation of either 0 or 1, will first be passed down the social hierarchy.
Listing 1.2: Layer agent belief base 1 ‘‘rules”: f 2 ‘‘all”: [‘‘targetClass1 −> targetIsSetosa”, 3 ‘‘targetClass2 −> targetClassVersicolor”, 4 ‘‘targetClass3 −> targetClassVirginica”] 5 g 1 ‘‘rules”: f 2 ‘‘all”: [‘‘targetIsSetosa −> ON1 targetIs1”, 3 ‘‘targetIsSetosa −> ON2 targetIs0”, 4 ‘‘targetIsSetosa −> ON3 targetIs0”, 5 ‘‘targetIsVersicolor −> ON1 targetIs0”, ...]g
At the highest level, the network agent is given knowledge of the class labels (Listing 1.1). The layer agents are then given rules to translate the target class label into target activations for individual output neuron agents (Listing 1.2).
Listing 1.3 shows how the output neuron agent determines whether their activation is on target, using the rules in lines 5-10.
Listing 1.3: Output neuron agent belief base 1 ‘‘facts”: f 2 ‘‘!forgetful”: [‘‘$self tolerance(0.1)”], 3 ‘‘forgetful”: [‘‘$self isForgetful”, ‘‘$self tolerance(0.4)”]g, 4 ‘‘rules”: f 5 ‘‘all”: [‘‘$self targetIs1 & $self act > (1 − $self tolerance) −> $self onTarget”, 6 ‘‘$self targetIs0 & $self act < $self tolerance −> $self onTarget”, 7 ‘‘$self targetIs1 & $self act < (1 − $self tolerance) −> $self actTooLow”, 8 ‘‘$self targetIs0 & $self act > $self tolerance −> $self actTooHigh”, 9 ‘‘$self actTooHigh −> !$self onTarget”, 10 ‘‘$self actTooLow −> !$self onTarget”], 11 ‘‘!forgetful”: [‘‘$other isForgetful −> 12 $other targetIs1 & $other act < (1 − $self tolerance#correction) −> $other actTooLow#correction”, 13 ‘‘$other isForgetful −> 14 $other targetIs1 & $other act > $self tolerance#correction −> $other actTooLow#correction”]g Forgetfulness demonstrates an example of personality dimension. Observe that the true value of the tolerance is known to all non-forgetful output neuron agents (line 2), whereas a forgetful agent will ‘misremember’ a more lenient version (line 3). In lines 11-14 we introduce the rules for non-forgetful agents which allow them to correct others that may have incorrectly determined themselves to be on target. Note that these rules are nested; the outer premise prevents an agent from asserting their belief of the tolerance unless another agent is forgetful. The correction intent is attached to the true value of the tolerance, which is later used to formulate the natural language as a correction. Normative Knowledge. In the deliberation portion of the dialogue, agents suggest actions to help the output neuron agent meet its target. We let hidden neuron agents propose weight changes for their connections to help a neuron agent in the next layer meet their target. As an example, the increase weight action is shown in Listing 1.4. The source state for this transition is the next neuron’s activation being too low.
Dialogue. We illustrate here two different dialogues that may be generated from the example application (see Table 1). In Dialogue A, ON1 has a stubborn personality. When it is discovered that they are not on target, a hidden neuron in the preceding layer suggests an action. ON1 refuses, insisting on its own course of action. In Dialogue B, ON1 has a forgetful personality. When they incorrectly find themselves to be on target, another ON corrects them. During deliberation, a hidden neuron then suggests an action. 6
A multi-agent dialogue system was demonstrated as a means of enriching backpropagation with dialogue. The generic dialogue framework ensures the ease of extending the system to support further examples. Our flexible system can produce entertaining and informative dialogue to help explain the mechanics of backpropagation.
Possible future work includes extending the dialogue topics to provide more variety
in the system’s output and improve explainability of related aspects of backpropagation,
such as error propagation or overfitting. Extended topics could focus on using
information aggregated over the course of many training iterations, preventing dialogue from
becoming repetitive over the course of many epochs. Evaluation with users would help
establish the advantages that this approach provides in understanding backpropagation
and assess the usefulness of specific explanations as perceived by users. Depending on
the familiarity of these users with ML concepts, such as neural networks, these studies
could also help identify where dialogues may require additional preamble to explain
the technical concepts discussed by agents. Currently, the model of personality is
limited; no partial dimensions or model of emotional state was implemented. The model of
agent personality could be adapted to adhere to a more flexible model, such as the PEN
[