In certain problem-solving tasks that require Human-AI interactions, a mutual understanding of the reasoning behind the performed actions can benefit both humans and artificial agents. However, identifying and predicting the cognitive strategies involved in such a hybrid setting, especially in novel, self-regulated exploratory tasks, is a challenging endeavour. Our aim is to identify behavioural properties relevant to young children's cognitive strategies that are present in problem-solving, with an emphasis on the Aha! moment as an intermediate step between exploratory actions, that typically relate to the development of tacit knowledge, and the generation of explanations that requires explicit knowledge. We use data from existing, previously published, behavioural studies with children 5 to 7 years old to explore these mechanisms in two selfregulated problem-solving tasks. In addition, we reflect on our observations of an Artificial Agent (Q-learning algorithm) that learns to solve the same task. Our findings indicate that while in current reinforcement learning practice, detecting the moment of the cognitive transformation of the problem representation normally translates into observing convergence curves of the objective functions being optimized, in young children this involves more complex behavioural properties, such as verbal metacognition. These behavioural processes can be used as a proxy for the identification of the Aha! moment. Finally, we propose a conceptual map which integrates the observed behaviours that are used to detect, communicate and corroborate learning both in humans and machines and we discuss the association of children's exploratory behaviours, the Aha! moments and ultimately their explanation generation.
For efective hybrid environments where humans
collaborate with Artificial Intelligence (AI) systems to make a
decision, a mutual understanding of the reasoning behind
certain actions or recommendations can be of catalytic
importance.
tual understanding and trust development [1], and can be
ing models can be explained towards a customized and
diverse set of audiences [
Based on a series of behavioural studies with children imagination, alone or with others and can take place
5 to 7 years of age, we identify behavioural properties rel- in various settings [10]. The connection of play with
evant to cognitive processes that are present in problem- the development of fundamental cognitive processes and
solving tasks, with an emphasis on identifying the Aha! human learning has been well-established [
In order to solve a problem, two mental representations the solution to a problem that up to this point had left
are needed: one of the current state and one of the goal her puzzled. In cognitive science it is referred to as
instate. As it is goal-oriented and contextualized, a plan sight problem solving and it is accompanied by a feeling
detailing the solution step by step is required. A constant of satisfaction for the solver. It has been related to
cremonitoring process is also required as each move has ative thinking [23, 24] and includes an exploratory phase
consequences that can bring the solver closer to or further where divergent thinking takes place, especially during
to the desired goal state. It also requires mental flexibility the early stages of the problem solving process. This
aland thus, inhibitory control [15]. If a first chosen solution lows the person to produce new ideas or connect existing
seems to be inappropriate, the solver has to adapt his ideas. The second phase is the convergent thinking phase
strategy. where a solution should be elected by synthesizing,
analyzing and monitoring the matching degree of the current
2.2. Social learning result to the expected one. Although the experience of
insight is sudden and can seem disconnected from the
imSocial learning is a crucial component of human intel- mediately preceding thought, recent research shows that
ligence, allowing us to rapidly adapt to new scenarios, insight is the culmination of a series of brain states and
learn new tasks, and communicate knowledge that can processes operating at diferent time scales. Elucidation
be built on by others [16]. The work of Lev Vygotsky of these precursors suggests interventional opportunities
who put forward this view already in the 1920s takes for the facilitation of insight [3], including concurrent
into account factors such as the language development verbalization [25]. As for every problem solving, most
and cultural influences in the cognitive development of of these strategies rely on a constant restructuring of
children [
They are considered as a cornerstone for knowledge co- Regarding explanation generation, there is a large body construction and can help independent problem-solving of works in various fields. But it always implies an exactivity. This leads to the diference of what a learner plainer and an explainee with their own respective charcan do with or without help as he described under the acteristics. Of particular interest across the fields is the concept of the Zone of Proximal Development (ZPD). The role of the Theory of Mind ie. the ability of a person social interaction with the use of linguistic and cultural to attribute mental states to the consequent behaviours tools facilitate the internalization of knowledge and its of herself or others [26]. The selection and evaluation transformation into cognitive tools supporting the de- processes of explanations depend on the explainer and velopment of new cognitive functions. The latter aspect explainee, but also on the characteristics of the context. has been considered for the design of artificial agents The nature of an interaction for explaining is diferent that are able to interact with others and internalize these in kindergarten between the teacher and a young child interactions in a similar way as humans [18, 19]. than between the cockpit desk and the pilots during a lfight. The role of beliefs has also been raised recently as a cornerstone. An explanation does not necessarily needs to be consistent with a person’s beliefs but should help promoting a revision component [27] thus allowing the evolution of the internal representations. Human explanations from social sciences became an integrated part of Artificial Intelligence (AI) through the XAI field in order to provide explanatory agents and to facilitate interactions between humans and machines.
Code Spontaneous musicking
Sound exploration
Assessment
Reasoning Deliberate musicking
Planning 11.05 Computer-supported music composition was selected as 15.87 an open-ended task which does not include a predefined 27.38 objective final “solution”; rather, it involves decision1183..606 making based on subjective criteria and self-regulated 14.04 goal identification and provides the context for the emergence of a variety of processes and interactions. We Table 1 identify two major findings relevant to the scope of this The taxonomy of behaviours that emerged during the open- paper; first, despite the unstructured and the highly exended self-regulated task of children’s collaborative music- ploratory nature of this task, we observed that children making and the percentage of occurrence per behaviour. exhibited behaviours that correspond to “making” and to “reflecting”. Spontaneous and exploratory actions were mixed with deliberate actions and planning while the latter were supported by assessment and reasoning. Second, the collaborative setting of this study facilitated children’s verbal interactions and negotiations during their decision-making process and consequently their reasoning and reflection on their actions. These process correspond to the mobilization of theirverbal metacognition part of which was the generation of explanations during the negotiation of their task-related decisions.
This means that given the opportunity (in this case collaborative setting), children as young as 5 years old acFigure 3: Average percentage of children’s behaviours in tively engage in self-initiated reflection on their actions Study 1, Making (C1, C2, C5 and C6) and Reflecting (C3 and and imagine the future outcomes while being able to C4). explain their reasoning to the collaborator. However, we observed that they often lacked the verbal abilities observations is high relative to the rate of change; and (c) and the terminology for accurate explanations. For this the observations are subjected to an intensive, trial-trial reason, they mobilised other available modalities, such analysis to infer the processes that give rise to change. as gestures, and used the afordances of the graphical The annotation of the data was based on children’s verbal user interface of the tool provided to complement their and non-verbal behaviours and the corpus included 7063 explanatory behaviours. annotated behaviours. The taxonomy of the behaviours that related to children’s cognitive processes and the 4.2. Study 2: An indication for the Aha! percentage of their occurrence appear in Fig. 1.
The results indicate that despite the fact that the par- moment ticipant children were of a relatively young age - which is The goal of this experiment was to test the impact of typically related to exploratory actions - the behaviours the type of a robot intervention on children’s problemof deliberate musicking (C5) and planning (C6) appeared solving process. We used the cognitive task of the Tower slightly more than the exploratory behaviours of sponta- of Hanoi (ToH) [31] which is used to measure children’s neous musicking (C1) and sound exploration (C2). planning abilities and inhibitory control. To reach the
Furthermore, a grouping of the behaviours that corre- optimal solution, it requires participants to involve inspond to reflective actions (C3 and C4) and the ones that hibition of impulsive moves that superficially bring the correspond to active music-making (C1, C2, C5 and C6) child closer to the goal, but are unhelpful for the longerreveals that the “reflecting” behaviours occurred 46.18% term solution [32]. We designed a experiment with three of the total cognitive behaviours, while the active music- phases: a baseline (single child), an intervention (manipumaking behaviours occurred 53.82% (see Fig. 3). These lation of the robot’s behaviour) and an evaluation (child’s results indicate that despite the young age of the partici- voluntary interaction with the robot) for = 20 chilpants, reflecting and reasoning about the musical choices dren 5 to 7 years old. For the intervention phase, we had appear as an integral part in children’s cognitive engage- two conditions; in Condition1, the robot and the child ment with music-making. solved the task in a turn-taking setting and in Condition2 we designed a child-initiated voluntary interaction with the robot. In this paper, we focus on a single child’s problem-solving process to explore behavioural properties relevant for our understanding of the transition from exploratory actions to the transformation of the problem representation, which requires the involvement of inhibitory control and the stabilization of the optimal performance. The details of the study appear in [33].
We evaluated the task performance in relation to the trajectory of optimal and suboptimal movements over the course of the task. The optimal movements are defined as the ones that lead to the solution of the task with the minimum number of movements. In addition, we measured the relevant speed of the movements in relation to the baseline of each participant. Given the assumption that during the task the children sustained the necessary attention, we identify point A in Fig. 4 as the point that separates the phase of mostly suboptimal moves (red peaks) with the phase of mostly optimal movements (blue peaks), which are also carried out faster. instability, the incremental optimization and the performance stabilization. During the exploratory phase, the children were reinforced by the results of their actions which eventually guided them to the restructuring of the problem representation and consequently the use of the strategy which is based on inhibitory control. After the Aha! moment, we observe a stabilisation of the optimal moves which indicates learning. One of the limitations of this study was the fact that it was not designed in a way to facilitate the child’s verbalisation of their thoughts, reasoning and reflections. For this reason, we were not able to make any inferences regarding the children’s reasoning, their verbal metacognition and the generation of possible explanations during the problem-solving process. 4.3. Study 3: Social Interaction and
Explanations
robot on children’s collective problem-solving and the 4.2.2. Reflection child-child social dynamics in a setting of two children We observed exploratory behaviours that typically were and one robot (see Fig. 5). We built upon study 2 and characterised by increased number of suboptimal moves. we used the same task, the ToH task and the same robot. We identify as an Aha! moment, the point when a trans- We designed a controlled 2X2 experimental study with formation of the mental representation of the problem = 86 children who all participated in a baseline session occurs which, in this task, is behaviourally manifested (without robot), an intervention (with the manipulation by the mobilization of inhibition as a strategy for the of robot behaviour, in terms of its cognitive reliability optimal solution of the task, meaning that the child in- and expressivity) and an evaluation session (with childhibits the impulsive move and performs the less obvious initiated form of interaction) to solve the Tower of Hanoi one that will lead to the stabilization of optimal solution task with an incremental dificulty level in the diferent of the task (see point A in Fig. 4). This is a cognitive experimental configurations without any expert’s interstrategy that in the age-group of the present studies does vention. For the purposes of this paper, we focus on the not appear intuitively. As shown in Figure 1, behavioural ifndings on the patterns of children’s social interactions properties that appear in the problem-solving process in and verbal negotiations and explanations during the colthe context of the given tasks include the performance lective task performance. The detailed research design, analysis and findings of the study appear in [ 22]. and planning appeared to be an integral part of the pro- as part of explanation generation. This was more evident
We observed that the setting of the study facilitated childchild social interaction and verbal reflection, reasoning cess which was lacking from study 2. To measure the team disparity, we define social interaction, , as the number of task-related interactions between children. = 1 + 2 where with = 1, 2 refers to the number of times child addresses their peer with a task-related verbal or non-verbal (i.e. pointing and gestures) behaviours and L refers to the number of movements needed by the team to solve the task. Our analysis showed that children had a higher rate during the sessions with the robot, namely the Intervention ( = 0.16, = 0.14 ) and the Evaluation ( = 0.13, = 0.092 ) which difered significantly from the Baseline session ( = 0.06, = 0.09 with = 0.08 and = 0.015 . Among the verbal manifestations we identified the utterances related to planning as one of the strategies children used to negotiate for the next movement on the ToH task. We identified the balance between children in the planning of the move) ments, and defined a planning disparity metric, as the absolute diference in the number of interactions initivalues: the LA2 tries to solve the game alone while being able to ask for help whenever its best action is not good enough (plot not on logarithmic scale as the agent asks for help at most 7 times) catalytic for the facilitation of their task-related planning in the sessions with the robot. One possible explanation for this is the fact that one of the conditions involved a robot that suggested suboptimal movements. In that case, the children engaged in child-child negotiations and explanation generation to collectively take a decision for the next move. Our observations indicate that two cognitive strategies were involved in children’s explanations, planning as a part of an a priori explanation of their reasoning for a certain decision and reflecting as a part of an a posteriori explanation. We need yet to analyse the association of the strategy of planning in the context of explanatory behaviours and its relation to a preceding Aha! moment. It should be noted that additional non-verbal manifestations, such as pointing and gestures, were mobilised in the cases that a child did not have the verbal maturity to formulate the planning or the explanation. 4.4. Study 4: Multi-agent setting This study in [34] consists of the same non-open-ended task (ToH) and collaborative setting: one learning agent ning performing better ( = 19, = 0.51, = 0.40 compared to groups with an unbalanced planning behaviour ( = 18, = 1, 61, = 0.98 ). In this case, planning was used as part of the explanation formation which was observed to be one of the strategies for children’s negotiations in problem-solving.
Children’s social verbal and non-verbal interaction during the problem-solving process in Study 3 appeared there was a significant diference in task performance ated by each child of the team: Our analysis showed that (LA) and one helping agent with focus on the voluntary interaction among artificial agents. In order to explore ( = 297, < 0.001 ) between teams with a balanced plan- if algorithms benefit from asking for help in collabora) tive problem-solving, as children do, two hypotheses are tested: learning.
ifnding the optimal solution compared to not on demand.
ing agent in a “turn-taking” scenario, which results in a canonical cognitive intervention from the expert. 2) LA2 solves the task independently, having the option to ask for help of the expert whenever (if) this is needed resulting in an on demand intervention. Two parameters are assessed: 1) Canonical intervention (help) rate (every 2, 3 or 4 turns), and 2) Ask-for-help threshold (from 0 to 1). The last parameter was created to simulate what happens when a child asks for help: if the best policy value is lower than the ask for help parameter, the expert will play instead of the LA. 4.4.1. Data analysis 2. The lack of a natural language interactive communication interface impedes questioning the system about its confidence in real time, or uncertainty estimates. 3. The dependence of a happy end solving the task constraints explanations during the learning phase. How can an agent communicate to its developer or user its struggles in solving the task when it has not yet achieved a satisfactory performance? Could common AI practices (data augmentation, fine-tuning), be accessible to the agent for it to communicate and be able to choose to change them? 4. Most deep RL models rely on baselines of other agents to assess their worth, lacking comparisons in multi-agent settings [36] with children learning. 5. The inability of current RL algorithms to communicate the continual progress beyond reporting a sole reward value obtained at the end of a convergence curve makes it challenging for LAs to explain their skills to solve the credit assignment problem, their dificulties or agility to complete sub-tasks, their acting self-confidence, or learned savvy behaviours. 6. The lack of alignment of explanations of LAs with meta-learning and trustworthy AI dimensions (such as the trust calibration meta-information taxonomy [1]) should be accounted for in the explanation generation process, in the same way as mechanisms to ensure the reproducibility of insights-built explanations.
From scenario 1 it is observed that the LA is more eficient when it is helped by the expert in a turn taking scenario, and that it is even more efective when helped every 4 turns rather than every 2. The importance of exploring on its own is showcased by the agent, rather than always having the optimal solution.
Even when all approaches converge in both scenarios, in scenario 2, the agent that asks for help becomes also faster and more efective: help is most useful at the beginning of learning. After asking for help many times during the first episodes it starts solving the task by itself, resulting in an increase of ineficient moves. The agent seems to gain confidence in movements which, while imperfect, allow the task resolution by exploring diferent states.
Compared to the LA not being helped, the asking-forhelp agent is a lot more eficient, but there is not much variation among the canonical and the help-asking con- Reflection ifgurations. This is probably due to the rather simple This is the only study not involving kids, but followsimulation of the trigger for the request of help. Simu- ing the same protocol as in the ToH studies. While RL lating the child’s behaviour is a complex task and more model developers communicate a model learning converemphasis needs to be placed on accurately describing it, gence due to reaching plateaus in learning curves, these including how to simulate the ”asking for help” function. changes should as well reflect key changes in kid beAdding mechanisms such as intrinsic motivation, about haviour. However, we showed this is not always evident the LA’s desire to solve the game on its own, could make to map. Providing the learning agent with signals such as the comparison more accurate. The agent asks for help the Aha! moment to adjust its self learning /hyperparamwhen it considers that a movement is not good enough eter changes could be paramount to avoid blind manual to be played, whereas the actual mechanisms that drive engineering (on e.g., reward function crafting) processes the child to ask for help are more complex. where no common procedures exist. We believe these
We identify emergent issues that should be part of the are the explainable dimensions that XAI for RL should explanation for the design of a LA, and that are not part of work on (identifying Aha! moments, categorising probthe modelled problem nor a representation is accessible lems, dificulty, environments, collaboration/competition for the agent and thus, for its explanation: dimensions, etc).
1. The lack of a multimodal input space [35] for the The dificulties to explain the learning process of a agent to perform in the action-interaction loop single LA could be reduced by involving interaction with of RL can restrain the agent from exhibiting the other agents. One could attend to social interaction [16] correct behaviour and communicating as humans and social influence as intrinsic motivation [ 37] learning would expect. metrics. Both showed to enhance learning in multiagent settings.
tives that agents depend on to progress. Once an agent learned, it is not enough that the agent performs tasks in less time, and better, but also that it uses other human factors or social outcome metrics such as in [38, 39].
We presented a set of behavioural studies and an experiment with a Q-learning algorithm in order to identify behavioural properties that relate to the Aha! moment, i.e., the moment of the restructuring of the problem representation (Fig. 1). These behavioural properties appear as part of the transition from exploratory to explanatory behaviours (tacit to explicit knowledge). They include task-related observations such as performance instability, incremental optimization and stabilization as well as verbal metacognitive manifestations (observed only in two of our studies with children) that involve reasoning, reflection and planning. These behavioural properties seem to facilitate the Aha! moment and eventually the generation of explanations by children.
In current RL practice, detecting this moment normally translates into observing convergence curves of the objective functions being optimized (normally reaching a plateau in cumulative reward or optimized loss, usually both). This is an external signal not usually leveraged by the agent. Although there are exceptions such as the use of artificial curiosity signals for self learning of the agent [40]), in regular AI model development practice, we must highlight the need for easier mechanisms to convey actions that demonstrate the dificulties of the agent until convergence plateaus and/or a suficient level of an XAI metric are reached.
We summarise the main points we propose to consider in approaches evaluating XAI, as follows: 1. The Aha! moment (or problem representation restructuring) acts as the intermediate step between non-explainable and explainable behaviours. In a deeper view, since the explanation acts as an interface between the model and a given target audience, the Aha! moment is a trigger signal for a model to start elaborating explanations. More efort should be put into specifying the meaning of the Aha! moment in various tasks that RL models are currently tackling. Defining and detecting high level policies characterising an Aha! moment (e.g. in terms of key/exploratory action sequences) can be signs we should be able to not only programmatically detect, but also communicate. In this way, we can achieve explainable and reliable models, since Aha! signs must act as an additional proxy to attain trustworthy systems. 2. Children understand but sometimes lack the cognitive and metacognitive skills to explain. Explanations are subject to both biological and artificial systems’ understanding of properties of a given task, and in young children explanations are subject to their verbal abilities. Children often use gestures such as pointing, which means that explanations that can support human-AI interaction are subject to tools responsible for social interaction. Aiming towards human-level AI requires a broader set of key social skills for complex embodied communication in multimodal settings within constantly evolving social worlds [41]. 3. The hybrid use of strategies of “planning” and “insight”: Questioning the false dilemma of logical reasoning vs machine learning, we argue for a synergy between these two paradigms in order to obtain hybrid AI systems. 4. Both social interaction [16] and social influence as intrinsic motivation [37] show to be enhancers for learning in multi-agent settings.
of our research on children’s problem solving in diferent settings and combine them with our research on XAI. However, due to space limitations, we are limited to provide overviews without in-depth analysis. We aim to tackle the latter in our future work.
We hope this work is useful beyond developmental robotics and AI, i.e., facilitating an efective and ethical deployment of RL systems, e.g. from energy building management to AI for health, where evaluating single reward functions simply does not reflect nor assess the complexity of the system nor the dificulties it has to deal with.
Future work should aim to involve more tangible evaluation metrics that both 1) optimize technical robustness more broadly, and 2) reflect a human-centered view where machine learning factors are questioned, monitored and explained in parallel ways to how children learn. Evaluation mappings across human and machine learning will allow us to better assess the trade-ofs between AI assisted decision making and policies.
JRC; Díaz-Rodríguez by IJC2019-039152-I funded by MCIN/AEI /10.13039/501100011033 by “ESF Investing in your future” and Google Research Scholar Program; and Merino by Programa Operativo FEDER Andalucia 20142020 through the project DeepBot (PY20_00817).