Current AI systems lack several important human capabilities, such as adaptability, generalizability, selfcontrol, consistency, common sense, and causal reasoning. We believe that existing cognitive theories of human decision making, such as the thinking fast and slow theory, can provide insights on how to advance AI systems towards some of these capabilities. In this paper, we propose a general architecture that is based on fast/slow solvers and a metacognitive component. We then present experimental results on the behavior of an instance of this architecture, for AI systems that make decisions about navigating in a constrained environment. We show how combining the fast and slow decision modalities, which can be implemented by learning and reasoning components respectively, allows the system to evolve over time and gradually pass from slow to fast thinking with enough experience, and that this greatly helps in decision quality, resource consumption, and efficiency.
AI systems have seen great advancement in recent years, on many applications that pervade our
everyday life. However, we are still mostly seeing instances of narrow AI that are typically focused
on a very limited set of competencies and goals, e.g., image interpretation, natural language
processing, classification, prediction, and many others. Moreover, while these successes can be
accredited to improved algorithms and techniques, they are also tightly linked to the availability
of huge datasets and computational power [
We believe that a better study of the mechanisms that allow humans to have these capabilities
can help [
In this paper we describe the SOFAI architecture, characterizing the System 1 and System
2 solvers and the role of the meta-cognitive agent, and provide motivations for the adopted
design choices. We then focus on a specific instance of the SOFAI architecture, that provides the
multi-agent platform for generating trajectories in a grid environment with penalties over states,
actions, and state features. In this instance, decisions are at the level of each move from one grid
cell to another. We show that the combination of fast and slow decision modalities, which can be
implemented as learning and reasoning components, allows the system to create trajectories that
are similar to human-like ones, compared to using only one of the modalities. Human-likeness
is here exemplified by the trajectories built by a Multi-alternative Decision Field Theory model
(MDFT) [
In this section we give an overview of the SOFAI architecture, additional details are available in the Appendix. SOFAI is a multi-agent architecture (see Figure 1) where incoming problems are initially handled by those System 1 (S1) solvers that possess the required skills to tackle them, analogous to what is done by humans who first react to an external stimulus via their System 1.
/Solver
• Updates the models • (Re)trains the
S1 solvers • Autonomous activation
• Based on past experiences • Acts in O(1) • Activates autonomously
Knowledge about environment impacted by agent’s decisions
Past decisions and their reward Proposed solution and confidence Adoption of system 1 solution
Knowledge and beliefs about other agents impacting the same environment
• Chooses between S1 solution and
S2 activation • Assesses value of success, resources, trustworthiness of solvers • Adopts a two-phase assessment OR
Activation
• Employs reasoning • Consumes more resources • Activated by meta
cognitive module Solution/decision 2.1. Fast and Slow Solvers As mentioned, incoming problems trigger System 1 (S1) solvers. We assume such solvers act in constant time, i.e., their running time is not a function of the size of the input problem instance, by relying on the past experience of the system, which is maintained in the model of self. The model of the world contains the knowledge accumulated by the system over the external environment and the expected tasks, while the model of others contains the knowledge and beliefs about other agents who may act in the same environment. The model updater agent acts in the background to keep all models updated as new knowledge of the world, of other agents, or new decisions are generated and evaluated.
Once an S1 solver has solved the problem (for the sake of simplicity, assume a single S1 solver), the proposed solution and the associated confidence level are available to the meta-cognitive (MC) module. At this point the MC agent starts its operations, with the task of choosing between adopting the S1 solver’s solution or activating a System 2 (S2) solver. S2 agents use some form of reasoning over the current problem and usually consume more resources (especially time) than S1 agents. Also, they never work on a problem unless they are explicitly invoked by the MC module.
To make its decision, the MC agent assesses the current resource availability, the expected
resource consumption of the S2 solver, the expected reward for a correct solution for each
of the available solvers, as well as the solution and confidence evaluations coming from the
S1 solver. In order to not waste resources at the meta-cognitive level, the MC agent includes
two successive assessment phases, the first one faster and more approximate, related to rapid
unconscious assessment in humans [
This architecture and flow of tasks allows for minimizing time to action when there is no need for S2 processing since S1 solvers act in constant time. It also allows the MC agent to exploit the proposed action and confidence of S1 when deciding whether to activate S2, which leads to more informed and hopefully better decisions by the MC.
Notice that we do not assume that S2 solvers are always better than S1 solvers, analogously
to what happens in human reasoning [
The MC module exploits information coming from two main sources: 1) the system’s internal models of self, world, and others; 2) the S1 solver(s), providing a proposed decision for a task, and their confidence in the proposed decision.
The first meta-cognitive phase (MC1) activates automatically as a new task arrives and a solution for the problem is provided by an S1 solver. MC1 decides between accepting the solution proposed by the S1 solver or activating the second meta-cognitive phase (MC2). MC2 then makes sure that there are enough resources for running S2. If not, MC2 adopts the S1 solver’s proposed solution. MC1 also compares the confidence provided by the S1 solver with the risk attitude of the system: if the confidence is high enough, MC1 adopts the S1 solver’s solution. Otherwise, it activates the next assessment phase (MC2) to make a more careful decision. The rationale for this phase of the decision process is that we envision that often the system will adopt the solution proposed by the S1 solver, because it is good enough given the expected reward for solving the task, or because there are not enough resources to invoke more complex reasoning.
Contrarily to MC1, MC2 decides between accepting the solution proposed by the S1 solver or activating an S2 solver for the task. To do this, MC2 evaluates the expected reward of using the S2 solver in the current state to solve the given task, using information contained in the model of self about past actions taken by this or other solvers to solve the same task, and the expected cost of running this solver. MC2 then compares the expected reward for the S2 solver with the expected reward of the action proposed by the S1 solver: if the expected additional reward of running the S2 solver, as compared to using the S1 solution, is large enough, then MC2 activates the S2 solver. Otherwise, it adopts the S1 solution.
To evaluate the expected reward of the action proposed by S1, MC2 retrieves from the model of
self the expected immediate and future reward for the action in the current state (approximating
the forward analysis to avoid a too costly computation), and combines this information with the
confidence the S1 solver has in the action. The rationale for the behavior of MC2 is based on the
design decision to avoid costly reasoning processes unless the additional cost is compensated by
an even greater additional expected reward for the solution that the S2 solver will identify for this
task. This is analogous to what happens in humans [
In the SOFAI instance that we consider and evaluate in this paper, the decision environment is a 9 × 9 grid and the task is to generate a trajectory from an initial state 0 to a goal state , by making moves from one state to an adjacent one in a sequence, while minimizing penalties.
Such penalties are generated by constraints over moves (there are 8 moves for each state), specific states (grid cells), and state features (in our setting, these are colors associated to states). For example, there could be a penalty for moving left, for going to the cell (1,3), and for moving to a blue state. In our specific experimental setting, any move brings a penalty of − 4, each constraint violation gives a penalty of − 50, and reaching the goal state gives a reward of 10.
This decision environment is non-deterministic: there is a 10% chance of failure, meaning that the decision of Figure 2: Example of the constrained grid moving to a certain adjacent state may result in a move decision scenario. Black squares to another adjacent state chosen at random. Figure 2 represents states with penalshows an example of our grid decision environment. ties. Penalties are generated also
Given this decision environment, we instantiate the when the agent moves left or SOFAI architecture as follows: (1) one S1 solver, that tbootatobmlu-reigohrt,aogrrweehnenstiattem. oTvhees uses information about the past trajectories to decide the red lines describe a set of tranext move (see below for details); (2) one S2 solver, that jectories generated by the agent uses MDFT to make the decision about the next move; (all with the same start and end (3) MC agent: its behavior is described by Algorithm 1; point). The strength of the red (4) model of the world: the grid environment; (5) model color for each move corresponds of self: it includes past trajectories and their features to the amount of trajectories em(moves, reward, length, time); (6) no model of others. ploying such move.
In Algorithm 1: • (, {, }) returns the number of times in state an action computed by solver ( means any solver) has been adopted by the system; if they are below 1 (a natural number), it means that we don’t have enough experience yet. • ( ) and () are respectively the partial reward of the trajectory and the average partial reward that the agent gets when it usually reaches state : if we are below 2 (between 0 and 1), it means that we are performing worse than past experience. • is the confidence of the S1 solver: if it is below 3 (between 0 and 1) it means that our attitude to risk does not tolerate the confidence level.
If any of the tests in line 1 are passed (meaning, the condition is not satisfied), the MC system (MC1) adopts the S1 decision. Otherwise, it performs a more careful evaluation (MC2): • 6 checks that the S2 solver has enough experience. If not, a random choice between S1 and S2 is made (line 3). • Otherwise, it checks if it is convenient to activate S2 (line 6), comparing the expected gain in reward normalize by its cost. 4 gives the tolerance for this gain. If it is convenient, MC activates the S2 solver (line 8), otherwise it adopts S1’s decision. In this evaluation, 2 and are respectively the average amount of time taken by S2 to compute an action and the remaining time to complete the trajectory; 2() and (, ) are the expected reward using S2 in state and the expected reward of adopting action (computed by S1) in state . The expected reward for an action in a state is: (|, ) = ∑︀∈, (|, ) * , where , is the set of all the rewards in state taking the action that are stored in the model of self; (|, ) is the probability of getting the reward in state taking the action . As the expected reward depends on the past experience stored in the model of self, it is possible to compute a confidence as follows: (, ) = ( ( (+1−0−.51)0)) ), where is the standard deviation of the rewards in taking an action , is the probability of taking action in state .
The S1 agent, given a state , chooses the action that maximizes the expected reward based on the past experience. That is: argmax((|, ) * (, )). MC1 and MC2 bear some resemblance to UCB and model-based learning in RL [28]. However, in SOFAI we decompose some of these techniques to make decisions in a more fine grained manner.
The S2 agent, instead, employs the MDFT machinery (see Section A.1.2) to make a decision, where the matrix has two columns, containing the Q values of a nominal and constrained RL agents, and the attention weights are set in three possible ways: 1) attention to satisfying the constraints only if we have already violated many of them (denoted by 01), 2) attention to reaching the goal state only if the current partial trajectory is too long (denoted by 10), and 3) attention to both goal and constraints (denoted by 02). We will call the three resulting versions
We generated at random 10 grids, and for each grid we randomly chose: initial and final states, 2 constrained actions, 6 constrained states, 12 constrained state features (6 green and 6 blue). For each grid, we run: (1) two reinforcement learning agents: one that tries to avoid the constraint penalties to reach the goal (called RL Constrained), and the other that just tries to reach the goal with no attention to the constraints (called RL Nominal). These agents will provide the baselines; (2) the S1 solver; (3) the S2 solver (that is, MDFT): this agent will be both a component of SOFAI and the provider of human-like trajectories; (4) SOFAI 01, SOFAI 10, and SOFAI 02. Each agent generates 1000 trajectories. We experimented with many combinations of values for the parameters. Here, we report the results for the following configuration: 1 = 200, 2 = 0.8, 3 = 0.4, 4 = 0, 6 = 1.
We checked which agent generates trajectories that are more similar to the human ones (exemplified by MDFT). Figure 3 reports the average JS-divergence between the set of trajectories generated by MDFT and the other systems. SOFAI agents perform much better than S1, especially in the 01 configuration.
We then compared the three versions of SOFAI to S1 alone, S2 alone, and the two RL agents, in terms of the length of the generated paths, total reward, and time to generate the Figure 3: Average JS divergence between the set of trajectories, see Figure 4 and 5. It is easy to trajectories generated by MDFT and the see that S1 performs very badly on all three other systems. criteria, while the other systems are comparable. Notice that RL Nominal represents a lower bound for the length criteria and an upper bound for the reward, since it gets to the goal with no attention to satisfying the constraints. For both reward and time, SOFAI (which combines S1 and S2) performs better than using only S1 or only S2.
We then passed from the aggregate results over all 1000 trajectories to checking the behavior of SOFAI and the other agents over time, from trajectory 1 to 1000. The goal is to see how SOFAI methods evolve in their behavior and their decisions on how to combine its S1 and S2 agents. Given that SOFAI 01 performs comparably or better than the other two versions, in the following we only show the behavior of this version and will denote it simply as SOFAI.
Figure 6 and 7 shows the length, reward, and time for each of the 1000 trajectories, comparing SOFAI to S1 and to S2. In terms of length and reward, S1 does not perform well at all, while SOFAI and S2 are comparable. However, the time chart shows that SOFAI is much faster than S2 and over time it also becomes faster than S1, even if it uses a combination of S1 and S2. This is due to the fact that S1 alone cannot exploit the experience gathered by S2 within SOFAI, so it generates much worse and longer trajectories, which require much more time. Perhaps the most interesting is Figure 8. The left figure shows the average time spent by S1 and S2 within SOFAI in taking a single decision (thus a single move in the trajectory): S2 always takes more time than S1, and this is stable over time. The right figure shows the average reward for a single move: S2 is rather stable in generating high quality moves, while S1 at first performs very badly (since there is not enough experience yet) and later generates better moves (but still worse than S2). The question is now: how come S1 improves so much over time? The answer is given by Figure 9 which shows the percentage of usage of S1 and S2 in each trajectory. As we can see, at the beginning SOFAI uses mostly S2, since the lack of experience makes S1 not trustable (that is, the MC algorithm does not lead to the adoption of the S1 decision). After a while, with enough trajectories built by (mostly) S2 and stored in the model of self, SOFAI (more precisely, the MC agent) can trust S1 enough to use it more often when deciding the next move, so much that after about 450 trajectories S1 is used more often than S2. This allows SOFAI to be faster while not degrading the reward of the generated trajectories. This behavior is similar to what happens in humans (as described in Section A.1.1): we first tackle a non-familiar problem with our System 2, until we have enough experience that it becomes familiar and we pass to using System 1.
We presented SOFAI, a conceptual architecture inspired by the thinking fast and slow theory of human decision making, and we described its behavior over a grid environment, showing that it is able to combine S1 and S2 decision modalities to generate high quality decisions faster than using just S1 or S2. We plan to generalize our work to allow for several S1 and/or S2 solvers and several problems for the same architecture, thus tackling issues of ontology and similarity. developmental inquiry., American psychologist 34 (1979) 906. [22] T. O. Nelson, Metamemory: A theoretical framework and new findings, in: Psychology of learning and motivation, volume 26, Elsevier, 1990, pp. 125–173. [23] M. T. Cox, A. Raja, Metareasoning: Thinking about thinking, MIT Press, 2011. [24] M. T. Cox, Metacognition in computation: A selected research review, Artificial intelligence 169 (2005) 104–141. [25] J. D. Kralik, et al., Metacognition for a common model of cognition, Procedia computer science 145 (2018) 730–739. [26] I. Posner, Robots thinking fast and slow: On dual process theory and metacognition in embodied AI (2020). [27] P. Kerschke, H. H. Hoos, F. Neumann, H. Trautmann, Automated algorithm selection:
Survey and perspectives, Evolutionary computation 27 (2019) 3–45. [28] R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, 2nd Edition, A
Bradford Book, Cambridge, MA, USA, 2018. [29] D. Kim, G. Y. Park, P. John, S. W. Lee, et al., Task complexity interacts with statespace uncertainty in the arbitration between model-based and model-free learning, Nature communications 10 (2019) 1–14. [30] J. R. Busemeyer, J. T. Townsend, Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment., Psychological review 100 (1993) 432. [31] J. M. Hotaling, J. R. Busemeyer, J. Li, Theoretical developments in decision field theory:
Comment on tsetsos, usher, and chater (2010)., Psychological Review (2010).
A.1. Background
We introduce the main ideas of the thinking fast and slow theory. We also describe the main
features of the Multi-alternative Decision Field Theory (MDFT) [
System 1 is guided mainly by intuition rather than deliberation. It gives fast answers to simple
questions. Such answers are sometimes wrong, mainly because of unconscious bias or because
they rely on heuristics or other short cuts [
When the problem is too complex for System 1, System 2 kicks in and solves it with access to
additional computational resources, full attention, and sophisticated logical reasoning. A typical
example of a problem handled by System 2 is solving a complex arithmetic calculation, or a
multi-criteria optimization problem. To do this, humans need to be able to recognize that a
problem goes beyond a threshold of cognitive ease and therefore see the need to activate a more
global and accurate reasoning machinery [
When a problem is new and difficult to solve, it is handled by System 2 [ 29]. However, certain problems, over time as more experience is acquired, pass on to System 1. The procedures System 2 adopts to find solutions to such problems become part of the experience that System 1 can later use with little effort. Thus, over time, some problems, initially solvable only by resorting to System 2 reasoning tools, can become manageable by System 1. A typical example is reading text in our own native language. However, this does not happen with all tasks. An example of a problem that never passes to System 1 is finding the correct solution to complex arithmetic questions.
A.1.2. Multi-Alternative Decision Field Theory
Multi-alternative Decision Field Theory (MDFT) [
Personal Evaluation: Given set of options = {1, . . . , } and set of attributes = {1, . . . , }, the subjective value of option on attribute is denoted by and stored in matrix M. In our example, let us assume that the cafeteria options are Salad (S), Burrito (B) and Vegetable pasta (V). Matrix M, containing the student’s preferences, could be defined as shown in Figure 10 (left), where rows correspond to the options (, , ) and the columns to the attributes and ℎ.
Attention Weights: Attention weights are used to express the attention allocated to each
attribute at a particular time during the deliberation. We denote them by vector W() where ()
represents the attention to attribute at time . We adopt the common simplifying assumption that,
at each point in time, the decision maker attends to only one attribute [
Contrast Matrix: Contrast matrix C is used to compute the advantage (or disadvantage) of
an option with respect to the other options. In the MDFT literature [
At any moment in time, each alternative in the choice set is associated with a valence value. The
valence for option at time , denoted (), represents its momentary advantage (or disadvantage)
when compared with other options on some attribute under consideration. The valence vector
for options 1, . . . , at time , denoted by column vector V() = [1(), . . . , ()] , is
formed by V() = C × M × W(). In our example, the valence vector at any time point in which
W() = [
Preferences for each option are accumulated across the iterations of the deliberation process until a decision is made. This is done by using Feedback Matrix S, which defines how the accumulated preferences affect the preferences computed at the next iteration. This interaction depends on how similar the options are in terms of their initial evaluation expressed in M. Intuitively, the new preference of an option is affected positively and strongly by the preference it had accumulated so far, while it is inhibited by the preference of similar options. This lateral inhibition decreases as the dissimilarity between options increases. Figure 10 (right) shows S for our example [31].
At any moment in time, the preference of each alternative is calculated by P( + 1) = S × P() + V( + 1) where S × P() is the contribution of the past preferences and V( + 1) is the valence computed at that iteration. Starting with P(0) = 0, preferences are then accumulated for either a fixed number of iterations (and the option with the highest preference is selected) or until the preference of an option reaches a given threshold. In the first case, MDFT models decision making with a specified deliberation time, while, in the latter, it models cases where deliberation time is unspecified and choice is dictated by the accumulated preference magnitude. In general, different runs of the same MDFT model may return different choices due to the attention weights’ distribution. In this way, MDFT induces choice distributions over set of options and is capable of capturing well know behavioral effects such as the compromise, similarity, and attraction effects that have been observed in humans and that violate rationality principles [30].