This workshop's call for participation poses the question: What should be in an explanation? One route toward answering this question is to turn to theories of how humans try to obtain information they seek. Information Foraging Theory (IFT) is one such theory. In this paper, we present lessons we have learned about how IFT informs Explainable Artificial Intelligence (XAI), and also what XAI contributes back to IFT.
We have been working to answer this question in a domain
often used for AI research, namely Real-Time Strategy (RTS)
games, from two sides. First, to understand what a
highquality supply of explanations might contain, we conducted a
qualitative analysis of the utterances of expert explainers [
There have been previous explorations into what should be in
an XAI explanation [
ExSS 2018, March 11, 2018, Tokyo, Japan.
gap and approach our investigation. IFT is based on a
predatorprey model [
In our user study [
Things we’ve learned from studying Prey
In IFT, predators seek prey, which are the pieces of information
in the environment they they think they need. In the context
of XAI, such prey are evidence of the agents decision process,
which are then used to create explanations for agents’ actions.
To investigate the information participants were trying to
obtain, we analyzed the questions that they asked each other.
We categorized their questions according to the Lim-Dey
intelligibility types [
Although most previous XAI research has found Why to be
highly demanded information, our participants rarely sought
Why or Why-Didn't information. Instead, our participants
showed a strong preference for asking What questions.
What was so interesting about What? The participants’ What
information seeking was about finding out more about state
than they currently knew. Our participants did so primarily in
three categories: drill down, higher level, and temporal.
Drill down Whats usually involved participants spatially
navigating around the map, sometimes opening up objects or
menus to access more detailed game state information. E.g.
“Is the human building any new stuff now?” The second
category, higher-level Whats, involved trying to abstract a little
above the details, to gain a higher level of understanding of
the game state. E.g. “What’s going on over there?” The third
category, temporal Whats, involved finding out more about
differences or similarities in state over time. E.g. “When did
he start building...?”
Finally, to investigate whether our distribution of What vs.
Why results were reasonably representative for this domain,
we compared our participants’ questioning (i.e., explanation
demand) against the answers (explanation supply) produced by
professional explainers in this domain, namely shoutcasters 2
2Shoutcasters are sportscasters for e-sports. They perform a similar
sort of analysis as our participants were doing, but with the added
[
Implications: Taken together, these results show that in this domain, participants placed very high value on state information — but not always at the same granularity, and not always restricted to a single moment in time. How an XAI system can satisfy these explanation needs may not be straightforward, but one of our findings suggests a way forward: Shoutcasters may be usable as a gold standard. That is, the remarkable similarity between the frequency of shoutcasters’ utterances (supply) and participants’ desired prey (demand) for most intelligibility types suggests that XAI explanation systems in the RTS domain may be able to model their explanation content, timing, and construction, around shoutcasters’ explanations. Things we’ve learned from studying Paths In IFT, prey exists within some patch(es), and the forager navigates between patches by following paths, made up of one or more links. Investigating the paths participants used revealed a great deal of information about the kinds of costs they can incur in the RTS domain when seeking information. Traditionally, IFT looks at the navigation cost to get to a patch (here, explanation), usually in number of clicks, and the cognitive cost of absorbing the necessary information in the patch once there. These costs are relevant to XAI too, but our investigation discovered participants incurred significant cognitive effort in both path discovery and path triage. Why so expensive? Professional RTS players perform several hundred actions per minute (APM), and each such action potentially destroys or updates the available foraging paths. This produces an information environment in which foraging paths are numerous, rapidly updating, and have limited lifespan. Thus, our participants were faced with many more potentially useful foraging paths than they could possibly follow, and had to spend significant effort just choosing a path.
Some coped with these costs by adhering to a single foraging path throughout the task, rewinding rarely. These participants minimized their cognitive costs of choosing, but paid a high information cost, because by not following other paths, they missed out on potentially explanatory information. Others chose not to pay this information cost, and instead paid a navigation cost by often rewinding and pausing to spatially explore. Rewinding also incurs substantial cognitive cost, as more context information must be tracked — but that extra context may provide useful explanatory power. constraint that they must analyze and explain the game in real-time, so they cannot pause or rewind. Interestingly, when costs of choice were low, participants’ explanation seeking followed fairly traditional foraging patterns. For example, early in the game, participants scrutinized the game objects carefully and in detail — a sharp contrast to late in the game when many more game objects and foraging paths were present. This could suggest that as the information environment grows in complexity, users in this domain will seek explanations at a higher level of abstraction (i.e., a group of units as opposed to a single unit).
Implications: XAI explanation system would benefit from incorporation of an explanation recommender. Such a recommender could take into account both the human cognitive cost of considering too many paths when few can be followed, and the information cost of neglecting some path too long. For example, if the domain is well known to an explanation system a priori, such a recommender may help guide users (reducing their cognitive cost of choosing) to the explanations that are the most important (reducing the information cost of missing important explanatory information). In this case, it appears we know Expansions are important before any analysis occurs. Things we’ve learned from studying Scent and Cues Recall that we requested that participants write down key decision points. To forage for these, they followed cues, which are information features connected to links in the environment. In our study, cues were the same across sessions, because everyone replayed the same game. Unlike cues, scent is “in the head” – it is foragers’ assessment of cues’ meaning. Unfortunately, participants missed information that we suspect they would have found key, because some cues distracted them. Our videos showed that what participants were looking at when they were distracted — the “distractor cues” — tended to be combat-oriented and affected even simple game states. For example, in Figure 2, a nearly full decision column suggests that participants tended to agree that this decision was key. A nearly full participant pair row suggests that this pair consistently found this type of decision to be key. Thus, missing dots (e.g., see the red box) correspond to times when a participant pair was distracted from a key event.
In fact, the scents emanating from some types of cues seemed
to overpower others consistently. Consider the example in
Figure 3, which shows how Fighting tended to overpower
Scouting . The top image in the figure shows all of the Scouting
decision points our participants identified, while the bottom
image shows all of the Fighting decision points. The red
line going through both images is the point at which combat
first begins – and also the time when scouting is usually last
noticed, despite being ongoing throughout the game.
Implications: Distractions abound, and may be systematic.
Some facets of the environment may elicit emotional response
and receive undue attention as a result. In this case,
participants preferred investigating Fighting over Scouting .
Another implication relates to an XAI explanation system’s
user interface’s support for human workflow. Paths are easily
forgotten in the presence of interruptions. Each new action
may interrupt the current foraging path, which leads to people
forgetting things – made worse by sheer path quantity.
Previous research has found that To-Do Listing [
In the previous sections, we focused on things we learned
about XAI by applying IFT to our data set. Now, we turn
the other direction, since this study is the first to apply IFT
to XAI and the RTS domain. The RTS domain presents an
extremely complex and rapidly changing environment, more
so than other IFT environments in the literature like Integrated
Development Environments (IDEs) and web sites [
As discussed, our participants were faced with many paths,
and had to rapidly triage which paths to follow. This presents
an interesting IFT challenge. Previous research [
Turning to prey, in the XAI setting, the prey is evidence of
the agent’s decision process. Establishing trust in an XAI
system requires the user to know how it behaves in many
circumstances. Thus, the prey is “in pieces” – meaning that bits
of it are scattered over many patches. As previous work [
Our use of the IFT lens allows us to leverage results obtained from applying IFT to non-XAI domains, while also improving the ability to transport/generalize findings among XAI domains. By connecting XAI to IFT foundations, we can bring to XAI a real theoretical foundation based on what informations humans want and how they look for it.
ACKNOWLEDGMENTS This work was supported by DARPA #N66001-17-2-4030 and NSF #1314384. Any opinions, findings and conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of NSF, DARPA, the Army Research Office, or the US government.