Enterprises are increasingly recognizing that they must integrate AI into all of their operational workflows to remain competitive. As enterprises consider competing AIs to support a particular business function, explainability is an advantage which gets a candidate AI a foot in the door. Our experience working with enterprise decision makers considering AI in a decision augmentation role reveals an additional and possibly more crucial aspect of choosing an AI: the ability of decision makers to interact fluidly with an AI. Fluid interactions are necessary when an AI's recommendation does not match a human decision maker's existing beliefs. Interactions that allow the (typically nontechnical) human to edit the AI, as well as allow the AI to guide the human, enable a collaborative exploration of the data that leads to common ground where both the AI and the human beliefs have been updated. We outline an illustrative example from our experience that models this dance. Based on our experiences, we suggest requirements for AI systems that would greatly facilitate their adoption in the enterprise.
INTRODUCTION We offer this position paper to the community to share our observations and learnings from our vantage point of being the R&D arm of a global “top 5” IT behemoth. Our parent company is active across a very wide spectrum of IT products, technologies, and services, and in that role interacts with a large variety of enterprises globally. As improvements in the abilities of AI systems, in particular the © 2018. Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes. ExSS '18, March 11, Tokyo, Japan dramatic improvements in the performance of machine learning systems have captured the popular imagination, enterprises worldwide have accepted the premise that an Augmented Intelligence enterprise is a necessity to survive and compete in the modern digital era. An enterprise with augmented intelligence, wherever possible: 1. 2. 3.
The process of onboarding an enterprise to augment its human decision making with AI typically follows a predictable script. A very common set of questions is asked by enterprise clients, typically comprising of:
2. Is there an AI system that can improve aspect X of
my enterprise’s workflow Y?
3. How do I choose, personalize, and integrate the
system in (2) above into my enterprise?
The answer to the first question – presented through the
capabilities of AI systems on external datasets – broadens
awareness at the highest levels of the typical enterprise to the
possibilities of modern AI, especially modern machine
learning (ML) systems. This typically leads to the second
question, which brings focus to a particular workflow Y in
the enterprise. When presented with a few candidate AI
systems that can improve this workflow Y, some
explainability of the AI system – typically around a
preselected dataset and prediction use-case – is table stakes
today. This builds some assurance in the client that they are
not bringing into their enterprise a runaway digital decision
maker. The final step involves a detailed evaluation of the
candidate AI system(s) using data proprietary to the
enterprise, which is typically handed off to the corresponding
leadership team and the human decision makers within it.
It is the perceived capabilities of the AI system in this third
step that determine its eventual adoption in the enterprise.
The stakeholders evaluating the AI in this stage are typically
business domain experts but generally not technical experts.
They generally have some strongly held business beliefs
about their domain, for example, about how to approach
sales in a particular region. These beliefs are borne out of
their collective professional experience, and sometimes
obtained at significant economic cost. Hence, they tend to
be sticky. A candidate AI may make a recommendation that
is aligned with or aligned against the belief of the business
stakeholder. When the AI is aligned with the business
stakeholder, it may be reviewed weakly and its
institutionalization may further existing biases as reflected in
the datasets. In this case, it is especially valuable for the AI
system to include bias determination [
TRANSPARENT AI
In our experience, the practical adoption of AI systems in
enterprises that are making the move to Augmented
Intelligence depends on empowering not just AI engineers
but crucially System Integration (SI) engineers and business
stakeholders. Current AI systems, which involve primarily
an AI engineer as the “human-in-the-loop”, leave out these
important constituencies. Based on our experiences, we
posit the follow 4 pillars of Transparent AI:
1. Accessible AI. SI engineers and business
stakeholders should be able to ask questions of AI
without going through the AI engineer’s interface.
Progress in this area is most robustly being led [2]
by the industry, because there is commercial
demand for this.
2. Explainable AI. The answer that the AI comes back
with should be accompanied with some
explanation, as the audience for this answer is now
no longer just the AI engineer. Progress is this area
is most robustly being led by DARPA’s XAI [
We call these sets of technologies collectively Transparent AI. The rest of our paper will describe aspects 1-3 of Transparency in the context of an example using a platform that we built called AI.AI, short for Accessible and Interactive AI.
RELATED WORK
DARPA’s XAI initiative [
Accessible AI: Amazon recently announced the release of a service called Sagemaker [6], a framework for developers and data scientists that helps manage the systems infrastructure involved in starting and running AI pipelines. DataRobot [2] offers an automated machine learning platform as well as services and education to jumpstart AI related processes. There are many more such services in the offing.
Explainable AI: The usefulness of explainable models has
been demonstrated across various application domains such
as recommendation systems [
Interactive AI: Towards the goal of democratizing AI access,
Google recently launched “AutoML Vision” [7], an AI
product that enables everyone to build their own customized
machine learning models without much expertise. In [
Tunable AI: This area is in its nascent stages. Services like Sagemaker and AutoVision claim to provide auto tuning facilities, but do not focus on the AI consumer.
TRANSPARENT AI FOR SALES “WIN” PREDICTIONS Recommendation systems are an important class of AI applications in the enterprise. In the example below, we show how various aspects of transparency were essential in the adoption of an AI system for predicting sales “wins”. This is an actual example of the process of selecting AI for an enterprise workflow; names have been anonymized. The user who was trying out this predictive AI system was the global SVP of sales for a large enterprise company. Let’s call her Allison. Allison used Business Intelligence dashboards custom built for her on a daily and weekly basis to look at various trends in sales data. The AI.AI platform made it easier for Allison to ask questions of the AI, and to receive answers as custom graphical representations with accompanying auto-generated text explanations. In this case, Allison’s initial ask to the AI was:
By means of an explanation, it provided graphical representations of contracts that were won vs. lost, with explanations.
A lower contract price as a winning sales strategy is not exactly music to a sales executive. Indeed, in this case, this particular recommendation immediately ran into a strongly held business belief of Allison’s. A certain percentage of contracts were “churn” contracts, essentially contract renewals with low price but high “win” probability. The AI’s response failed Allison’s belief test, and her next ask was:
This was the beginning of an extensive series of edits that Allison performed using the platform to update the AI by asking it to look at a variety of subsets of the original data, asking it various questions along the way. The process ended once she arrived at an AI-driven insight: most contracts, despite not being coded as such in the dataset, had churn like characteristics. This was a huge insight at the level of a sales SVP, enabled because of her ability to fluidly interact with the AI and pose questions and get immediate answers. Allison then asked:
Figure 1. Use high-resolution images, 300+ dpi, legible if printed in color or black-and-white. Number all figures and REinFcLluEdCeTcIOapNtiSonFsORbelAoIwT,RuAsNinSgPIAnRseErNt,CCYaption. All 1-line Acbasptrtaiocntisnsghofuroldmbethceenteexraemd;pjluestidfeyslcornibgeerd cianptitohnes.previous section (and other examples from our industrial research experience), we’d like to offer the following perspective, captured in Figure 1.
Human experiences tend to be highly dimensional; there are many aspects to the human experience. There is also variability to those experiences. Comparatively, human beliefs, which are borne out of human experiences, may be described as being lower in dimensionality as well as in variability. When we introduce digital actors, digital data, and digital decision making (AI), we end up at different points on the Dimensionality-Variability graph of Figure 1. Because digital data may not capture everything that is experienced, we may view digital datasets as having lower dimensionality than the data underlying human experiences. The predictions made by AI from digital datasets, may then be further lower in dimensionality, similar to the dimensionality difference between experiences and beliefs. Two issues show up when a human being is presented with AI decisions. If they don’t believe them because they do not align with their beliefs, they point to the lack of awareness of the dataset with respect to their experiences. Let’s call this the Awareness Gap. The awareness gap is often used as a first line of defense to reject AI that offers no way to edit it, independent of its explainability features.
Similarly, if an AI’s decision is not aligned with the user’s beliefs, it is important that the AI be able to understand this gap and persuade the user by applying techniques from cognitive science. One issue we see in the explainability literature is too much of an implicit assumption that rationality is a winning persuasive argument whereas in reality this is far from the case. Closing the Persuasion Gap requires, in our experience, the ability of the AI to engage mechanisms that human beings regularly use to update their belief systems, and recourse to rationality is only one such mechanism.
We suggest that research on the ability of the human actor to guide the AI to help it close the Awareness Gap, and on the ability of the AI actor to guide the human to help close the Persuasion Gap is essential for practical human-AI collaboration.
CONCLUSION The justified excitement about modern AI has brought many people in non-technical roles in the enterprise into the sphere of AI interaction. Enterprises are re-architecting themselves to go from “Intelligences Apart” – human and machines intelligences being separate – to true human-AI collaboration. In many enterprises, incorporating AI into workflows goes through a pivotal stage of testing if it can work well with the existing human decision makers in that workflow. Human decision makers use alignment with their existing beliefs as a way of accepting AI into their team, much as they might for accepting a new human team member. For AI to pass this test, in addition to being explainable, it needs to be easily accessible and interactive. AI that is transparent in these ways can be edited usably by non-technical stakeholders when it fails their belief tests, and engenders trust in the process. In addition, we suggest that AI look to mechanisms from the cognitive sciences to both identify beliefs in their users and ways of updating those beliefs that leverage techniques in addition to rational explanation.