Working with Beliefs: AI Transparency in the Enterprise Ajay Chander Ramya Srinivasan Suhas Chelian Fujitsu Laboratories of America Fujitsu Laboratories of America Fujitsu Laboratories of America Sunnyvale, California, USA Sunnyvale, California, USA Sunnyvale, California, USA Jun Wang Kanji Uchino Fujitsu Laboratories of America Fujitsu Laboratories of America Sunnyvale, California, USA Sunnyvale, California, USA ABSTRACT dramatic improvements in the performance of machine Enterprises are increasingly recognizing that they must learning systems have captured the popular imagination, integrate AI into all of their operational workflows to remain enterprises worldwide have accepted the premise that an competitive. As enterprises consider competing AIs to Augmented Intelligence enterprise is a necessity to survive support a particular business function, explainability is an and compete in the modern digital era. An enterprise with advantage which gets a candidate AI a foot in the door. Our augmented intelligence, wherever possible: experience working with enterprise decision makers considering AI in a decision augmentation role reveals an 1. Augments human sensing with sensors (IoT) additional and possibly more crucial aspect of choosing an 2. Augments human decision making with AI, and AI: the ability of decision makers to interact fluidly with an 3. Augments human action with software and AI. Fluid interactions are necessary when an AI’s hardware robots. recommendation does not match a human decision maker’s The process of onboarding an enterprise to augment its existing beliefs. Interactions that allow the (typically non- human decision making with AI typically follows a technical) human to edit the AI, as well as allow the AI to predictable script. A very common set of questions is asked guide the human, enable a collaborative exploration of the by enterprise clients, typically comprising of: data that leads to common ground where both the AI and the human beliefs have been updated. We outline an illustrative 1. What can AI do for me and for my enterprise? example from our experience that models this dance. Based 2. Is there an AI system that can improve aspect X of on our experiences, we suggest requirements for AI systems my enterprise’s workflow Y? that would greatly facilitate their adoption in the enterprise. 3. How do I choose, personalize, and integrate the system in (2) above into my enterprise? Author Keywords AI; Transparent AI; Accessible AI; Explainable AI; The answer to the first question – presented through the Interactive AI; Tunable AI; Beliefs; Enterprise capabilities of AI systems on external datasets – broadens awareness at the highest levels of the typical enterprise to the ACM Classification Keywords possibilities of modern AI, especially modern machine H.5.m. Information interfaces and presentation (e.g., HCI): learning (ML) systems. This typically leads to the second Miscellaneous; I.2.m. Artificial Intelligence: Miscellaneous. question, which brings focus to a particular workflow Y in INTRODUCTION the enterprise. When presented with a few candidate AI We offer this position paper to the community to share our systems that can improve this workflow Y, some observations and learnings from our vantage point of being explainability of the AI system – typically around a pre- the R&D arm of a global “top 5” IT behemoth. Our parent selected dataset and prediction use-case – is table stakes company is active across a very wide spectrum of IT today. This builds some assurance in the client that they are products, technologies, and services, and in that role interacts not bringing into their enterprise a runaway digital decision with a large variety of enterprises globally. As maker. The final step involves a detailed evaluation of the improvements in the abilities of AI systems, in particular the candidate AI system(s) using data proprietary to the enterprise, which is typically handed off to the corresponding © 2018. Copyright for the individual papers remains with the authors. leadership team and the human decision makers within it. Copying permitted for private and academic purposes. ExSS '18, March 11, Tokyo, Japan It is the perceived capabilities of the AI system in this third step that determine its eventual adoption in the enterprise. The stakeholders evaluating the AI in this stage are typically business domain experts but generally not technical experts. They generally have some strongly held business beliefs about their domain, for example, about how to approach sales in a particular region. These beliefs are borne out of their collective professional experience, and sometimes obtained at significant economic cost. Hence, they tend to We call these sets of technologies collectively Transparent be sticky. A candidate AI may make a recommendation that AI. The rest of our paper will describe aspects 1-3 of is aligned with or aligned against the belief of the business Transparency in the context of an example using a platform stakeholder. When the AI is aligned with the business that we built called AI.AI, short for Accessible and stakeholder, it may be reviewed weakly and its Interactive AI. institutionalization may further existing biases as reflected in the datasets. In this case, it is especially valuable for the AI RELATED WORK system to include bias determination [11, 12] so that they DARPA’s XAI initiative [1] has ignited broad interest in may alert around biased beliefs. When the AI is aligned exploring issues related to the transparency of AI models. As against the business stakeholder, it tends to receive special AI is increasingly integrated into a wide variety of settings, scrutiny. In this case, it is crucially important that the from enterprise assistants to self-driving cars, a wide variety business stakeholders (i.e., the human decision makers) can of users are now interested in understanding the decisions of interact with the AI fluidly as they would with an external AI systems. Accordingly, various notions of transparency human consultant who gives them news that they may not are emerging across different application domains and like at first. In both cases, a successful AI system in the different end-user types. A summary of the feasibility and enterprise is a Belief Worker: it has to learn and stay aware desirability of transparency related notions from an AI of institutional beliefs, and assist in updating them by being engineer’s perspective is offered in [3]. In [5], the authors accessible to a wide variety of potential enterprise users that propose a general taxonomy for the rigorous evaluation of may come to rely on it. interpretable machine learning. A survey of the desired features of transparent AI systems as viewed from a social TRANSPARENT AI and behavioral sciences perspective is provided in [4]. In our experience, the practical adoption of AI systems in Below, we organize other related work within the 4 pillars of enterprises that are making the move to Augmented Transparent AI. Intelligence depends on empowering not just AI engineers but crucially System Integration (SI) engineers and business Accessible AI: Amazon recently announced the release of a stakeholders. Current AI systems, which involve primarily service called Sagemaker [6], a framework for developers an AI engineer as the “human-in-the-loop”, leave out these and data scientists that helps manage the systems important constituencies. Based on our experiences, we infrastructure involved in starting and running AI pipelines. posit the follow 4 pillars of Transparent AI: DataRobot [2] offers an automated machine learning platform as well as services and education to jumpstart AI 1. Accessible AI. SI engineers and business related processes. There are many more such services in the stakeholders should be able to ask questions of AI offing. without going through the AI engineer’s interface. Progress in this area is most robustly being led [2] Explainable AI: The usefulness of explainable models has by the industry, because there is commercial been demonstrated across various application domains such demand for this. as recommendation systems [16] and healthcare [17], to just 2. Explainable AI. The answer that the AI comes back name a couple. A good survey of research around with should be accompanied with some explanations in machine learning can be found in [18]. One explanation, as the audience for this answer is now of the first efforts in this area [13] looked at explaining the no longer just the AI engineer. Progress is this area decisions of classifiers in a model agnostic manner. is most robustly being led by DARPA’s XAI [1] However, a majority of subsequent work has been in project. explaining the decisions of deep learning models using 3. Interactive AI. The non-AI engineer does not have various strategies such as saliency maps [8], influence a dataset to evaluate the AI’s answer against. What functions [9], logical primitives [14, 15], and causal they do have is beliefs. It should be possible for the frameworks [10]. non-AI engineer to interact fluidly with the AI Interactive AI: Towards the goal of democratizing AI access, system to edit the AI, perhaps by editing its dataset Google recently launched “AutoML Vision” [7], an AI in response to its answers. This process would product that enables everyone to build their own customized continue until either the AI is updated or the beliefs machine learning models without much expertise. In [22], are updated or both. researchers present a new system that automates the model 4. Tunable AI. Interactive AI systems enable a selection step, even improving on human motivated user to update an AI through easy performance. Systems that can learn interactively from their interactions. Taking that a step further, Tunable AI end users are gaining importance. [20] is one of the early refers to sets of technologies that can, given an AI efforts in this area. While most progress has been fueled by system, automatically identify usable “tuners” for advances in machine learning, the authors in [19] explore the an AI that can be utilized by end-users. notion of interactivity from the lens of the user. Recently, model-specific interactivity is being introduced through efforts such as [21]. Tunable AI: This area is in its nascent stages. Services like Sagemaker and AutoVision claim to provide auto tuning facilities, but do not focus on the AI consumer. TRANSPARENT AI FOR SALES “WIN” PREDICTIONS Recommendation systems are an important class of AI applications in the enterprise. In the example below, we show how various aspects of transparency were essential in the adoption of an AI system for predicting sales “wins”. Figure 1: The Search for Common Cognitive Ground This is an actual example of the process of selecting AI for an enterprise workflow; names have been anonymized. the AI and pose questions and get immediate answers. Allison then asked: The user who was trying out this predictive AI system was the global SVP of sales for a large enterprise company. Let’s What’s the impact of total contract price on the remainder? call her Allison. Allison used Business Intelligence And received the answer: dashboards custom built for her on a daily and weekly basis to look at various trends in sales data. The AI.AI platform Lower price is no longer better. made it easier for Allison to ask questions of the AI, and to Figure 1. Use high-resolution images, 300+ dpi, legible if receive answers as custom graphical representations with printed in color or black-and-white. Number all figures and accompanying auto-generated text explanations. In this case, include captions REFLECTIONS FOR below, using Insert, Caption. All 1-line AI TRANSPARENCY captions should Abstracting frombethe centered; justify example longer captions. described in the previous Allison’s initial ask to the AI was: section (and other examples from our industrial research How do I increase overall win % on sales contracts? experience), we’d like to offer the following perspective, The AI answered: captured in Figure 1. Total contract price does, lower priced is better. Human experiences tend to be highly dimensional; there are many aspects to the human experience. There is also By means of an explanation, it provided graphical variability to those experiences. Comparatively, human representations of contracts that were won vs. lost, with beliefs, which are borne out of human experiences, may be explanations. described as being lower in dimensionality as well as in A lower contract price as a winning sales strategy is not variability. When we introduce digital actors, digital data, exactly music to a sales executive. Indeed, in this case, this and digital decision making (AI), we end up at different particular recommendation immediately ran into a strongly points on the Dimensionality-Variability graph of Figure 1. held business belief of Allison’s. A certain percentage of Because digital data may not capture everything that is contracts were “churn” contracts, essentially contract experienced, we may view digital datasets as having lower renewals with low price but high “win” probability. The AI’s dimensionality than the data underlying human experiences. response failed Allison’s belief test, and her next ask was: The predictions made by AI from digital datasets, may then be further lower in dimensionality, similar to the Hmm. Churn contracts (i.e., contract renewals) are affecting dimensionality difference between experiences and beliefs. the result. Let’s remove them. Two issues show up when a human being is presented with And the AI’s response: AI decisions. If they don’t believe them because they do not Same result after removing churn contracts. align with their beliefs, they point to the lack of awareness of the dataset with respect to their experiences. Let’s call this This led to Allison digging in: the Awareness Gap. The awareness gap is often used as a first line of defense to reject AI that offers no way to edit it, Really? I wonder why. Show me the data that matches these conditions. independent of its explainability features. This was the beginning of an extensive series of edits that Similarly, if an AI’s decision is not aligned with the user’s Allison performed using the platform to update the AI by beliefs, it is important that the AI be able to understand this asking it to look at a variety of subsets of the original data, gap and persuade the user by applying techniques from asking it various questions along the way. The process ended cognitive science. One issue we see in the explainability once she arrived at an AI-driven insight: most contracts, literature is too much of an implicit assumption that despite not being coded as such in the dataset, had churn like rationality is a winning persuasive argument whereas in reality this is far from the case. Closing the Persuasion Gap characteristics. This was a huge insight at the level of a sales requires, in our experience, the ability of the AI to engage SVP, enabled because of her ability to fluidly interact with mechanisms that human beings regularly use to update their 7. AutoML Vision, Google, 2018. Retrieved February 10, belief systems, and recourse to rationality is only one such 2018 from https://www.cnbc.com/2018/01/17/google- mechanism. launches-cloud-automl.html We suggest that research on the ability of the human actor to 8. R. Selvaraju et.al. 2017. Grad-CAM: Visual guide the AI to help it close the Awareness Gap, and on the Explanations from Deep Networks via Gradient-based ability of the AI actor to guide the human to help close the Localization, In International Conference on Computer Persuasion Gap is essential for practical human-AI Vision. collaboration. 9. P. Koh et.al. 2017. Understanding Black-box Predictions via Influence Functions. In International Conference on Machine Learning. CONCLUSION The justified excitement about modern AI has brought many 10. D. Alvarez et.al. 2017. A Causal Framework for people in non-technical roles in the enterprise into the sphere Explaining the Predictions of Black-box sequence to of AI interaction. Enterprises are re-architecting themselves sequence models. In ArXiv Report to go from “Intelligences Apart” – human and machines 11. S. Tan et.al. 2017. Detecting Bias in Black-box Models intelligences being separate – to true human-AI Using Transparent Model Distillation. In ArXiv Report. collaboration. In many enterprises, incorporating AI into 12. T. Bolukbai, et.al. 2016. Man is to Computer workflows goes through a pivotal stage of testing if it can programmer as Woman is to Homemaker? Debiasing work well with the existing human decision makers in that Word Embeddings, In ArXiv Report. workflow. Human decision makers use alignment with their existing beliefs as a way of accepting AI into their team, 13. M.T.Ribeiro et.al. 2016."Why Should I Trust You?": much as they might for accepting a new human team Explaining the Predictions of Any Classifier. In member. For AI to pass this test, in addition to being Conference on Knowledge Discovery and Data Mining. explainable, it needs to be easily accessible and interactive. 14. H. Lakkaraju, et. al. 2016. Interpretable Classifiers AI that is transparent in these ways can be edited usably by using Rules and Bayesian Analysis. In The Annals of non-technical stakeholders when it fails their belief tests, and Applied Statistics 9(3), 1350-1371. engenders trust in the process. In addition, we suggest that 15. T.F.Wu, et. al. 2016. Learning And-Or Models to AI look to mechanisms from the cognitive sciences to both Represent Text and Occlusion for Car Detection and identify beliefs in their users and ways of updating those Viewpoint Estimation, In IEEE Transactions on beliefs that leverage techniques in addition to rational Pattern Analysis and Machine Intelligence, 38(9), explanation. 1829-1843. 16. K. Muhammad et. al. 2016. Explanation-based REFERENCES Ranking in Opiniated Recommender Systems, In Irish 1. D. Gunning. 2016. Explainable Artificial Intelligence Conference on Artificial Intelligence and Cognitive (XAI). Retrieved December 20, 2017 from Science. https://www.darpa.mil/program/explainable-artificial- 17. H. Lakkaraju, et. al. 2017. Learning Cost-Effective and intelligence Interpretable Treatment Regimes, In International 2. DataRobot. 2017. Automated Machine Learning. Conference on Artificial Intelligence and Statistics. Retrieved December 20, 2017 from 18. O. Biran et. al. 2017. Explanation and Justification in https://www.datarobot.com/ Machine Learning, In International Joint Conference 3. Z. Lipton. 2016. The Mythos of Model Interpretability, on Artificial Intelligence. In International Conference on Machine Learning 19. S. Amershi, et.al. 2017. Power to the People: The Role Workshops. of Humans in Interactive Machine Learning, In AI 4. T.Millers et.al. 2017. Explainable AI: Beware of Magazine. Inmates Running the Asylum, In ArXiv Report. 20. C. Isbell et. al. 1999. The Parallel Problems Server: an 5. F.D.Velez, et.al. 2017. Towards a Rigorous Science of Interactive Tool for Large Scale Machine Learning. In Interpretable Machine Learning, In ArXiv Report. Neural Information Processing Systems. 6. AWS RE:INVENT, 2017. Retrieved February 10, 2018 21. Y. Hu et.al, 2011. Interactive Topic Modeling, In from https://techcrunch.com/2017/11/29/aws-releases- Association of Computational Linguistics. sagemaker-to-make-it-easier-to-build-and-deploy- 22. T. Swearingen et.al. 2017. ATM: A distributed, machine-learning-models/ collaborative, scalable system for automated machine learning. In IEEE International Conference on Big Data.