Introduction

Machine learning consists of methods that use data and algorithms to build models that make inferences about an application from provided examples [1]. Both the opportunities and limitations of machine learning are rooted in its reliance on building models from data and, therefore on the quality of the data used to train and test the models [2]. As our society's dependence on machine learning grows, it is important to ensure that machine learning models perform well and are interpretable and transparent. This is a tradeoff that is often augmented by opaque transformations in the input data (i.e., feature engineering), which makes it challenging to assess the effectiveness of the input data on the outcome [3]. Numerous challenges persist, including biases, discrimination, lower performance, lack of transparency, and explainability. While popular approaches have emerged for explaining predictions of classifiers (Layerwise relevance propagation, Local Interpretable Model-Agnostic Explanations, and many others) [4], there are criticisms about explanations being dependent on the choice of hyperparameters and how these models have different explanations for similar instances in the data. The objective of this research is to investigate how to improve machine learning explainability by incorporating domain knowledge of the application. The contribution is to propose a framework for progressing from human mental models to machine learning models through the use of conceptual models. Based on the framework we develop a ConceptSuperimposition method for increasing explainability of machine learning models. We illustrate the application of this method in the home mortgage domain.

Machine Learning and Conceptual Modeling

Supervised learning guides the learner in acquiring knowledge in a domain through examples, so new cases can be handled in a manner most appropriate based on the knowledge learned from similar cases. Modern supervised machine learning has been taking advantage of the availability of data, and developing methods and techniques (e.g., deep learning neural networks, reinforcement learning), which rely on large volumes of high-quality data for performance improvement. The increase in the use of complex machine learning models has brought about challenges in explaining the decision logic of these models. Transparency research in AI is a growing societal concern and a growing research area [4,5]. A generally overlooked approach to explainability, however, is how to incorporate domain knowledge that a user or designer might possess. This knowledge would manifest itself in mental models which contribute to the development of conceptual models that support the interpretation of machine learning models and outcomes. Conceptual modeling formally describes "some aspects of the physical and social world around us for the purposes of understanding and communication" [6, p. 2]. Humans use conceptualizations about domains for representing specific or abstract situations in domains. Recent research has proposed combining conceptual modeling with artificial intelligence or, specifically machine learning [7,8,9,10,11,12,13]. The main argument is that doing so can provide reliable rules about the domain without being dependent on extracting them from the data. Despite these efforts, conceptual models are rarely used in the process of machine learning. At the same time, machine learning invariably relies on human mental modelsrepresentations of reality in the minds of data scientists or users of machine learning models, who either develop or interpret machine learning solutions, in light of their mental models. Unlike conceptual models, mental models are not explicit, and hence may contain biases.

Research in psychology and other disciplines (e.g., philosophy, cognitive science), dealing with decision-making, has argued that, when making decisions, humans construct one or more

Mental Models

Subjective/Acquired knowledge

Conceptual Models

Shared understanding

Entities and events

Application Domain mental models of a domain [14,15]. A mental model is a representation of a possible state of affairs in reality [15]. Mental models are the central mechanism for coping with diversity and change, allowing someone to act in an informed and effective manner. In a typical machine learning scenario, we may encounter different customers purchasing goods or services from a store. However, we can reduce this complexity by constructing a handful of mental models that segment customers (e.g., loyal repeat customer, high volume customers, one-time buyer). Mental models, are important for problem-solving, including in ML contexts, but are also prone to bias and error. If left unchecked, the biases arising from the suboptimal formation and use of mental models, could affect the judgment of machine learning developers, and result in a variety of machine learning problems. Hence, we seek to augment mental models with conceptual models so the representations will be more formal and externalized and better equipped at mitigating the cognitive biases inherent in mental models. If, in reality, mental models are inconsistent, or even contradictory, with the machine learning model or the data, it is possible that: (1) subjects have selected inadequate conceptual statements for a situation, (2) perception and measurement about reality are distorted, or (3) conceptual statements are flawed.

Data Scientists ConceptualizationsPhysical

In this paper we, therefore, propose improving explainability of complex machine learning models based on the externalizing the mental models via conceptual models. Mental models [14] can represent components, states and structural relations, basic operational rules and on general scientific principles, events and processes, and ontological knowledge. They represent individual knowledge extracted from either physical or abstract domains that is used for various cognitive tasks, including understanding, communication, navigation, and decision making. Mental models are often understood as surrogate models used for simulation and prediction as well as a means for constructing advanced mental models [16]. Conceptual models are shared representations expressed in various forms, such as texts and graphics. According to model theory, conceptual models are projected and abbreviated representations made by human experts and used for a purpose [17]. In contrast, machine learning models are statistical abstractions derived by algorithmic fitting mathematical functions to data according to a purpose given by an objective functions. Data used for model fitting and also for construction of conceptual models are representations of domains themselves. In essence, mental models are individually constructed, conceptual models are socially constructed, and machine learning models are algorithmically constructed. Designing information systems means that all three model types are synchronized and balanced by negotiation for finding a satisfiable equilibrium between these models. (See Figure 1). A gap exists between mental models and conceptual models versus machine learning models because internal representations of machine learning models are generally inaccessible. Thus, research is needed on interpretability and explainability of machine learning models as a means for alignment of all three model types.

ConceptSuperimposition Method

Following the Framework for Mental Models and Conceptual Models for Machine Learning (cf. Figure 1), we advance a new method -ConceptSuperimposition. This is a method which extends our early work on Superimposition method. Superimposition adds structural semantic information to the outputs of machine learning to support explainability [10]. While this information is absent in current ML practice, it is routinely employed by humans to understand their day-to-day experiences. As per our Framework, a machine learning model is a model of some domain (e.g., credit card fraud, image classification, online auctions). The machine learning model is a set of rules for estimating a value of interest or discriminating among the cases of interest based on previously provided domain examples.

Superimposition maps the output of machine learning models (i.e., the features, rules and transformation functions) onto a conceptual model of the domain. The method suggests indicating inside the conceptual model information about the rules of the machine learning models. This step depends on the type of machine learning model. For example, if a regression model is used, these rules can be represented as feature weights or feature coefficients. These coefficients can be appended to the attributes in the conceptual model, or the attributes can be highlighted differently to indicate the different relative importance of each attribute. Features are conceived and defined based on domain knowledge. For instance, the feature 𝑝𝑟𝑜𝑝𝑒𝑟𝑡𝑦_𝑣𝑎𝑙𝑢𝑒 is necessary for evaluating a loan application. A conceptual model clusters features into concepts (e.g., classes, entity types). The relations (e.g., association, type of, part of) provide connections between the concepts. Generally, concepts that are directly connected via a relation are semantically closer to each other than indirectly connected concepts.

The final step of the Superimposition method involves analyzing the resulting conceptual model to gain a clearer understanding of the underlying rules machine learning models use to makes its decisions, and to identify opportunities to improve the machine learning model further.

A key limitation of the early version of the Superimposition method is in the lack of representation of concepts and their relationships. The method merely attached feature weights to the conceptual model. Yet, the understanding of the impact of the concepts (e.g., entity types) on the target is also of critical importance. Furthermore, the early formulation of the Superimposition lacked formalization. We address both shortcomings in the formalized and holistic ConceptSuperimposition method. We propose a ConceptSuperimposition method for assessing the alignment between conceptual models and machine learning models. This method consists of three steps:

Marginal contribution

Machine learning (ML) models are globally fitted to datasets. Often, ML models are black boxes without direct access to feature contributions to outcomes. Therefore, simpler models (surrogate models) are locally fitted ex-post to ML models. Surrogate models provide information on local contribution of features to outcomes (e.g., LIME or SHAP [18]). SHAP (Shapley Additive Explanation) values are Shapley values of a conditional expectation function of the machine learning model, i.e., the fitted model is used for determining local contribution of single features to an outcome.

Shapley values formalize coalition games and determine additive marginal contributions of single players to an overall payoff of the coalition of players. They are defined by an operator 𝜑 that assigns for each game 𝑣 a vector of payoffs 𝜑(𝑣) = (𝜑 1 , ..., 𝜑 𝑛 ). 𝜑 𝑖 (𝑣) is player i's marginal and additive contribution to the outcome of a game over all permutations with all other players [19]. Shapley values are locally accurate, i.e. match the original model 𝑓 (𝑥), is not affected by missing values and are consistent wrt. unequality relation between two models 𝑓 (𝑥) and 𝑓 ′ (𝑥) [18]. The Shapley value of a feature 𝑖 is a weighted mean of its marginal value of feature 𝑖, averaged over all possible subsets of features:

𝜑 𝑖 = ∑︀ 𝑆⊆𝐹 ∖{𝑖} |𝑆|!(|𝐹 |−|𝑆|−1)! |𝐹 |! [𝑓 𝑆∪{𝑖} (𝑥 𝑆∪{𝑖} ) − 𝑓 𝑆 (𝑥 𝑆 )]

with 𝐹 the set of all features and 𝑓 𝑆∪{𝑖} with a model trained with feature 𝑖 and 𝑓 𝑆 (𝑥 𝑆 ) without. SHAP values determine additive contributions of input features 𝑋 on the prediction of an output feature 𝑦, i.e., 𝑦 = 𝑓 (𝑋), agnostic to the machine learning model used with 𝑓 the original model and Φ a vector of all Shapley value 𝜑 𝑖 for all independent features. Input feature 𝑥 ∈ 𝑋 is transformed into simplified vector 𝑧 ∈ 𝑍 with 𝑧 ∈ {0, 1}, i.e., SHAP value 𝜑 𝑖 𝑧 𝑖 is zero if feature 𝑧 𝑖 is missing and 𝜑 𝑖 carries the whole marginal contribution of a feature to an outcome otherwise.

Feature contribution

We now use marginal contributions for defining conceptually generalized contributions of input features. Given a conceptual model 𝐶𝑀 with a bidirectional mapping of concepts to all features in 𝑂. For each concept 𝑐 in 𝐶𝑀 , a n-ary concept contribution vector 𝑔 𝑜 𝑐 is constructed by Hadamard product of Shapley vector Φ and input vector 𝑥 𝑐 for an output feature 𝑜 in output concept 𝑂. 𝑥 𝑐 has only feature values associated with concept 𝑐 and value 0 everywhere else. Vector 𝑔 𝑜 𝐶 is the contribution of all input concepts on an outcome feature 𝑜.

𝑔 𝑜 𝑐 = Φ ∘ 𝑥 𝑐 and 𝑔 𝑜 𝐶 = Π 𝑐∈𝐶 𝑔 𝑜 𝑐 ∘ 1 𝑛

First the average of all SHAP values per input feature is calculated. 𝑔 ¯𝑜 𝑐 is the normalized value for all 𝑔 𝑜 𝑐 , ie., relative contribution of feature to an outcome feature. By normalization over all input data 𝑋, 𝑔 ¯𝑜 𝑐 is determined that provides relative contributions of each feature of concept 𝑐 with respect to output feature 𝑜, ie. input features are superimposed on 𝑐 relative to 𝑜.

Concept contribution

Concept contributions depend on the type of outcome features, i.e. application of classification or regression. For binary classification, we define 𝑓 𝑂 (𝑋) as the sum of contributions of all feature contributions of input concepts on a 𝑜 in output concept 𝑂 except contributions of features in 𝑂 due to implicit strong collinearity with the output feature 𝑜:

𝑓 𝑂 (𝑋) = ∑︀ 𝑐∈𝐶∖{𝑂} 𝑔 ¯𝑜⊤ 𝑐 • 1 𝑛

For regression, feature contributions are evaluated relative to features of concept 𝑂. The mean of output features except outcome feature 𝑜 are calculated (𝜔). Only those input features are considered with a larger feature contribution than 𝜔.

𝑓 𝑂 (𝑋) = ∑︀ 𝑐∈𝐶∖{𝑂} 𝑔 ¯𝑜⊤ 𝑐 • (1/𝜔 𝑛 − 1)

Concept contributions are the weighted summation of all summed up feature contributions. The weight 𝑤 is the count of all features minus selected features plus 1. This accounts for decreasing feature contribution values with the number of features. Only features with strong feature contributions are selected. If, for instance, only one input feature is selected, feature contributions are not affected while a large 𝑤 will reduce feature contributions on concept contributions.

𝜅 𝑂 (𝑋) = 1/𝑤 * ∑︀ 𝑜∈𝑂 𝑓 𝑂 (𝑥)

It shall be noted, that 𝜅 𝑂 (𝑋) depends on the type of prediction. Values for 𝜅 𝑂 (𝑋) for classifications can be compared with one another and for regression respectively but 𝜅 𝑂 (𝑋) cannot be compared between classification and regression.

𝜅 𝑂 is determined for permutations over all homogeneous concepts 𝑐 ∈ 𝐶, i.e., 𝜅 𝑂 is determined for all concepts 𝑐 ∈ 𝐶. This provides a measure for local contributions of input concepts on a homogeneous concept given input 𝑥. Concept contributions on heterogeneous concepts requires a functional model that integrates features 𝑜 ∈ 𝑂.

A concept 𝑐 with little concept contribution 𝜅 𝑂 (𝑋) on an output concept 𝑐 𝑝 has a weak conceptual relation with 𝑐 𝑝 , i.e., the outcome is only weakly affected by the presence of 𝑐 𝑖 . Conceptual relations are directed form 𝑐 𝑖 to 𝑐 𝑝 due to the game-theoretic construction of Shapley values. The semantics of conceptual relations are constrained by the concepts.

ConceptSuperimposition

Feature contributions provide a directed similarity measure between input features and output features analog to local similarity measures in social networks (e.g., [20]). Feature contributions are consistent with the properties of additive feature attribution methods, ie. local accuracy, missingness and consistency [18].

Concept contribution abstracts from features to concepts and determines directed contributions between directly connected concepts (cf. Game 1 [21]). Additive feature attribution properties are not maintained because lack of output values on concept level. Therefore, concept contribution is a score that measures directed local contribution of one concept to another derived by feature contributions.

We call the attribution of concepts by concept contributions ConceptSuperimposition. It closes a cycle between conceptual models, data and ML models that consists of three steps. First, conceptual models provide constraints on data that is considered for ML model development. Second, data is used for constructing ML models. Concept contributions elevate patterns found by ML models to a conceptual level that is fed back to conceptual models. Thus, concept contributions can be used for confirmation of conceptual models, i.e., for evaluation whether identified patterns from data are consistent with conceptual models and, thus, shared understanding of actors involved.

Example

To illustrate the application of ConceptSuperimposition, we use publicly available data (10 GB) from the Home Mortgage Disclosure Act (HMDA) website (https://www.consumerfinance.gov/data-research/hmda/). This data contains the 2020 mortgage application data collected in the U.S. under the Home Mortgage Disclosure Act. The dataset consists of a sample of 3,481,348 applications for single-family, principal residence purchases, i.e. 5% of the HMDA dataset. The data is comprised of 99 variables including payment history, credit history, credit mix, demographics, income, and characteristics of the loan (e.g., purpose of the loan, interest rate, total loan costs), and census data (e.g., census tract, tract population).The target variable is to predict whether a mortgage is originated (target = 1) or denied (target = 0). Of the sample applications in the dataset, 44.3% of the applications were for refinancing, 32.42% were for home purchase, 14.96% for cash-out refinancing. 83.56% of the applications were approved. 69.99% of the applications belonged to White applicants and 6.15% to Black or African American applicants (approval rate for Whites was 85.33% and Blacks was 71.51%). (see Figure 3 with a fragment of the conceptual model).

For comparison of concept contributions, data needs to be standardized. Categorical features are transformed by one-hot-encoding except output feature 𝑦. After data engineering, the dataset contains 52 features from which 20 are associated to the concept applicant, 14 to the concept loan and 18 to the concept 𝑙𝑜𝑎𝑛_𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛.

Feature Contribution

All five concepts are used for making a binary decision on loan applications, i.e. concept Decision with a feature 𝑎𝑐𝑡𝑖𝑜𝑛_𝑡𝑎𝑘𝑒𝑛. We use XGBoost Classification for predicting 𝑎𝑐𝑡𝑖𝑜𝑛_𝑡𝑎𝑘𝑒𝑛.

Performance metrics for the model on the test dataset are: accurary (0.995), precision (0.999), recall (0.995), F-measure (0.997).

For all three concepts, i.e., Applicant, Loan and LoanApplication, we determined feature contributions on the concept Decision modeled by a single feature (𝐴𝑐𝑡𝑖𝑜𝑛𝑇 𝑎𝑘𝑒𝑛) and three permutations, i.e.

Concept contributions

For each input concept (here: applicant, loan application, and loan), concept contributions are determined. As a result, 𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑛𝑡 provides a strong concept contribution 𝜅 𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑛𝑡 (𝑋) of 0.610 to LoanApplication while 𝑙𝑜𝑎𝑛𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛 provides a relatively high concept contribution 𝜅 𝑙𝑜𝑎𝑛𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛 (𝑋) of 3.177 to 𝑙𝑜𝑎𝑛 (cf. Table 2). We only used one feature per outcome concept for simplicity.

For the HDMA dataset, concept contributions indicate a strong directed connection from applicant to 𝑙𝑜𝑎𝑛_𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛 (cf. Figure 4). This supports the establishment of a relation between these two concepts. Same holds for 𝑙𝑜𝑎𝑛𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛 to 𝑙𝑜𝑎𝑛 and 𝑙𝑜𝑎𝑛 to 𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑛𝑡. The latter concept contribution is not captured by the original conceptual model (cf. Figure 3). This is an example how concept contributions 𝜅 𝑝 can be leveraged for automatically scrutinization of conceptual models associated with datasets and database implementations. In this example, we found support for a revised conceptual model with relations between loan towards applicant. It shall be noted that concept contributions do not provide link semantics, such as cardinalities or type of links. Proposals for revisions based ConceptSuperimposition help domain experts and ML developers and business analysts to align conceptual models with data and ML models.

Conclusion

This research proposes that ML models and conceptual models can be used effectively for analyzing prior conceptual knowledge used for data selection. A conceptual model represents agreed-upon domain knowledge. In this work we first provide a theoretical basis for using conceptual models in the domain of explainable AI. Within use the Mental Models Framework to develop a formalized and holistic ConceptSuperimposition method. The method is formalized and its utility is demonstrated in the application to the home mortgage domain.

The new ConceptSuperimposition method can be used to improve ML explainability and should be especially effective for situations where there is insufficient data to extract relevant domain knowledge in a data-driven manner. Future work is needed to apply the framework and method to other examples in other domains. In future studies we hope to evaluate the increased transparency due to the new method by conducting interviews, focus groups, and laboratory experiments with the stakeholders looking to understand the decision logic behind machine learning models. We also plan to investigate the benefit of this method together with other existing approaches to explainability. We do not position this method as an alternative, rather, we believe it could complement existing methods by extrapolating their outputs onto the conceptual models.

In future work, we will address the integration of concept contributions for classification and regression.