A recurring problem discussed in explainable AI is the so-called epistemic opacity problem, that is, a problem about the epistemic accessibility and reliability of algorithms. In the present work, we provide an original epistemological characterization of the opacity of algorithms based on a tripartite analysis of their components. Against this background, we introduce a formal framework by modifying the neighborhood semantics for evidence logic introduced in [1]. This setting allows one to reason about an agent's epistemic attitudes toward an algorithm and investigate what are the conditions that should be met to achieve epistemic transparency.
The explosion in the use of computational algorithms in several domains of human life prompted
the development of explainable AI and, more in general, of what can be labelled as a
humancentered approach to algorithms to help shed some light on the nature of AI models. From this
perspective, as Seaver puts it, “it is not the algorithm, narrowly defined, that has sociocultural
efects, but algorithmic systems - intricate, dynamic arrangements of people and code” ([
To characterize the epistemic opacity of algorithms, we follow the methodology proposed by
Durán and Formanek [
© 2022 Author:Pleasefillinthe\copyrightclause macro
This characterization of opacity crucially relies on the fact that an agent “X does not
know”, which, in turn, presupposes an account of what knowledge is. However,
unfortunately, Humphreys leaves this question open. Traditional epistemology takes knowledge as
corresponding to justified true belief. This analysis has been challenged by Gettier’s famous
counterexample (see [
One important feature of the previous definition is that the components I, P, and O of the
algorithm are, at least in principle, independent. As a consequence, the lack of epistemic
justification for any component constitutes a suficient condition for epistemic opacity. Let
us clarify the definition via an example of a specific facial recognition algorithm used in
AI. Convolutional neural networks (CNN, see [
Definition 2.1 can be considered as a tentative epistemological characterization of epistemic opacity. However, to reason formally about the epistemic attitudes of an agent toward opacity, one needs to recast that definition in logical terms. To do so, we will borrow some tools from the toolkit of epistemic logic and match each component of Definition 2.1 with an epistemic modality. Roughly, we consider that the fact that an agent has an epistemic justification for I can be logically represented by , that is, an agent “knows the input ” of the algorithm. The fact that an agent has an epistemic justification for P can be seen as □ , that is, the agent “has an evidence for the procedure ”. Finally, the fact that an agent has an epistemic justification for O can be considered as that is, the agent “believes in the output ”. Let us now present more carefully this formal framework.
The semantics we are proposing is a modification of a neighborhood semantics for evidence
logic provided by van Benthem et al. [
The language ℒ is defined as follows: ∶= | | | ¬ | ∧ | | □ | where ∈ , ∈ , ∈ , and in is defined on the domain , in □ is defined on the domain , in is defined on the domain .
The intended interpretation of is “the agent believes in the output ,” where stands for a proposition. The interpretation of □ is “the agent has evidence for procedure .” By “having an evidence for a procedure” we mean that an agent has an understanding of a particular way of producing an output based on a given input. The interpretation of is “the agent knows the data ,” where by “knowing the data” we mean that the agent is aware of the inputs of the algorithm. From this perspective, we are not dealing with propositional factive knowledge in this case. For instance, an agent may know the data used by the algorithm as an input even if these data are incorrect. In what follows, we indicate by the union of , , and , and we add a superscript on variables , , in order to mark them, respectively, as the output, the procedure, and the input of the same algorithm .
In order to interpret the language ℒ we use the evidence models introduced by van Benthem
et al. [
Definition 3.1. worlds, ⊆ × ( ) ( ) for the set { | } ∅ ∉ ( ) and ∈ ( )
.
An evidence model is a tuple ℳ = ⟨ , , ⟩ , where is a non-empty set of is an evidence relation, ∶ → ( ) is a valuation function. We write
. Two constraints are imposted on the evidence sets: For each ∈ , Definition 3.2. A -scenario is a maximal collection ⊆ ( ) that has the finite intersection property: for each finite subfamily { 1, ..., } ⊆ , ∩1≤≤ ≠ ∅.
Definition 3.3. Let ℳ = ⟨ , , ⟩ as follows:
be an evidence model. Truth of a formula ∈ ℒ is defined • ℳ, ⊧ if ∈ () ; • ℳ, ⊧ if ∈ () ; • ℳ, ⊧ if ∈ () ; • ℳ, ⊧ ¬ if ℳ, ⊧ ̸ ; • ℳ, ⊧ ∧ if ℳ, ⊧ and ℳ, ⊧ ; • ℳ, ⊧ □ if there exists such that and for all ∈ , ℳ, ⊧ ; • ℳ, ⊧ if for each -scenario and for all ∈ ∩ , ℳ, ⊧ ; • ℳ, ⊧ if for all ∈ , ℳ, ⊧ .
The satisfiability and validity are defined as usual.
The main peculiarity of our approach lies in distinguishing three types of domains for variables and limiting the application of each modal operator by its corresponding domain. Technically, we have defined the same function for all the three types of variables, and the evidence sets are defined so that they can contain any type of variables. However, conceptually, satisfiability of each type of variables in a world represents diferent situations. In particular, ℳ, ⊧ means that proposition is true in a world , which is standard. By ℳ, ⊧ we mean that the procedure belongs to the world . This, in turn, by definition of ¬, means that ℳ, ⊧ ¬ should be interpreted as the fact that the procedure does not belong to the world . Similarly, ℳ, ⊧ means that belongs to the world , and ℳ, ⊧ ¬ means that does not belong to the world . These considerations are useful for understanding the definitions of , □ , and . In particular, ℳ, ⊧ means that after considering all the evidences for and against , the truth of is consistent with these evidences. The condition for ℳ, ⊧ □ states that an agent has an evidence for a procedure if is present in some evidence set available in . Notice that it is possible that an agent has an evidence both for and ¬ . This is in line with our informal reading, because an agent can have evidences for a procedure being applicable in a certain context, but inapplicable in some other. Finally, ℳ, ⊧ means that the data is present in all worlds considered by the agent, i.e., the agent has full access to the data in all contexts.
On the basis of the previous definitions, we are able to define opaque algorithms semantically. We do not need to define an algorithm in our semantics, but we take it as a system containing all inputs, procedures, and outputs, associated with . Following Definition 2.1, an opaque algorithm ( ) is one for which the agent lacks a justification for at least of one of its components. We can now rephrase this definition in logical terms.
Definition 3.4 (Logical). An algorithm is epistemically opaque relative to an epistemic agent in a world ∈ ℳ if: ℳ, ⊧ if ℳ, ⊧ ¬ 1 ∨ ¬□ 2 ∨ ¬ 3.
Now we are able to provide semantic models for representing transparent and opaque algorithms.
Example 3.1 (Transparent algorithm). Let ℳ = ⟨ , , ⟩ such that = { , 1, 2, 3}, ( ) = {{ , 1, 2, 3}}, and ( ) = ( ) = ( ) = . Clearly, in this model we have ℳ, ⊧ ℳ, ⊧ □ , and ℳ, ⊧ . Thus, ℳ, ⊧ ¬ . , Example 3.2 (Opaque algorithm - 1). In this example we provide a model for an algorithm, the opaqueness of which is due to the lack of epistemic justification for the input, in presence of epistemic justifications for the procedure and the output. Let ℳ = ⟨ , , ⟩ such that = { , 1, 2, 3}, ( ) = {{ , 1, 2, 3}, { , 1, 2}}, and ( ) = { , 1, 3}, ( ) = { , 1, 2}, ( ) = { , 1, 2, 3}. In this model we have ℳ, ⊧ ¬ , because ℳ, 2 ⊧ ¬ ; and we have ℳ, ⊧ □ , ℳ, ⊧ . Thus, ℳ, ⊧ .
Example 3.3 (Opaque algorithm - 2). Now we model a situation in which an algorithm is opaque due to the lack of epistemic justification for the procedure, in presence of epistemic justification of the input and of the output. Let ℳ = ⟨ , , ⟩ such that = { , 1, 2, 3}, ( ) = {{ , 1, 2, 3}, { , 1, 2}}, and ( ) = { , 1, 2, 3}, ( ) = { 1}, ( ) = { , 1, 2, 3}. In this model, we have ℳ, ⊧ ¬ □ , because for all such that there exist ∈ such that ℳ⊧ ̸ ; and we have ℳ, ⊧ , ℳ, ⊧ . Thus, ℳ, ⊧ .
Example 3.4 (Opaque algorithm - 3). Here we consider a model, in which an algorithm is opaque because the agent lacks an epistemic justification for the output, in the presence of the justifications both for the input and the procedure. Let ℳ = ⟨ , , ⟩ such that = { , 1, 2, 3}, ( ) = {{ , 1, 2, 3}, { , 1, 2}}, and ( ) = { , 1, 2, 3}, ( ) = { , 1, 2}, ( ) = { , 1, 3}. We have ℳ, ⊧ ¬ , because for 2 ∈ ∩ , ℳ, 2⊧ ̸ ; and we have ℳ, ⊧ , ℳ, ⊧ □ . Thus, ℳ, ⊧ .
We provided an original epistemological definition of algorithmic opacity based on a tripartite analysis of algorithms. On the basis of this definition, we introduced a formal framework that allows one to analyze the epistemic attitudes of an agent towards a possibly opaque algorithm. The transition from the epistemological to the formal framework is made by respecting the tripartite structure of I, P, and O and by attributing to them the , □ , and modality, respectively. Let us call this analysis the IPO model of algorithmic opacity. In future work, we aim at deepening the IPO model, both from an epistemological and formal perspective. Regarding the former, in the literature on algorithmic opacity, it is often pointed out that one of the major dificulties in analyzing opacity is its multilayered nature. Some authors have proposed taxonomies for diferent forms of epistemic opacity. For instance, Burrell [8] distinguishes between intentional, illiterate, and intrinsic opacity. From this perspective, we intend to compare the epistemological definition we introduced with the forms of opacity analyzed in the literature, in order to understand whether our definition is general enough to encompass all possible forms of opacity. From a formal point of view, we have adapted the evidence models to provide a general semantic framework to reason about an agent’s epistemic attitudes toward an algorithm. The next natural step in this investigation is to introduce a logical system for reasoning about opacity and prove its completeness with respect to the evidence models. Moreover, recently, other approaches for dealing with the notion of evidence were proposed, for instance by Carnielli and Rodrigues [9] and Artemov [10]. Carnielli and Rodrigues [9] provide an interpretation of paraconsistent and paracomplete logics in terms of evidence. Even though this reading permits one to deal with possibly inconsistent evidence, which is also possible by using the semantics we adopted, the relation between evidence and justification is less straightforward in this logical framework. For this reason, we consider our approach more promising for the aims of the current work. Artemov [10] introduced a logic of justification by supplementing the standard modalities of epistemic and doxastic logic with explicit terms, which provide the reason for believing in a proposition. It seems that the IPO model can also be represented in this framework. We leave the task of adapting the logic of justification to the analysis of opaque algorithms for future investigations.
The authors acknowledge the support of the Project PRIN2020 BRIO - Bias, Risk and Opacity in AI (2020SSKZ7R) awarded by the Italian Ministry of University and Research (MUR). The research of Ekaterina Kubyshkina is funded under the “Foundations of Fair and Trustworthy AI” Project of the University of Milan. Mattia Petrolo gratefully acknowledges the support of the French National Research Agency (ANR) through the Project ANR-20-CE27-0004. M. Arbib (Ed.), The handbook of brain theory and neural networks, 2nd ed., The MIT press, 1998, pp. 255–258. [8] J. Burrell, How the machine ‘thinks’: Understanding opacity in machine learning algorithms, Big Data & Society 2 (2016) 1–12. [9] W. Carnielli, A. Rodrigues, An epistemic approach to paraconsistency: a logic of evidence and truth, Synthese 196 (2019) 3789–3813. [10] S. Artemov, The logic of justification, The Review of Symbolic Logic 1 (2008) 477–513.