Abstract argumentation frameworks to promote fairness and rationality in multi-experts multi-criteria decision making

Abstract

In this work, we propose to model Multi-Experts MultiCriteria Decision-Making (MEMCDM) problems using Abstract Argumentation Frameworks. We specifically design our model so as to emulate fairness and rationality in the decision-making process. For instance, when, of two expert's decisions, one is unfair, we impose an attack between these two decisions, forcing one of the two decisions out of the argumentation network's resulting extensions. Similarly, we specifically put irrational decisions in opposition to force one out. In doing so, we aim to enable the prediction of decisions that are themselves fair and rational. Our model is illustrated on a toy example.

Keywords

• Multi-Experts Multi-Criteria Decision Making
• Disagreement
• Fairness
• Rationality
• Argumentation Framework
• Model

-

Expert analysis and decisions arguably provide high-quality and highly-valued support for action and policy making in a wide variety of fields, from social services, to medicine, to engineering, to grant funding committees, and so on. However, the use of experts can be prohibitive due to either lack of availability, high cost, or limited time frame for action – this is the case particularly more so in impoverished areas. As such, it is desirable to be able to replicate / predict such decisions when beneficial even in the absence of experts. Unfortunately, there are many obstacles that still hinder an accurate simulation of expert decisions. First, it is hard to understand, and therefore replicate, the way each expert “aggregates” information/assessment along several criteria. In addition, even if we had a reasonable insight about it, any expert may make inconsistent decisions across similar scenarios. Finally, in the case of multiple experts, despite looking at the same information, two (or more) experts may disagree on the decisions to be made.

In spite of such challenges, traditional approaches seek to combine prior known decisions of experts into a classification of scenarios (machine learning approaches) or into some aggregation function that allows to best replicate the experts’ decisions. Unfortunately, this line of approaches tends to overlook the irrationality and/or lack of fairness of experts, aggregating all available prior information regardless of quality.

In this work, we propose to model Multi-Experts Multi-Criteria DecisionMaking (MEMCDM) problems using argumentation frameworks. We specifically design our proposed model so as to emulate fairness and rationality in decisions. For instance, when, of two expert’s decisions, one is unfair, we impose an attack between these two decisions, forcing one of the two decisions out of the argumentation network’s resulting extensions. Similarly, we specifically put irrational decisions in opposition to force one out. In doing so, we aim to enable the prediction of decisions that are themselves fair and rational. Our model is illustrated on two toy examples.

In what follows, we start by recalling preliminary notions, then we proceed with describing our model in details and illustrate our model in the case of Software Quality Assessment by multiple experts along multiple criteria. 2 2.1

Preliminary Notions

Multi-Criteria Decision Making (MCDM)

Multi-criteria decision-making (MCDM) involves selecting one of several different alternatives, based on a set of criteria that describe the alternatives. However, there are numerous problems that make comparing these alternatives difficult. For instance, very often, decisions are based on several conflicting criteria; e.g., which car to buy that is cheap and energy efficient. In addition, what happens when we have a group of decision makers that must come to some sort of consensus? This is known as multi-expert multi-criteria decision making (MEMCDM). In MEMCDM, there are several new problems to be addressed. One such problem is how to handle expert disagreement and come to a consensus/decision in the first place. Another problem, as stated earlier, is that of predicting future decisions based on decision data from multiple experts along multiple criteria. Again, the question of “which expert/decision-making process to follow?” is a major challenge in solving such problems.

Approaches to MCDM In general, on a daily basis, when the decision is not critical, in order to reach a decision, we mentally “average / sort” these criteria along with their satisfaction levels. This corresponds to aggregating values of satisfaction with weights on each criterion, reflecting its importance in the overall score (a.k.a. additive aggregation), that is, calculating the overall score of an alternative with the weighted sum of the criterion scores. In other words, weights assigned to different sets of criteria in the weighted-average approach form an “additive measure”. Additive aggregation, however, assumes that criteria are independent, which is seldom the case [ 2 ]. Non-linear approaches also prove to lead to solutions that are not completely relevant [ 6 ].

This should change when considering possible dependence between criteria. For example, if two criteria are strongly dependent, it means that both criteria express, in effect, the same attribute. As a result, when we consider the set consisting of these two criteria, we should assign to this set the same weight as to each of these criteria – and not double the weight as in the weighted sum approach. In general, the weight associated to different sets should be different from the sum of the weights associated to individual criteria. In mathematics, such non-additive functions assigning numbers to sets are known as non-additive (fuzzy) measures. It is therefore reasonable to describe the dependence between different criteria by using an appropriate non-additive (fuzzy) measure. Combining the fuzzy measure values with the criteria satisfaction can be done using the Choquet integral, which integrals are actively used in Multi-Criteria Decision Making [ 5 ].

However, to make this happen, fuzzy measures need to be determined: they can either be identified by a decision maker/expert or by an automated system that extracts them from sample data. Since human expertise might not always be available and getting accurate fuzzy values (even from an expert) might be tedious [ 9 ], fuzzy measures are usually automatically extracted from prior decision decision data. To the original problem, this approach adds an optimization problem that can be tedious to solve. Although it was solved with success for some data sets [ 10 ], the overall prediction quality is not satisfactory and the approach limits the number of criteria that can be taken into account (the number of variables to determine is exponential in the number of criteria) [ 8 ]. 2.2

Argumentation Frameworks

In this section we briefly summarise the background information related to classical AAFs [ 4 ]. We focus on the basic definition of an AAF (see Def. 1), on the notion of defence (Def. 2), and on extension-based semantics (Def. 3). Definition 1 (Abstract Argumentation Frameworks). An Abstract Argumentation Framework (AAF) is a pair F = hA, Ri of a set A of arguments and a binary relation R ⊆ A × A, called the attack relation. ∀a, b ∈ A, aR b (or, a b) means that a attacks b. An AAF may be represented by a directed graph (an interaction graph) whose nodes are arguments and edges represent the attack relation. A set of arguments S ⊆ A attacks an argument a, i.e., S a, if a is attacked by an argument of S, i.e., ∃b ∈ S.b a. a b c d

e

Definition 2 (Defence). Given an AAF, F = hA, Ri, an argument a ∈ A is defended (in F ) by a set S ⊆ A if for each b ∈ A, such that b a, also S b holds. Moreover, for S ⊆ A, we denote by SR+ the set S ∪ {b | S b}.

The “acceptability” of an argument [ 4 ] depends on its membership to some sets, called extensions : such extensions need to satisfy the properties required by a given semantics, and they characterise a collective “acceptability”. In the following, stb, adm, prf , gde, com, and sem, respectively stand for stable, admissible, preferred, grounded, complete, and semi-stable semantics. The intuition behind these semantics is outside the scope of this work (e.g., see [7, Ch. 3]). Definition 3 (Semantics). Let F = hA, Ri be an AAF. A set S ⊆ A is conflict-free (in F), denoted S ∈ cf (F ), iff there are no a, b ∈ S, such that (a, b), (b, a) ∈ R. For S ∈ cf (F ), it holds that: – S ∈ stb(F ), if foreach a ∈ A\S, S a, i.e., SR+ = A; – S ∈ adm(F ), if each a ∈ S is defended by S; – S ∈ prf (F ), if S ∈ adm(F ) and there is no T ∈ adm(F ) with S ⊂ T ; – S = gde(F ) if S ∈ com(F ) and there is no T ∈ com(F ) with T ⊂ S; – S ∈ com(F ), if S ∈ adm(F ) and for each a ∈ A defended by S, a ∈ S holds; – S ∈ sem(F ), if S ∈ adm(F ) and there is no T ∈ adm(F ) with S+ R ⊂ TR+.

We recall that for each AF, stb(F ) ⊆ sem(F ) ⊆ prf (F ) ⊆ com(F ) ⊆ adm(F ) holds, and that for each of the considered semantics σ (except stable) σ(F ) 6= ∅ always holds. Moreover, in case an AF has at least one stable extension, its stable, and semi-stable extensions coincide. Finally, gde(F ) is always unique, and gde(F ) ∈ com(F ).

Consider the F = hA, Ri in Fig. 1, with A = {a, b, c, d, e} and R = {(a, b), (c, b), (c, d), (d, c), (d, e), (e, e)}. We have that stb(F ) = sem(F ) = {{a, d}}, and gde(F ) = a . The admissible sets of F are ∅, {a}, {c}, {d}, {a, c}, {a, d}, and prf (F ) = { } {{a, c}, {a, d}}. The complete extensions are {a}, {a, c}, {a, d}.

In the proposed model (precisely in Sec. 3.2) we take advantage of symmetric AAFs [ 3 ]: Definition 4 (Symmetric AAFs [ 3 ]). A symmetric (Abstract) Argumentation Framework is a finite Argumentation Framework F = hA, Ri where R is assumed symmetric, non empty and irriflexive.

This leads to some properties related to the computed semantics: for instance, ∀S ∈ prf (F ) then S ∈ stb(F ), cf (F ) = adm(F ), and, since each argument in our model is attacked, gde(F ) = ∅ always holds [ 3 ]. Note also that in symmetric AAFs, the computation of the sceptical/credulous state (see Def. 5) of an argument becomes easier [ 3 ] (e.g., P instead of NP ). Definition 5 (Acceptance state). Given a semantics σ and a framework F , an argument a is i) sceptically accepted iff ∀S ∈ σ(F ), a ∈ S , and ii) credulously accepted if ∃S ∈ σ(F ), a ∈ S.

Since Argumentation-based decision-making checks the justification state of arguments in order to rank decisions (see a brief summary in the following paragraph), such decision process can benefit from this simplification derived from using symmetric AAFs in our model.

Decision-making with Arguments In this section we simplify part of the content in [7, Ch. 15]. Solving a decision problem amounts to defining a preordering, usually a complete one, on a set D = {d1, . . . , dn} of n candidate options. Argumentation can be a means for ordering this set D, that is to define a preference relation < on D. An argumentation-based decision process can be decomposed into the following steps: 1. Constructing arguments in favour/against statements (beliefs or decisions). 2. Evaluating the strength of each argument. 3. Determining the different conflicts among arguments. 4. Evaluating the acceptability of arguments. 5. Comparing decisions on the basis of relevant accepted arguments.

We need to characterise the subsets of practical arguments that are respectively in favour (Ff ), or against (Fc) a given option in di ∈ D: – Ff : D → 2A is a function that returns the arguments in favour of a candidate decision. Such arguments are said pros the option. – Fc : D → 2A is a function that returns the arguments against a candidate decision. Such arguments are said cons the option.

In Def. 6 we present one of the possible ways to prefer (<) one decision instead of another. This unipolar principle only refers to either the arguments pros or cons.

Definition 6 (Counting arguments pros/cons). Let DS = (D, F ) be a decision system, where F is an AAF, and Accstb (F ) collects the sceptically accepted arguments of a framework F under the stable semantics. Let d1, d2 ∈ D.

d1 < d2 ⇐⇒ |Ff (d1) ∩ Accstb (F )| ≥ |Ff (d2) ∩ Accstb (F )|

The aim of (part of) future work (see also Sec. 5) is to apply similar techniques to derive the best decision about our model, e.g., an evaluation about the software. 3

Proposed Model for MEMCDM using Argumentation Frameworks Here, we describe our model: given an MEMCDM problem with n criteria and p experts, how do we “translate”/model it as an AAF? In other words, which arguments and attacks should compose it? Note that, through this section we will use letters S and R to identify “Software”, “Ranking” (unlikely to Sec. 2.2, where these letters represent a subset of arguments and the attack relation respectively). 3.1

Arguments

• What does the data we use (i.e., experts’ evaluation of software in this case) tell us about the arguments to add to the network? We differentiate arguments that come from the data (i.e., Expert i said that Software j is good) from arguments that are implicit (i.e., Software k is Poor). 1. Expert i gives Item j a total quality Dij (which, in the case of Software Quality Assessment – SQA, can be Bad, Poor, Fair, Good, or Excellent):

Argument (Ei, Sj , Dij )

Let us call such arguments, arguments of type ESD.

2. Expert i judges that Item j satisfies criterion m up to quality Dijm

Argument (Ei, Sj , cm, Dijm)

Let us call such arguments, arguments of type EScD.

• Which implicit arguments should be part of the argumentation network for this specific type of problem? 1. For each item, independently from what experts say, there will be a decision made. This decision will be in the form of a final ranking, ranging over all possibly ranking values (in the case of SQA: Bad, Poor, Fair, Good, Excellent). So regardless of ESD arguments, we add to the argumentation network the following arguments:

∀ item Si, ∀ ranking Dj : Argument (Si, Dj )

Let us call such arguments, arguments of type SD. 2. For each criterion of evaluation, regardless of which item is being evaluated and of what experts will decide, a ranking will be associated. So regardless of EScD arguments, we add to the argumentation network the following arguments: ∀ criterion ck, ∀ ranking Dm : Argument (ck, Dm) Let us call such arguments, arguments of type cD. Such arguments are expected to be useful for prediction of the decision of experts on items not part of the original data, but for which we do have an indication of their quality per criterion. • Coalitions of Arguments Here we aim to model the fact the n decisions of any expert on the n criteria of the problem at hand belong together: they together form the “support” for the expert’s final decision on the given item. As a result, for any expert Ei and any item Sj , we define a coalition of “supporting” decisions as:

∀Ei, ∀Sj , Coalition: {(Ei, Sj , ck, Di,j,k), k ∈ {1, . . . , n}} Let us call such coalitions of EScDs, extended arguments of type CoEScD. The result of modeling such coalitions is that all arguments in the coalition will be forced to be altogether either in or out of extensions. Per se, we are enforcing an equality constraint on the belonging of these arguments to any extension.

Note that here we do not use the term “support” as in classical Bipolar AAFs [ 1 ] (or BAAFs ), which exploit the notion of a support binary-relation among arguments. There, the support relation is totally independent of the attack one. 3.2

Attacks

In this subsection, we answer the following question: What are the attacks (edges of the network ) between these arguments (nodes)? Note: All attacks we define are reciprocal, hence the edges are always set bidirectionally. For attacks too, we differentiate between attacks that come from inconsistencies in the decision data (disagreement between experts, inconsistency in decisions of a single expert, lack of fairness, irrationality). An assumption that we make in designing the network model is that experts should be rational: in this, we mean that even if they are not (which we know), they should be and we aim to elicit decisions that are as rational as can be. • Attacks derived from lack of fairness Here, we assume that if an expert is fair, then s/he should derive the same final ranking from the same criteria rankings. For instance, if there are 3 criteria (c1, c2, and c3) to assess items and an expert E has the following decision history: and: (with Si 6= Sj )  E, Si, c1, D1 

E, Si, c2, D1  E, Si, c3, D1 −→

E, Si, D  E, Sj, c1, D1 

E, Sj, c2, D1  E, Sj, c3, D1 −→ E, Sj, D0 where D 6= D0, then we should see arguments (E, Si, D) and (E, Sj, D0) are a lack of fairness in judgment and therefore add the following attack in the argumentation network: (E, Si, D) ←→ (E, Sj, D0).

More generally, assuming that the criteria that are considered by the experts are ck, with k ∈ K, and that the possible rankings are denoted by Dr, with r ∈ R, then we add the following rule to our model: ∀E, Si, Sj, s.t. i 6= j and ∀k ∈ K, ∃r ∈ R, (E, Si, ck, Dr) and (E, Sj, ck, Dr) : if (E, Si, Di) and (E, Sj, Dj) and Di 6= Dj then Attack (E, Si, Di) ←→ (E, Sj, Dj) • Attacks derived from lack of rationality Let us recall that we assume that the rankings Dr, with r ∈ R, are totally ordered. However, with n criteria, the set of n-tuples of rankings is only partially ordered: (D1, D2, . . . , Dn) ≺ (D10, D20, . . . , Dn0) iff : ∀i ∈ {1, . . . , n} : (Di 6= Di0) −→ Di < D0 i Now: ∀Ei and ∀Sj, we denote by (D1,i,j, . . . , Dn,i,j) the set of n decisions made by Expert Ei on each of the criteria c1, . . . , cn for Item Sj, and by Di,j the final decision of Expert Ei on Item Sj.

Being rational for any given expert Ei means that if for Item Sj, s/he ranks criteria lower (w.r.t. above partial order) than s/he ranks the criteria of Item Sk, then his/her final ranking of Sj should not be higher than his/her ranking of Sk. Formally, it is expressed as follows: ∀Ei, ∀Sj, ∀Sk(j 6= k) : if: (D1,i,j, . . . , Dn,i,j) ≺ (D1,i,k, . . . , Dn,i,k) and: Di,j > Di,k then: Attack (Sj, Ei, Di,j) ←→ (Sk, Ei, Di,k) • Attack related to implicit arguments: SD and cD In this subsection, we describe the following attacks: – attacks between implicit arguments SD (resp. cD), and – attacks across SD and ESD (resp. cD and EScD). 1. Attacks among SDs: SD Arguments associate an item with a ranking. For each item Si, there are p SD arguments if there are p possible ranking levels. Each of these p arguments attack each other (they form a complete subgraph). In other words:

∀Si, ∀r1, r2 ∈ R, with r1 6= r2, Attack: (Si, Dr1 ) ←→ (Si, Dr2 ) 2. Attacks among cDs: In a fashion similar to attacks among SDs, we have: ∀cj , ∀r1, r2 ∈ R, with r1 6= r2, Attack: (cj , Dr1 ) ←→ (cj , Dr2 ) 3. Attacks between SDs and ESDs: For any given item Sj , an argument saying that Si is evaluated Dh is in contradiction (and therefore attacks – and vice-versa) with any argument (Ei, Sj , Dk) as soon as Dh 6= Dk. As a result: ∀Ei, ∀Sj , (Dh 6= Dk) →

Attack: (Sj , Dh) ←→ (Ei, Sj , Dk) 4. Attacks between cDs and EScDs: Similarly as above, for any given criterion cm, an argument saying that cm is evaluated Dh is in contradiction (and therefore attacks – and vice-versa) with any argument (Ei, Sj , cm, Dk) as soon as Dh 6= Dk. As a result: ∀Ei, ∀Sj , ∀cm, (Dh 6= Dk) →

Attack: (cm, Dh) ←→ (E,Sj , cm, Dk) • Attacks between Coalitions and ESDs Here we aim to model the fact that coalitions of decisions on criteria support experts’ decisions. In other words: ∀Ei, ∀Sj , {(Ei, Sj , ck, Di,j,k), k ∈ {1, . . . , n}} supports (Ei, Sj , Di,j )

In terms of attacks, this is expressed as follows:

∀Ei, Ej ∀Sk : Di,k 6= Dj,k →

Attack: {(Ei, Sk, cl, Di,k,l), k ∈ {1, . . . , n}} ←→ (Ej , Sk, Dj,k) 4

An Example

Here, let us look at a scenario in which experts independently assess given pieces of software, based on several given evaluation criteria. We describe the resulting argumentation networks (arguments/nodes and attack/edges). Table 1 summarises our example, by reporting all the Poor/Fair/Good quality-evaluation about two different criteria (1 and 2) and the overall quality related to three different software products (S1/S2/S3). Such scores are produced by three different experts (E1/E2/E3). For instance, E1 estimates that the overall quality of S1 is fair, with Criterion 1 evaluated as poor, and Criterion 2 as good.

The graph in Fig. 2 represents the AAF given by following the model proposed in Sec. 3 on the data in Tab. 1. The yellow nodes represent explicit arguments from the data. The green nodes are the implicit arguments. The blue nodes are the coalitions. The black bold lines represent attacks due to lack of fairness and lack of rationality. The dotted line attacks are those based on implicit arguments. Finally, the grey bold lines are coalition supports of expert decisions. In this work, we proposed a model for MEMCDM problems, based on classical AAFs [ 4 ], that allows to emulate fairness and rationality. This allows discrimination among input decision data (from experts’ prior decisions) between data of value and data that should just not be taken into account. Next steps include operationalising the whole process (from input processing to results filtering) and then adding weights to the attacks to simulate the extent of disagreements and allow lineance towards small errors (e.g., unfairness / irrationality that are really minimal, minor disagreements). Furthermore, we will take inspiration from classical decision-making techniques [7, Ch. 15] with the purpose to rank decisions and decide, for instance, if a software is good or poor. We will even develop new techniques exploring weights on attacks. Also part of future work, we plan to explicitly acknowledge in the AAF that disagreement can be at two different levels: epistemic and pragmatic, and to make use of argumentation frameworks to identify disagreement configurations (epistemic and pragmatic, epistemic only, pragmatic only).

Acknowledgments. S. Bistarelli was partially supported by MIUR-PRIN “Metodi logici per il trattamento dell’informazione”. M. Ceberio’s work was partially supported by the National Science Foundation, NSF CCF grant 0953339 and the American Association for the Advancement of Science, AAAS MIRC (agreement date 112612). F. Santini was partially supported by MIUR PRIN “Security Horizons”.

References

1. L. Amgoud , C. Cayrol , and M.-C. Lagasquie-Schiex. On the bipolarity in argumentation frameworks . In J. P. Delgrande and T. Schaub, editors, NMR , pages 1 - 9 , 2004 .

2. M. Ceberio and F. Modave . An interval-valued, 2-additive Choquet integral for multi-criteria decision making . In Proceedings of the 10th Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU'04) , Perugia, Italy, July 2004 .

3. S. Coste-Marquis , C. Devred , and P. Marquis . Symmetric argumentation frameworks . In L. Godo, editor, ECSQARU , volume 3571 of LNCS , pages 317 - 328 . Springer, 2005 .

4. P. M. Dung . On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games . Artif . Intell., 77 ( 2 ): 321 - 357 , 1995 .

5. M. Grabisch and C. Labreuche . A decade of application of the choquet and sugeno integrals in multi-criteria decision aid . 4OR , 6 ( 1 ): 1 - 44 , 2008 .

6. F. Modave , M. Ceberio , and V. Kreinovich . Choquet integrals and OWA criteria as a natural (and optimal) next step after linear aggregation: A new general justification . In Proceedings of MICAI'2008 , pages 741 - 753 , 2008 .

7. I. Rahwan and G. R. Simari . Argumentation in Artificial Intelligence . Springer Publishing Company, Incorporated, 1st edition , 2009 .

8. X. Wang , M. Ceberio , S. Virani , A. Garcia , and J. Cummins . A hybrid algorithm to extract fuzzy measures for software quality assessment . Journal of Uncertain Systems , 7 ( 3 ): 219 - 237 , 2013 .

9. X. Wang , A. F. G. Contreras , M. Ceberio , C. D. Hoyo , L. C. Gutierrez , and S. Virani . Interval-based algorithms to extract fuzzy measures for software quality assessment . In Proceedings of Annual Conference of North American Fuzzy Information Processing Society (NAFIPS' 2012 ), Berkeley, CA, August 2012 .

10. X. Wang , J. Cummins , and M. Ceberio . The Bees algorithm to extract fuzzy measures for sample data . In Proceedings of Annual Conference of North American Fuzzy Information Processing Society (NAFIPS' 2011 ), El Paso, TX, March 2011 .