Inferences through knowledge driven approaches have been researched extensively in the eld of Arti cial Intelligence. Among such approaches argumentation theory has recently shown appealing properties for inference under uncertainty and con icting evidence. Nonetheless, there is a lack of studies which examine its inferential capacity over other quantitative theories of reasoning under uncertainty with realworld knowledge-bases. This study is focused on a comparison between argumentation theory and non-monotonic fuzzy reasoning when applied to modeling the construct of human mental workload (MWL). Di erent argument-based and non-monotonic fuzzy reasoning models, aimed at inferring the MWL imposed by a selection of learning tasks, in a third-level context, have been designed. These models are built upon knowledgebases that contain uncertain and con icting evidence provided by human experts. An analysis of the convergent and face validity of such models has been performed. Results suggest a superior inferential capacity of argument-based models over fuzzy reasoning-based models.
Uncertainty is inevitable in many real-world domains. Several theories in the
eld of Arti cial Intelligence (AI) have been studied for dealing with
quantitative reasoning under uncertainty, such as Probability calculus and its variations:
Possibility Theory and Imprecise Probabilities, Dempster-Shafer Theory,
Argumentation Theory and Multi-valued Logics like Fuzzy Logics. More speci cally,
Fuzzy Reasoning [
The rest of the paper continues with section 2 presenting related work on fuzzy reasoning, AT and with a short description of the construct of MWL. The design of a comparative experiment and methodologies for the development of argument-based and fuzzy-reasoning based models are detailed in section 3. Section 4 introduces the results followed by a discussion. Section 5 concludes the research by highlighting its contribution and proposing future work. 2
Reasoning and explanation under incomplete and uncertain knowledge have been
investigated for several decades in AI. On one hand, classical propositional logic
has demonstrated to be inadequate, due to its monotonicity property, for dealing
with real-world argumentative activities often involving inconsistent and con
icting information [
Fuzzy reasoning, as proposed by [
Classical argumentation, from its roots within philosophy and psychology, deals
with the study of how arguments or assertions are de ned, discussed and solved
in case of divergent opinions. In AI, argumentation refers to that body of
literature that focuses on techniques for constructing computational models of
arguments. Such models have become increasingly important for
operationalising non-monotonic reasoning [
Mental workload (MWL) can be intuitively described as the amount of cognitive
activity exerted to accomplish a speci c task under a nite period of time [
Translation of knowledge-base into interactive defeasible arguments
Elicitation of knowledge-base and resolution of inconsistencies 3) evaluation of con icts
Final inference
4) dialectical status of arguments
5) accrual of
acceptable arguments
few decades [
face validity { it determines the extent to which a measure of MWL appears
e ective in terms of its stated aims (measuring mental workload);
convergent validity { it refers to the extent to which di erent MWL measures
that should be theoretically related, are in fact related [
A primary research study was designed and it included a comparison between the inferential capacity of AT and non-monotonic fuzzy reasoning considering three knowledge-bases produced within the MWL domain. It is demonstrated how these knowledge-bases, built by an expert upon the features extracted from the original NASA-TLX mental workload assessment technique (as per section 2.3), can be translated into defeasible argument-based models and into non-monotonic fuzzy reasoning models. Three main parts compose the non-monotonic fuzzy reasoning models: a fuzzi cation module, an inference engine and a defuzzi cation module. Argument-based models are de ned as in gure 1 (section 2.2). A comparison of the inference produced by fuzzy reasoning and AT was made in terms of their di erences in convergent and face validity. Employed data was composed by the answers of the NASA-TLX questionnaire from 213 students who performed di erent learning tasks in third-level classes. The overall design of the research is summarised in gure 2. Due to space constraints, the full knowledgebases produced by the expert with the aid of the authors can be seen online1. 1 https://doi.org/10.6084/m9.figshare.6979865.v1 Knowledge-base 1
Knowledge-base 2 Non-monotonic fuzzy
reasoning models 1. Fuzzi cation 2. Inference engine 3. Defuzzi cation
Dataset
Knowledge-base 3
Construction of reasoning models Argument-based models 1. Structure of arguments 2. Con icts of arguments 3. Evaluation of con icts 4. Dialectical status 5. Accrual of arguments
Fuzzy Argument-based models inferences models inferences
Validity comparison
Fuzzi cation module The knowledge-bases of the interviewed expert can be represented by rules of the form \IF ... THEN ...". The antecedent (before THEN) is a set of premises associated to a number of workload attributes, while the consequent (after THEN) is associated to a possible MWL level. Examples: - Rule 1: IF low mental demand THEN underload - Rule 2: IF low effort THEN tting load
Each MWL level (consequent of a rule) was described by a number of FMFs
in di erent ways ( gure 3). According to the domain expert's knowledge two
options were designed: from [0, 100] having 4 membership functions associated to
it and from [
- Exception 1: high effort refutes Rule 1
Exceptions can be tackled by Possibility Theory, as implemented in [
It is important to highlight that the theory developed in [
on the construction of forecast arguments as it follows:
F orecast argument : premises ! conclusion This structure is composed by a set of premises built upon the features considered in the NASA-TLX mental workload assessment instrument and a conclusion (MWL level) derivable by applying an inference rule !. The categories 2 https://doi.org/10.6084/m9.figshare.6979865.v1 associated to these conclusions are the same as the ones described in section 3.1. However, since no notion of gradualism is considered here, they are strictly bounded in well de ned ranges (example, low mental demand in one knowledgebase is de ned in the range [0; 33) 2 R). An example of a forecast argument is given below (it matches Rule 1 of section 3.1):
ARG 1: low mental demand ! underload
inconsistencies and invalid arguments, mitigating arguments [
M itigating argument : premises ) B
Both forecast and mitigating arguments follow a similar notion of defeasible rules,
as de ned in [
UA1: high effort ) ARG 1 All the designed arguments and attacks can now be seen as an argumentation framework (AF), as depicted in gure 6.
can be elicited with data. Forecast and mitigating arguments can be activated
or discarded, based on whether their premises evaluate true or false. Attacks
between activated arguments are considered valid, while the others are discarded.
Contrarily to fuzzy systems, there is no partial refutation, so a successful attack
always refutes its target. From the activated forecast/mitigating arguments and
valid attacks, a sub-argumentation framework emerges (sub-AF), as in gure 7
(this is equivalent to the Abstract Argumentation proposed by Dung [
Y such that Y attacks X; - complete i X is admissible and X contains all arguments it defends, where
X defends x if and only if X attacks all attackers of x; - grounded i X is minimally complete (with respect to ); - preferred i X is maximally admissible (with respect to )
the reasoning process, a nal inference has to be produced for practical purposes.
In case multiple extensions are computed, one extension might be preferred over
the others. In this study, the cardinality of an extension (number of accepted
arguments) is used as a mechanism for the quanti cation of its credibility.
Intuitively, a larger con ict-free extension of arguments might be seen as more
credible than smaller extensions. In case some of the computed extensions have the
same highest cardinality, these are all brought forward in the reasoning process.
After the selection of the larger extension/s, a single scalar is produced through
the accrual of its/their arguments. This is de ned by the set of accepted forecast
arguments within an extension (those that support a MWL level). Mitigating
arguments already had their role by contributing to the resolution of con icting
information (layer 4) and thus are not considered in this layer. For each
forecast argument, a nal scalar is generated for its representation. This scalar is
essentially a linear relationship from the range of the argument's premise to the
range of the argument's conclusion. For instance, if argument ARG 1 is
activated by the lowest value of the mental demand range, then its nal scalar will
be the correspondent lowest value in its conclusion's range. The overall MWL
level brought forward by an extension is computed by aggregating the scalars of
its forecast arguments. This aggregation can be done in di erent ways, for
instance considering measures of central tendency. Here, the average is considered.
Table 3 summarises the design of the argument-based models.
A number of third-level classes have been delivered to students at Dublin
Institute of Technology. After each class, students had to ll in the questionnaire
associated to the NASA-TLX instrument (as described in section 2.3). Students
were from 23 di erent countries (age 21-74, mean 30.9, std= 7.67). Four di
erent topics of the module `research methods' were delivered in di erent semesters
during the academic terms 2015-2017, as per table 4. Three di erent delivery
methods were used: 1) traditional direct instruction, using slides projected to a
white board; 2) multimedia video of content (same as in 1) projected to a white
board; 3) constructivist collaborative activity added to 2. Summary statistics
can be found in table 4. Beside completing the NASA-TLX questionnaire,
participants were required to ll in another scale providing an indication of their
experienced mental workload ( gure 8). This was designed as a baseline and as
a form of ground truth. It is believe that only the person executing the task can
provide a precise account of the mental workload experienced [
In details, in order to evaluate the inferential capacity of the models (built in tables 2 and 3) two forms of the validity of their inferences (scalar values) were adopted. As suggested in section 2.3, these are convergent validity and face validity. The former has been assessed through an analysis of the correlation of the inferences, produced by designed models, and the scores produced by the Nasa-Task Load Index. The latter has been assessed through an investigation of the error of designed models against the mental workload scores reported by students, using the scale of gure 8. Table 5 summarises these two forms of validity and the statistical test associated to them. The answers of the NASA-TLX questionnaire were used to elicit the designed non-monotonic fuzzy reasoning and argument-based models (tables 2, 3).
Convergent validity Figure 9 depicts the Spearman correlation coe cients of the inferences of the designed models and the NASA-TLX indexes. This statis3 MSE = n1 Pin=1 Yi
2
Xi , where Y is the vector of inferences made by designed models and X the vector of self-reported values. tical test was used because the assumptions behind the Pearson correlation were not met. Moderate to high correlation (coe cients: 0.50-0.76) were generally observed. This indicates that, the assumption of the theoretical relationship between the NASA-TLX measure, known to fairly model the construct of mental workload, and the designed models in fact exists. As a consequence, it can be said that the non-monotonic fuzzy reasoning models and the argument-based models are fairly modelling mental workload in the designed experiment, regardless of the operators used to aggregate premises of fuzzy rules (Zadeh, Product, Lukasiewicz) or the semantics used in layer 4 (grounded/preferred).
Face validity Figure 10 depicts the mean squared errors of the inferences (scalar values) of each designed model. As it is easy to observe, argument-based models had a lower error when compared to the baseline instrument (NASATLX). The average of MSEs associated to fuzzy reasoning models (F1-18) was 407:75 while the average of MSEs of argument-based models (A1-6) was 229:83. Additionally, among the fuzzy reasoning models, the di erence in the scores of those employing the centroid as defuzzi cation method (labelled with an upper dot) appear to be better than the ones employing the mean of max. As for argument-based models, there is only a slight di erence across knowledge-bases and no signi cant di erence among semantics (grounded/preferred). It is important to highlight that knowledge-base KB3 is certainly richer than KB1 and KB2, as it carries more interacting pieces of knowledge ( gure 6). Despite this higher richness, the mean squared error did not decrease signi cantly when compared to KB1 and KB2 both for the fuzzy reasoning or argument-based models. However, when comparing the mean squared error of the fuzzy reasoning models against the ones of the argument-based models, it can be stated that the latter have a better inferential capacity over the former for the speci c tasks and dataset employed. This statement holds regardless of the fuzzy operators employed in the fuzzy engines (fuzzy reasoning models); the semantics adopted in the con ict resolution layer (argument-based models); and the knowledge-bases considered. Argumentation, overall, had a better face validity and was superior in approximating the target (self-reported mental workload reported by students) than non-monotonic fuzzy reasoning. An analysis of the convergent validity of the models showed that their inferences can be considered valid. They are positively and moderately correlated to the well known (NASA-TLX), thus likely modelling mental workload too. A negative or null correlation would have implied the invalidity of the models since they would have probably modelled another construct. With a good convergent validity, the ndings from the analysis of the face validity can be considered more reliable. This analysis indicated a better inferential capacity of the argument-based models over the fuzzy reasoning models for the selected tasks and data, despite the internal con guration and underlying knowledge-base employed. Argument-based models consistently showed a signi cantly lower mean squared error (di erence between self-reported MWL and the inferences by designed models) over fuzzy reasoning models, in addition to a slight improvement also against the baseline instrument (NASATLX). This demonstrates the potential of argumentation as a modelling tool for knowledge-bases characterised by uncertainty, partiality and con ictual info. 5
This study presented a comparison between non-monotonic fuzzy reasoning and
non-nonotonic (defeasible) argumentation using three di erent knowledge-bases
coded from an expert in the domain of mental workload. A primary research has
been conducted including the construction of computational models using these
two non-monotonic reasoning approaches to represent the construct of mental
workload and to allow its assessment (inference). Such models were elicited with
data provided by the NASA-Task Load Index questionnaire that was lled in by
students who performed a set of learning tasks in a third-level context. The
output of these models was a single scalar representing a level of mental workload
that was used for comparison purposes. The selected metrics for evaluation of
the inferential capacity of constructed models were convergent and face validity.
Findings indicated how both the models built with the non-monotonic fuzzy
reasoning mechanism and defeasible argumentation had a good convergent validity
with the NASA-TLX, con rming mental workload was actually the construct
being modelled. However, the argument-based models had a signi cantly
better face validity over the non-monotonic fuzzy reasoning models for the selected
tasks and data. The novelty of this research lies in the quanti cation of the
impact of argumentation through a novel empirical research in a real-world context
employing primary data gathered from humans. Future work will concentrate on
replicating this experiment by considering additional knowledge-bases and by
extending the comparison of argumentation with other reasoning approaches such
as expert systems. Moreover, the creation of inference models adopting fuzzy
reasoning and argumentation such as in [
Lucas Middeldorf Rizzo would like to thank CNPq (Conselho Nacional de Desenvolvimento Cient co e Tecnologico) for his Science Without Borders scholarship, proc n. 232822/2014-0.