This paper is aimed at formalizing the interplay among a person to be assisted, an assistive agent-based software, and a caregiver. We propose principles that agents should follow in such interplay, this principles may have impact in di erent agent-based assistive technology. We propose a mechanism to integrate individual's information into the nal decision-making process of an agent. Moreover, we endow agents with mechanisms for evaluating the distance between independent and supported activity execution, the so called zone of proximal development (ZPD) in four scenarios: I) independent activity execution by a person; II) ZP DH activity performance of a person supported by another person (e.g. a therapist); III) the ZP DS representing a potential activities when a person is supported by a software agent; and IV) the ZP DH+S when a person is supported by a caregiver and a software agent. Formal argumentation theory is used as foundation. Our formal models were tested using a prototype using augmented reality as assistive software. A pilot study with older adults and health-care personnel was performed and formal and empirical results are presented.
This paper is aimed at investigating assistive technology using argumentationbased agents and the interplay with individuals that require physical assistance and their caregivers.
We present formal and empirical results on how intelligent software adapts
to support activities of individuals including, those who need assistance and care
givers. The focus of the paper is on the provision of human-like characteristics
to software agents in order to provide adaptable support, namely common-sense
and re ection on action decision. The proposed agent model is oriented to reason
about human activities, i.e., identify and interpret activities, and support
individuals during the execution of physical activities. To this end, representations
of complex activities from Activity Theory [
Finally, as core of our research, we propose a model of adaptation of a support
level for agents, based on a computation version of the so-called zone of proximal
development (ZPD) [
The research questions (RQ) addressed in this paper are the following: { RQ1: how an agent may infer potential activities that an individual needs and performs with and without its assistance? { RQ2: in a smart environment scenarios, where individuals require support from others to execute an activity, how an agent-based software may adapt autonomously to team-up with humans to enhance such support? { RQ3: how rational agents can \re ect" on decision to make when a human is in the loop?
This paper is structured as follows: Section 2 presents de nitions about how human activities are structured, we also introduce some de nitions about formal argumentation theory and argument-based reasoning. In Section 3 our main contributions are presented trying to solve RQ1-3. Using a medical scenario, we developed a prototype to test empirically the level of assistance, some results are presented in Section 4. Conclusions and a discussion is presented in Section 5. 2
In this paper, the human perspective is investigated using Activity Theory [
In this paper, activity theory is used with two purposes: 1) for structuring the knowledge of an agent following a hierarchical model; and 2) to understand the potential level of activity achievement of a person.
Activity as structure. An activity consists of a set of actions. At the lowest level, an action consists of a set of operations. Actions are oriented to goals and are executed by the actor at a conscious level, in contrast with operations which 3 Hereinafter we will identify a rational software agent as just an agent. do not have a goal of their own and which are executed at the lowest level as automated, unconscious processes. An activity model (AT) (see De nition 1) corresponds to information of a person that an agent uses to reason about an activity.
ior rules of an activity. LP denotes the set of atoms which appear in a program P . An AT model is a tuple of the form hAx; Go; Opi in which: { Ax = fax1; : : : ; axj g(j > 0) is a set of atoms such that Ax
the set of actions in an AT model. { Go = fg1; : : : ; gkg(k > 0) is a set of atoms such that Go
the set of goals in an AT model. { Op = fo1; : : : ; olg(l > 0) is a set of atoms such that Op the set of goals in an AT model.
LP . Ax denotes LP . Go denotes LP . Op denotes
In arti cial intelligence literature, this hierarchical structure has been used
as framework to represent knowledge of software agents, e.g. in [
An activity framework corresponds to the goals, observations and actions of
an agent oriented to assist a human during the execution of an activity, which
in turn is represented by the AT model. To capture the knowledge of an agent
about its environment, we use extended logic programs [
ActF according to De nition 2 de nes the space of knowledge of assistive agents without considering external assistance, for example from other assistive agents (human or software) actions. In this knowledge space, an argument-based process (see Figure 1) can be performed to obtain sets (or sets of sets) of explainable structures support-conclusion for what is the best assistive action to take.
sure the level of development not through the level of current performance, but
through the di erence (\the distance") between two performance indicators: 1)
an indicator of independent problem solving, and 2) an indicator of problem
solving in a situation in which the individual is provided with support from other
people [
ICF quali ers. The notion of quali er to specify the extent of the
functioning or disability of an individual was introduced by the International
Classi cation of Functioning, Disability and Health (ICF)4 [
Argumentation-based systems, have become in uential in arti cial intelligence
particularly in multi-agent systems design (see [
Knowledge base
Activity fragments construction
Activity fragments STEP 1
Formal argumentation process aCnoanlyflsicist Afcorcfaegapcmttaievbniittlysity
Argument framework STEP 2
Extensions STEP 3
Justified conclusions
Argument-based
Conclusions STEP 4
Reasoning about human activities may be performed using a bottom-up manner, building fragments of an activity explaining the current situation of a person. This process compresses the STEP 1 in Figure 1.
De nition 3 (Hypothetical fragments). Let ActF = hP; HA; G; O; ATi be an activity framework. A hypothetical fragment of an activity is of the form HF = hS; O0 ; h; gi such that: 1) S P; O0 0 O; h 2 HA; g 2 G; 2) S [O0 [fhg0 is consistent; 3) g 2 T such that ASP (S [ O [ fhg) = hT; F i; and 4) S and O are minimal w.r.t. set inclusion. ASP (S) is a function that returns an answer-set solution of an ELP program, i.e., it provides a common-sense reasoning process given a program as input.
In short, an hypothetical fragment is a consistent manner to explain (interconnected) parts of an activity. Some of these fragments may be contradictory given inconsistent information in the AT model or/and defeasible information captured by an agent (STEP 2 in Figure 1). 4 http://www.who.int/classi cations/icf/en/
Let ActF = hP; HA; G; O; Actsi be an activity framework. Let HF1 = hS1; O10; a1; g1i, HF2 = hS2; O20; a2; g2i be two fragments such that HF1; HF2 2 HF . ASP (Supp(HF1)) = hT1; F1i and ASP (Supp(HF2)) = hT2; F2i denote the semantic evaluation of the support, then HF1 attacks HF2 if one of the following conditions hold: 1) 2 T1 and : 2 T2:; 2) 2 T1 and 2 F2:
An argumentation framework is a pair hArgs; atti in which Args is a nite
set of arguments and att Args Args. In [
ity framework of the form hP; HA; G; O; Actsi; let HF be the set of fragments w.r.t. ActF and AttHF or simply Att the set of all the attacks among HF .
AAF = hActF; HF ; Atti
Dung in his seminal work [
De nition 6. Let AAF = hActF; HF ; Atti be an activity argumentation framework AAF with respect to ActF = hP; HA; G; O; Actsi An admissible set of fragments S HF is stable extension if and only if S attacks each argument which does not belong to S. preferred extension if and only if S is a maximal (w.r.t. inclusion) admissible set of AAF. complete extension if and only if each argument, which is acceptable with respect to S, belongs to S. grounded extension if and only if it is a minimal (w.r.t. inclusion) complete extension. ideal extension if and only if it is contained in every preferred set of AAF.
The sets of arguments suggested by an argumentation semantics are called extensions. Let SEM () be a function returning a set of extensions, given an AF such as an AAF. We denote SEM (AAF ) = fExt1; : : : ; Extkg as the set of k extensions generated by an argumentation semantics w.r.t. an activity argumentation framework AAF .
De nition 7. 1) An fragment HFA 2 HF is acceptable w.r.t. a set S of fragments i for each fragment HFB 2 HF : if HFB attacks HFA, then HFB is attacked by S. 2) con ict-free set of fragments S in an activity is admissible i each fragment in S is acceptable w.r.t. S.
Using these notions of fragment admissibility, di erent argumentation semantics can draw given an activity argumentation framework (STEP 3 Figure 1):
De nition 8. Let AAF = hActF; HF ; Atti be an activity argumentation framework following De nition 5. An admissible set of fragments S HF is: 1) stable if and only if S attacks each fragment which does not belong to S; 2) preferred if and only if S is a maximal ( w.r.t. inclusion) admissible set of AAF ; 3) complete if and only if each fragment, which is acceptable with respect to S, belongs to S; and 4) the grounded extension of AAF if and only if S is the minimal ( w.r.t. inclusion) complete extension of AAF .
Conclusions of an argument-based reasoning about an activity (see STEP 4 in Figure 1) may be obtained using a skeptical perspective, i.e., accepting only irrefutable conclusions as follows:
AFP = hArgP ; At(ArgP )i be the resulting argumentation framework from P and SEMArg be an argumentation semantics. If SEMArg(AFP ) = fE1; : : : ; Eng(n 1), then Concs(Ei) = fConc(A) j A 2 Eig(1 i n): Output = Ti=1:::n Concs(Ei):
generated by SEMArg(AFP ) are denoted as E 3
Re ection on decisions about human activity In this section, we report formal results to understand how autonomous agents may change the level of assistance in di erent scenarios. Two main formal results are presented in this section: 1) a mechanism for agent's decision-making based on the individual's information analyzing consequences of hypothetical actions a mechanism that we called re ection; and 2) a formalism to determine the potential of activity performance in four di erent cases: independence, supported by another person, supported by a software agent and supported by a team person-agent. 3.1
Conclusions of an argument-based process (De nition 9) about an activity, may contain sets of goal-based conclusions sets, indicating that the agent has di erent available decision alternatives which are consistent. We propose adding a mechanism for selecting an appropriate decision but considering those agent's actions that maximize humans' goals (Go) in an activity model (AT model De nition 1). An AT model captures all the information necessary to de ne a human activity. We condensed this process in Algorithm 1.
In short, Algorithm 1 takes as input the AT model and the set of extensions from a previous common-sense reasoning output. In lines 8-15 of Algorithm 1 a quali er is calculated (line 12) over sets of sets of fragments (the so-called extensions in argumentation theory, see Appendix 2). This quali er calculation is based on computing a similarity function between the current achievement of human goals in AT (OGo) w.r.t. a set of goal reference (RefGo line 12). The Algorithm 1: Goal-based action re ection input output 1 H ; 2 Go ; 3 Ref ; 4 numExt = jEj 5 numArg = jAj 6 0 7 decisionLat < ; h >= // list of agent's decisions
// list of human's goals // list of human's reference goals
// number of extensions // number of arguments per extension // numeric value of a qualifier (0 4) // lattice of decisions 8 for i 9 for j 10 h 11 12 13 0 to numExt do 0 to numArg do
Act (hfj) O Obs (hfj)
Q(OGo; RefGo) // Qualifier function considering
observations and a reference value w.r.t. person goals Go
decisionLat ( ; h)hfj // decision tuple is qualifier and an
agent's decision w.r.t the current fragment
14 end
15 end
16 return max( ; h)
Q function depicted in line 12, follows the quali er idea presented in previous
approaches [
The importance of Algorithm 1 lies on the mechanism for associating a human activity quanti cation with the internal action decision of an agent. Proposition 1 and Proposition 2 present two special cases of agent's behavior when Algorithm 1 is used5. One is the possibility to have a conclusion with no action, and the second expresses an inconclusive behavior given that stable semantics may return ; as output.
using a skeptic semantics, grounded or ideal, may result in a conclusive empty
decision.6
Proposition 2. An agent calculating a goal-based action re ection Algorithm 1
using the credulous semantics: stable, may result in a inconclusive decision.7
5 We refrain of describing fully the proofs of these propositions due the lack of paper
space
6 Proof sketch: output of grounded and ideal may include f;g. See [
In this section, based on the common-sense reasoning of activities using argumentation theory, we propose a theory to calculate the following four scenarios in assistive agent-based technology:
agent which takes a decision which is purposefully do nothing to support a person, or the decision is empty. More formally, the type of fragments (De nition 3) generated by the agents are with the form HF = hS; O0 ; h ; gi such that h 2 HA = f;; do N othingg. In this setting, all the extensions generated by SEM (AFP ) = E during a period of time will create an activity structure. In other words, the cumulative e ect of generating fragments, re-construct an activity in a bottom-up manner. Moreover, Algorithm 1 returns only values of , i.e. the current value of a quali er when the agent does not take any support action. This context de nes the baseline of activity execution independence of a person.
II. ZP DH : activities supported by another person Similarly to previous scenario, the role of the software agent is being an observer. However, built fragments have the form HF = hS; O ; h ; gi such that h 2 HA = f;; do N othingg and O = O0 [ O00 , where O is the set of joint observations from the agent's perspective about the individual supported (O0 ) and the supporter O00 . We have that O0 O00 , and O0 ; O00 6= ;. In this scenario, O00 is considered a reference set of observations (Ref lines 3 and 12 in Algorithm 1). Algorithm 1 will return a value of which measures in what extent an individual follows the guide provided by another person.
When multiple extensions are collected during the period of time that the individual is supported, then a di erent set of activities than individual activity execution may be re-generated in a bottom-up manner.
agent takes a decision oriented to uphold human interests. This is a straightforward scenario where h 2 HA 6= f;; do N othingg.
IV. ZP DH+S : human-agent supporting In this scenario, the main challenge for the agent perspective is detect: 1) actions that an assistant person executes, and 2) observations of both, the person assisted and the person who attends. This is similar to ZP DH but with fragments built from HA 6= f;; do N othingg. In this case, the level of ZP DH+S is given by Algorithm 1, and the set of extensions E with aligned goals between agent and human assistant.
Proposition 3. Let OGo be a set of observations about human goals (Go) and actions (Ax) framed on an activity, captured by an agent using an activity model AT. Let G and HA be agent's goals and its hypothetical actions. In order to provide non-con icting assistance two properties have to be ful lled: { PROP1: OGo \ G 6= ; { PROP2: OAx \ HA 6= ;
PROP1 and PROP2 provides coherence among human-agents actions and goals. This two properties may de ne a rst attempt to establish consistency principles of agent-based assistance. This is a future work in our research. 4
In this section, we present results of an empirical evaluation of di erent values of ZPD. The pilot test was illustrative, aimed at exploring ZPD values in a real setting, then make a qualitatively comparison with our formal approach. 4.1
The scenario selected to test our approach was framed on supporting an older adult in the activity: medication management using a smart medicines cabinet. In a smart environment8 developed at the user, interaction and knowledge management research group9, we setup the smart cabinet 10 (see Figure 2). smart medicines cabinet
projector Kinect sensor 2
II Text Recognition
Google API Kinect sensor 1 Kinect sensor 3
Gesture Recognition
Argument-based
Reasoning I
III Local machine
Goal-based action
Reflection IV Medicine database V (doses)
Augmented
reality projection Fig. 2. Smart medicines cabinet using argument-based reasoning and an augmented reality projection. I) Gesture recognition using three Kinect cameras, one for client body capture, another for assistant personal gesture recognition, last one (Kinect sensor 2) on the top of the cabinet to recognize text from medicines boxes; II) Google API for text recognition; III) common-sense reasoning; IV) goal-based action re ection to consider human side; V) database containing doses and timing of pill intake. 8 360 degrees view of the lab: https://goo.gl/maps/rq3YiF1c5An 9 Computing Science department Umea University- Sweden 10 Due the lack of space, we brie y describe the smart cabinet prototype which is connected to our agent-based platform
Architecture summary: Our prototype consists of ve main parts: 1)
gestures recognition: obtaining observations from individuals using Kinect cameras;
2) text recognition using another Kinect camera with Google API text
recognition (https://cloud.google.com/vision); 3) argument-based reasoning: the
main agent-based mechanism of common sense reasoning; 4) goal-based action
re ection generating an augmented reality feedback: a module to generate
support indications as projections in the smart environment; and 5) a database of
medicine doses to obtain appropriate messages11. We use three 3D cameras to
capture: 1) observations of an individual that needs help in a physical
activity; 2) observations of the smart environment, including a supporting person;
and 3) information of the handle gestures of medicine manipulation. A central
computer was connected to the cameras, processing the information in real-time
analyzing gestures of individuals as observations for the agent. The agent
platform (JaCaMo) was used to build the agent. An argumentation process was used
using an argumentation library previously developed (see [
Pilot evaluation setting summary: This pilot study recruited ve
participants, see Table X. All of the participants had technology experience using
computers . The procedure comprised three stages: 1) baseline interview
(subjects: TA-1, TA-2, TA-3 + S-1, S-2); 2) interview with a nurse (S-2); and 3)
prototype evaluation (subjects: TA-1, TA-2 + S-2). For a lack of space we only
describe the third stage in which participants interact the smart platform. In
the third stage, TA-1 participated the evaluation in his home and the other
two participates evaluated the system. They were asked to read the
instruction message from the augmented reality projection and then, distribute three
medications with di erent prescriptions by using the system. The Assessment
of Autonomy in Internet-Mediated Activity protocol (AAIMA) [
Formal argumentation can be seen as a process consisting of the following 11 Sources and documentation of the prototype can be found in https://github.com/ esteban-g
Our main contribution in this paper is in general, a formal understanding of
the interplay among an assistive agent-based software, a person to be assisted and
a caregiver. Moreover, as far as we know, this is a rst attempt to formalize the
behavior of rational agents using formal argumentation theory, in four scenarios:
I) independent activity execution, which resembles the so-called zone of current
development ZCD [
We propose an algorithm to integrate individual's information (the AT model
De nition 1) into the nal decision-making process of an agent. This mechanism
captured in Algorithm 1, resembles a process of \re ection" which in humans is
a re-consideration of actions and goals given some other parameters. In fact, our
re ection mechanism maybe seen as an action- ltering process with the
humanin-the-loop12. We also analyze di erent outputs of Algorithm 1 considering two
groups of argumentation semantics (Propositions 1 and 2).
12 A concept to integrate human information in cyber-physical systems [
We evaluate our approach in a three stages pilot study using a scenario of medication management as a complex activity. In this regard, we conducted an experiment with older adults and practitioners to evaluate such activity. We developed a prototype platform using augmented reality projecting assistive messages about medication when a person required some support. For lack of space, we did not fully report in this paper, the full functioning of the platform neither the process of co-design and expert feedback.