1 The text reports findings of a case study based on the investigation of reminding activities and request practices in the specific context of supported living. These activities turn out to be highly adaptive processes that are embedded in complex assistive networks. The process of reminding and requesting represents a central practice deployed by the assistive institutional and social environment. It suggests to provide a consistent structure that meets individual needs in everyday life of cognitively impaired people. In the light of the development and engineering of assistive technologies we discuss if and how human practices could serve as a basis for modeling an Embodied Conversational Agent (ECA) based assistive system for cognitively impaired people with respect to the adherence of their autonomy.
1
People with cognitive impairments as well as elderly people
require special assistance in managing their daily routines like
household activities or managing the everyday structures when
living independently. Cognitive or physical challenges often affect
or lead to a decrease of the quality of life. Hence, maintaining an
autonomous life in a familiar social environment and home for as
long as possible has become a central issue in today’s societies [
Research on technical assistive systems strives to suggest
solutions for this social challenge, e.g., in the realm of Ambient
Assisted Living and Social Robotics. To this end, multimodal
dialogue systems represented by Embodied Conversational Agents
seem particularly suited, as they can be easily integrated in private
homes using modern TV sets, allowing for intuitive
humanmachine interfaces, using means of natural communication when
entering and managing appointments and being reminded of
individual tasks or events [
The question of autonomy arises when considering the
integration of an assistive technology to support independent living
and in the setting of supported living with distributed actions.
Results of ethnographic research in an institution of supported
living for people with cognitive impairments, i.e. people with
special needs in independent living, presented in this study, reveal
practices of reminding and requesting as essential to preserve
wellstructured everyday routines. Besides the moment of acute
reminders, the complex process of reminding and requesting
practices that precedes the actual reminder is relevant to form an
2
2.1
understandable request-reminder and its accomplishment. These
processes are closely interwoven and coordinated with an assistive
social and institutional network. Set against this background the
integration of an assistive technology into already existing assistive
networks carries a strong ethical issue with respect to the
preservation of the individual autonomy [
This study shows how ethnographic research serves as a valid
approach to user centered design respecting the Human Value
Approach [
Based on sociological analyses of human activities and technology,
Rammert speaks of “distributed action[s]” [5: 18] and “distributed
agency” [5: 5] and describes them as a multiplicity of actions which
are distributed over temporal and factual dimensions. In this
context technical engineers, have to consider how system
influences human relations, hierarchies, competences and the
division of work. Winner stresses that “The things we call
‘technologies’ are ways of building order in our world.” [6: 127]
and so, they shape society, individuals and their actions. Thus, the
design of technical systems always reflects implicit or explicit
values and can never be neutral. While the approach of Value
Sensitive Design suggests to integrate the needs of human users
and values [
In the design process of technical systems ethical issues have to
be considered not only from the individual perspective but also
from an institutional and social viewpoint. The model for ethical
evaluation of socio-technical arrangements (MEESTAR) [
In the area of assistive living, especially when focusing on assistance in the field of time management and support of temporal orientation by an ECA-based assistant, the dimension of autonomy plays an essential role. Technical artifacts that remind, or request users to perform a task or to keep an appointment, and afterwards check the accomplishment of a task, raise the question of agency and autonomy on the one hand, but also contribute to various levels of individual security and participation. 2.2
Human Computer Interaction (HCI) studies on time management
support and calendars show how reminders can be designed as
requests and argue to design them in a multimodal way to be
effective, usable and accessible for a diverse user group [
Though those HCI studies refer to context they do not show the
dependencies and fine-grained coordinative practices of an
assistive network [
Requests as a subject of research have a long tradition within
linguistics and there are several attempts to describe and define
requests [
In sum, linguistics, CA and HRI (Human Robot Interaction)
research widely defines requests as represented by singular verbal
utterances even though, there are hints at the influence of
contextual, interactional and sequential circumstances for the
production a singular utterance. Besides, especially research in care
settings has primarily focused on requests made by the
carereceiving party in face-to-face interaction. Our aim is to expand
this perspective by describing requests in a broader sense that takes
not only the sequential structure of interactions into account, but
also the social and institutional perspective. To provide valid
statements for the implications for an ECA-based assistant [
3
3.1
The research is based on focused ethnography [
The inspection of individual (outreach work) and institutional settings (attendance service) revealed that there is a differentiation between required support levels depending on independence and autonomy of people with special needs. There are three merging levels of support: care, assistance and integration (Fig. 1). Clients with special needs in the care area are supported exclusively within inpatient care and intensive social and physical support. Clients with special needs in the area of assistance live either in in- or outpatient care with individually adjusted support depending on the area of support. On the support level of integration the clients with special needs are living in outpatient care mostly at their own homes and work in so-called sheltered workshops.
To get a comprehensive overview of what assistive practices
actually look like, how they are communicated and coordinated
within the assistive social and institutional network of the client,
the ethnographic study took place in different areas and settings
within the institution. As the integration aid is based on two forms
of assistance, we first focused on the central office of the
attendance service as the “center of coordination” [
Following the principles of focused ethnography [
The fine-grained analysis of video data is based on CA and
provides access to the understanding of micro-sequential processes
in interactions [
The ethnographic study reveals the activity of reminding within
assisted time management in the context of an assistive living
institution as a process of reminding. It is framed and coordinated
by a client's assistive network [
We identified different steps that evolve as a process of reminding after the initial appointment registration. The core process consists of two essential practices applied by the assistive social and institutional network: successive reminders that have an instructive character and acute reminders that function as requests.
coordination of an assistive social and institutional network time Successive reminders appear to have a twofold function for the clients with special need in temporal orientation: in the long term they provide reliability regarding planning activities and support temporal orientation on the one hand and on the other hand help to anticipate acute reminders. Acute reminders have a request character due to their temporal proximity to appointments. When comparing this with the findings regarding requests in care settings we see a contrast in the performance of a request and the form not an isolated utterance, but embedded in a request context. The concept model of reminding as a process finishes with the actual appointment or optional post-processing.
The process of reminding relies on highly complex and adaptive assistance networks, involving official institutional staff as well as an informal social environment involving family, friends and colleagues. This highly personalized flexible support network is being formed to respect and support the participants’ competences and capabilities. 5
The following case study focusses on the process of successive reminding during an assistance meeting and is temporally located after the appointment registration and before the acute reminder (see chapter 6). The analyzed segment is a record of an assistance meeting where a support worker (S) and a client (C) discuss upcoming and past issues at C's home. C has no temporal orientation and therefore depends on explicit and recurrent reminders and requests. S's successive reminding strategies are produced in different formats and temporal stages during the assistance meeting with C (Fig. 3 I-IV).
(a) Announcement: first appointment reminder: After discussing recent events S starts the first announcement on reminding C of an upcoming appointment for an assistance plan meeting in three days on a Thursday at one o’clock. The last assistance plan meeting was cancelled and the appointment has now been rescheduled. This appointment involves not only C and S but also C's legal representative. As there is an institutional network engaged there is the need for coordination. Another rescheduling or cancelling of the appointment due to a possible non-appearance of C would imply additional organizational expenditure for the assistive network. So, C's punctual appearance has an increased significance in this context. A successive reminding process is central during assistance meetings and an essential key to assure a punctual appearance to appointments.
With his question (Fig. 3 I 01-03) S is reassuring and checking that C is already aware of the upcoming appointment for the assistance plan meeting. C confirms with yes. After the positive confirmation of C, S names the time. By using a conjunction and a temporal adverb he marks the time and this time at one o’clock (05) as deviating from the norm. After a short sequence in which C explains why she couldn’t make it to the appointment last time S formulates a second appointment reminder. time : g n i t e e m e c n a t s i s s (b) Second appointment reminder + instruction: In the second stage (Fig. 3 II 06-10) S formulates another question to reassure that C will keep the appointment (II 06-07) and adds an additional instruction by providing practical guidance so C can manage to get to the appointment on time. By advising her to go to the appointment right after lunch break,- (II 09-10) he uses a time category that is manageable for C and provides an understandable reference point in time. Due to C's difficulties with temporal orientation the provided temporal link or ‘landmark’ is an assistive verbal strategy that bridges C's difficulties with estimating durations. After a short discourse S initiates, the two final steps in the reminding process.
(c) Third appointment reminder: In the final steps of the
successive reminding process (Fig. 3 III, IV) S is not asking an
explicit question like in step I and II but is making a statement
which is marked by a dropping pitch at the end of the sentence (III
11). However, after C confirms the statement with mhmh, S
transforms his statement into a question by adding the sentence
final question particle right, with a rising pitch.
(IV) Referring to a further reminder: In the ensuing sequence S
gives a prospect of further steps in the reminding process. He
names the exact day on which he will come over for the next
assistance meeting (IV 16-17). After that, he inserts a parenthesis
to explain why he came in today exceptionally and that he is
covering for another support worker who called in sick. C responds
by producing the back channeling signal mhmh, and therefore
signals sustained attention to the interaction [
The analysis has shown how a successive process of reminding unfolds at different points in the interaction and how precisely and recurrently the upcoming appointment is referred to. The described strategies of successive reminding establish a basis for an upcoming acute reminder on the one hand and they provide planning certainty and reliability for C on the other hand. 6
The following analysis shows how an actual reminder is produced as a process in its complexity of modalities and presuppositions in human-human-interaction. The extract was recorded after a regular communal breakfast organized by the operator of the external-carebased assisted living. C2 is accompanied by her friend (F) who is part of her informal support network. As mentioned in chapter 5, C has no temporal orientation, whereas F is temporally oriented and keeps plans and appointments in mind. The acute reminder emerges from the need to take the next bus. F's reminder strategies illustrate an interplay of attention getting and subtle reminder upgrade strategy.
F
C
F
C 01 F_gaz: |@watch|~ | 02 Silence: |(1.0) |(0.2)| ! 03 F_gaz: |@C------------------------------| 04 F_ver: |the *BUS comes *in a moment;| 05 F_han: *LH: C-shoulder*
Figure 4. Multimodal disp!lay of the reminder activity (a) Attention getting and embodied anticipation of a new activity: C is involved in a group interaction while F has put his jacket on and then, joins the group. This preparatory action of putting the jacket on serves as a change of context and as a visual cue for C. Besides the completion of breakfast, it initiates reminder activities in a subtle way without a manifest display of urgency. F initiates a first stage of reminder activities, i.e. attention getting while C is involved in interaction: F stands behind C and taps on C's back with both hands. This tapping could be interpreted as a subtle form of attention getting which is found in subsequent steps of reminder activities, too. However, its first occurrence is characterized by absence of verbal activity. The first steps of the acute reminder process serve as attention getting devices and do not contain explicit requests or a display of urgency.
(b) First explicit naming of appointment: Explicit multimodal
forms of a reminder are displayed not until F has got C's attention
that becomes manifest through C's gaze [
(c) Subtle reminder upgrade: F retries the multimodal request procedure in the subsequent interaction another three times after monitoring C's reactions. The retries occur with rising frequency and appear as a subtle increase in urgency. The reminder procedure shows a fine-grained coordination of modalities: The retries start with F's observation of C's attention (head orientation, gaze) while she is involved in a conversational task. When C's head movement becomes observable, F anticipates C's orientation towards him. C's change of orientation is followed by F's utterance of the request and a tactile underlining (see section (b)) while F directs his gaze at C directly. It is noteworthy that F embeds the requests precisely in the ongoing interaction and respects C's conversational tasks: he does not interrupt C's utterances, but uses multimodal options for turn taking such as pauses, changes of C's bodily orientation and gaze to secure her attention. So, though he works on the task of reminding, he is also involved in the overall interaction. The subtle upgrade as well as the precisely coordinated placement of reminders ensures C's autonomy and role as a competent participant within the overall interaction. (d) Negotiating and relativizing the reminder-request: After a total of four reminder retries, F interrupts the interaction for a fifth reminder by varying the attention getting device: he skips the attention-securing via gaze and uses the tactile modality to get C's attention and repeating the verbal request (Fig. 5). C makes this upgraded reminder-request conversationally relevant by turning to F and relativizing the reminder-request with the utterance that there is no urgency in taking exactly this bus (Fig. 5: i must- (.) i MUST not_this-). It becomes clear that the reminder-request is perceived and understood by C, but that she still is involved in a conversational task (of ensuring to meet R (researcher) in the following week). After R's reassurement, C and F leave. The negotiation of the reminder underlines C's involvement in the interaction, the solving of a conversational task first, and so, the autonomous prioritization of tasks in interaction and her autonomy in changing an action plan due to contingencies in social interaction. Even though the task of taking the bus seems clear, other tasks are more important and the initial action plan has to be adjusted to contingencies in social and interactional activities.
F and C's reminder system appears to be an evolving process which is adaptive and flexible enough to be embedded in complex social interactions as well as to react to changing circumstances. The analysis shows that it is well-practiced within contingent social interactions to jointly handle complex tasks.
The study has revealed how reminder practices are produced and integrated in the everyday lives of people with special needs and coordinated within their assistive networks in a German institution for supported living:
(a) Personal Successive Reminders: The case study in section 5 shows how a joint planning process of the supportive network and the client emerges. The conversational practices applied by the support worker (e.g. explicit instructions and references to future reminder steps) provide security, planning certainty and reliability for the client who needs support in planning and temporal orientation. Joint planning is the basis for a meaningful and transparent establishment of upcoming reminders and provide individual information about the context of appointments.
(b) Acute Reminders: Section 6 shows how appointments are contextualized and how the participants' implicit knowledge about consequences and meaningful activities work when a reminder occurs. The analysis shows the evolving micro-process and complex interplay of getting attention / securing contact and applying a subtle reminder upgrade strategy. The reminder process is highly adaptive and flexible and allows to react to changing circumstances within social situations based on close observation (or monitoring) practices.
When applying the supportive network's tasks and practices to the development of a technical system, the empirical data and concept model of the process of reminding give hints for implications for system design but also raise issues for a discussion of assistive technologies in the light of ethics.
(c) Verbal practices and timing: Adaptive procedures characterize human planning and reminding processes and activities of acute reminders. Following this model, an ECA needs technical and verbal structures to produce recurring successive reminders that lead to acute reminders and effective requesting strategies. The exact timing of these strategies bears not only a technical challenge, but also regarding the design of actual formulation and wording, i.e. interaction conceptualization to ensure that requests or interruptions by the ECA are not being perceived as unexpected or impolite.
(d) Multimodal monitoring: Continuous and extensive
multimodal monitoring-processes need to be implemented as a
precondition for the implementation of accurately applied verbal
strategies. These monitoring processes should encompass the
monitoring of gaze and head orientation as well as body orientation
(e.g. via Eye tracker). Besides these requirements, the system needs
a structure to classify the different states of the participant in the
process of reminding after an appointment has been registered (Fig.
2) to produce meaningful reminders that are timed and
synchronized with the classified state. These strategies need to be
adapted to needs and competences of each participant [
(e) Ethical considerations regarding assistive technologies:
Assistive technologies that are developed neglecting complex
social and institutional structures probably end up at being an
isolated solution for solitary tasks and so, are questionable in their
use and effects. It should be discussed what technology is able to
provide and how technical assistance could be integrated in the
assistive networks meeting the individual needs of each user [
By applying the MEESTAR evaluation dimensions we have to ask what autonomy means within the human assistive setting in the light of distributed action and agency. In the context of supported living, clients already are involved in different forms of distributed action and agency in a human network. Which role and task can then the ECA undertake when discussing autonomy and requests (as reminders)? The question of legitimization of an agent making requests is a fundamental ethical issue that has to be discussed in the context of autonomy: We have to conceptualize, define and uncover the role and boundaries of the technical system as either a representative of the support worker or as the enhancement of the client. These conceptualizations and definitions have consequences on the declaration of consent and the use of collected data.
Another ethical issue arises from the matter of system access. In the current system, the ECA is solely able to register appointments and perform acute reminders. It has to be reflected what happens in-between, i.e. should the tasks of support workers be implemented into the system and if yes, how? Or should the perspective be twisted to better integrate the technical system into the assistive network. It is also necessary to discuss the issue of the system’s transparency. Facing users that have no expertise in designing assistive systems, it has to be asked, if the human assistive network is allowed to enter tasks or appointments into the technical system, to what extent the origin of these entries has to be made transparent for the participants. One additional implication that emerges from this perspective comprises an explicit marking of the appointment origin on the interface.
(f) Research on interaction in settings with people with
special needs: Research on interaction with assistive technologies
for time management and organizational tasks widely focuses on
the ageing population, while the group of people with special needs
in independent living is not well documented so far. Our paper
follows this direction and hints at the special competences of
clients, the challenges and tasks of support workers, as well as the
complex social structures including formal and informal assistive
networks. As integration means to enable participation [
This research was supported by the German Federal Ministry of Education and Research (BMBF) in the project ‚KOMPASS’. We would like to thank the participants of the ethnographic research for their time and support.