In this paper we present our work towards the development of an application for personalized reminiscence therapy in people with dementia. The reminiscence process aims at recalling personal memories by combining user-speci c knowledge, dialogue-based human-robot interaction and multimedia content. The application is part of a robotic software framework for companion robots, investigated in the EU MARIO project and under evaluation in di erent dementia care settings.
our approach, with a knowledge base (KB) that stores user-speci c information and supports dialogue-based interaction tasks, from basic language generation to language understanding. Ontologies enable a formal de nition and conceptualisation of domain knowledge. With respect to conventional database models, ontology models and their instantiation are independent of speci c data modeling strategies or implementations, fostering reusability and interoperability across heterogeneous applications. In addition, reasoning techniques can be used to perform inference on both the schema and the data, so as to entail implicit information and derive new information from existing knowledge. 2
As discussed in [
The reminiscence application, whose reference architectural model is shown in
Fig. 1, aims at supporting so-called simple reminiscence [
The de nition of life events relies on the time-indexed situation ontology
design pattern6 and includes event participants, the location where the event
took place, and a temporal dimension, to represent events that occurred in a
speci c date (e.g., a marriage) or over a period of time (e.g., attendance to
college). Speci c life events and their properties are explicitly modeled as frames,
to cover typical domains including work and education (e.g., school attendance
and working experiences), personal and family events (such as a marriage and the
birth of a child), and living and travel experiences. A frame provides a schema
for conceptualising the description of an event type and its participants in terms
of frame elements or semantic roles [
The association between media objects and other entities relies on a semantic tagging approach, as de ned in a tagging ontology module designed so that any object (including frames or even named graphs) can be used to categorise or describe the entity being tagged. This allows to de ne, for example, life events and persons as tags for an image, in addition to simple properties expressing where and when a photo was taken.
Authoring Tool. A Web-based Authoring Tool supports caregivers and family members in the process of building user-speci c knowledge, centered around user's pro le, family/social relationships and life events. It also enables the provision and tagging of multimedia objects, and is responsible for storing the gathered data in the knowledge base.
Reminiscence Sessions. The Reminiscence Manager is responsible for engaging the patient in the reminiscence process exploiting the available knowledge. A dialogue-based reminiscence session is driven by an extensible repertoire of interaction patterns, that allow prompting the user through speci c questions, 6 http://ontologydesignpatterns.org/wiki/Submissions:TimeIndexedSituation associated with media objects such as images that are contextually shown on the touchscreen available onboard the robot. An interaction pattern consists of: (i) a precondition, with constraints expressed as queries over the knowledge base, de ning under which conditions the prompting question can be used; (ii) a parametric prompting question to be used for triggering reminiscence, represented as a partially-formulated question template containing variables to be instantiated; (iii) one or more queries over the knowledge base providing a binding for the variables in the prompting question. Targeted questions are de ned to cover the aforementioned knowledge elements, including life event types, people and tagged media objects. As informally shown in Fig. 2, given a photo with information on where it was taken, and who appears in the picture, examples of parametric prompting questions that also exploit family/social relationships include \Is that your {familyRelationship} {personName} in the photo with you?" or \That's you {patientName} in the photo with your {familyRelationship} {personName}. Where was this taken?". Similarly, the association between photos and life events can be exploited to formulate questions about the event. Assuming, for example, that there is a marriage event where the PWD is one of the partners, prompting questions such as \ {patientName}, you got married to {partnerName} in {eventDate}. Where did you get married?" can be formulated.
Currently, prompting questions assume a speci c, known answer, from a
simple positive/negative answer to speci c persons, places, dates or events. This
constrains the language interpretation domain and understanding capabilities
are exploited to check user's answer and then either providing the patient with
intermediate hints or the expected answer, or moving to the next question. The
selection of the interaction patterns is a dynamic process, driven either by
patient's replies or by traversing the links in the knowledge graph on the basis of
the dialogue context and history. So, for example, a question about when a photo
was taken can be followed by a question concerning a person that appears in the
picture, and then move to a life event where the person participated in, and so
on, exploiting the properties in the represented entities and the semantic roles
in the represented frames. The additional complexity coming from open-ended
questions (e.g., \How was your wedding day?") that aim at stimulating
conversation is under investigation. In this case, the understanding capabilities focus
on the recognition of named entities (e.g., mentioned people or places) and the
detection of frames matching with life events, exploiting the machine reading
capabilities provided by FRED [
The reminiscence application we presented will be deployed for validation as part of the MARIO software framework on eleven Kompa-2 robots that are currently being trialled for acceptability with patients in di erent dementia care settings in Ireland, the UK and Italy. Initial validation steps, supervised by caregivers, aim at qualitatively evaluating the approach in real-world settings. Gathered feedbacks and trial results will drive the evolution of the application to enable the robot to undertake reminiscence sessions in an unsupervised and autonomous way. Ongoing and future work includes the ability to acquire factual knowledge from the conversational interaction with the user, as well as the introduction of sentiment analysis capabilities to assess emotional aspects and identify triggers generating positive feelings to be favoured in reminiscence.
Acknowledgments. The research leading to these results has received funding from the European Union Horizon 2020 { the Framework Programme for Research and Innovation (2014-2020) under grant agreement 643808 Project MARIO \Managing active and healthy aging with use of caring service robots".