In this paper we present a framework for conversational awareness and conflict resolution in spoken dialogue systems for home care assistance. Conversational awareness is supported through OWL ontologies for capturing conversational modalities, while interpretation and incremental context enrichment is facilitated through Description Logics reasoning. Conflict resolution further assists the interaction with end users, facilitating exception handling and context prioritisation by coupling defeasible logics with medical and profile information.
Spoken dialogue systems aim to assist end users in satisfying their information needs, hiding the complexity of knowledge representation and query languages. Of the numerous domains of interest, conversational assistance in healthcare is a notable case where natural language interfaces provide unique solutions to patients and medical experts. In addition, multimodal dialogue-based systems overcome the limitations of dialogue systems that use speech as the only communication means, collecting and analysing information from multiple sources and modalities.
The presented framework focuses on enriching multimodal dialogue-based agents with (a) intelligent context aggregation for conversation understanding, and (b) conflict resolution of domain inconsistencies and conflicts. To this end, OWL ontologies are used for modelling multimodal information (e.g. verbal and non-verbal modalities) and the semantics that underpin the interpretation logic, while defeasible logics [1] provide the non-monotonic semantics needed to deliver advanced conflict resolution strategies. In the domain of natural language interfaces and dialogue-based systems, ontologies such as WordNet and BabelNet, provide the vocabulary and semantics for content disambiguation [2]. Ontologies and Description Logics (DL) [3] have also been used in NLP for co-reference resolution [4]. In multimodal fusion, ontologies are used for fusing multi-level contextual information [5]. For example, [6] presents a framework for coupling audio-visual cues with multimedia ontologies. Relevant approaches are also described in [7] for multimedia analysis tasks. As far as defeasible reasoning is concerned, the non-monotonic semantics of the logic has been mainly used for building argumentative dialogue-based systems [8] or resolving conflictual arguments through counterarguments [9]. Through the use of DL reasoning for conversational awareness and defeasible rules for conflict resolution, this work focuses on conversation understanding and high-level conflict resolution. 3
Contextual information, such as multimedia information (e.g. speech analysis, named entities and concepts) and video analysis (e.g. gestures, facial expressions) is mapped to ontological entities in a hierarchical manner. The topic hierarchy defines the way conversational observations can lead to the derivation of high-level interpretations. In terms of DL semantics, the Topic (root) class is defined as:
For example, the recognition of a topic that indicates a pain problem is defined as:
Topics can be further specialized hierarchically, defining additional contain
property restrictions. For example, for the recognition of certain symptoms of pain, e.g.
headache based on language analysis and deictic gestures, (
(
The hierarchical topic decomposition also facilitates the descriptive modelling of topic-related semantics, i.e. to model descriptive information that does not directly define the conversational topic but provides useful information to drive the interaction with the user (see Section 5). Descriptive context is modelled in terms of the DescriptiveContext hierarchy, whose root class is defined as:
The descriptive context of a topic is specified through one or more requires property assertions about domain concepts. For example, PainTopic can be further associated with structures denoting the intensity or the part of the body:
Similarly, the descriptive context of headache may contain structures relevant to sleep quality or coffee consumption:
⊓ (∃requires.SleepQuality ⊔ ∃requires.CoffeeConsumption)
(
Context-based reasoning aims at coupling the semantics of conversational awareness with background knowledge, such as medical and profile information, in order to acquire a better understanding of the situation, resolve conflicts and provide the most plausible responses. Each conversational topic is associated with a defeasible rule base that handles domain contextual semantics. Assuming that is the set of all conversational topics supported (∀ ∈ , ⊑ ), we define
∀ ∈ , = { : ( ) ↬ ( )} where is a unique label of the rule, ( ) is the antecedent, ( ) is the consequent and ↬ indicates the rule type: strict (→), defeasible (⇒) or defeater (). Intuitively, the detection of triggers the inference mechanisms of the defeasible rule base . 5
We describe the simulated evaluation of our framework that is part of the KRISTINA agent [10] (Fig. 1) and involves interaction with users at a home in order to acquire information about their condition and suggest treatments for frequent problems. In one of the evaluation scenarios, the user informs the agent about feeling pain (“I feel pain”). The Dialogue Manager (DM) collects the incoming verbal observation, which involves a hurt reference captured by language analysis, and builds the current context:
The context is then passed to Conversational Awareness to interpret the topic and,
according to axioms (
(
The new contextual information is passed to Conversational Awareness to reason
again on the current context. The enriched context now satisfies (
Generic Content
Language Generation
Avatar
Ontology-based Question Answering
Conversational Awareness
Topic Understanding
DL reasoning
Descriptive Context Context-based Reasoning
Conflict Resolution
Defeasible rules decides not to further enrich the context (e.g. by asking questions about sleep problems or coffee consumption habits) and propagates the current context (HeadacheTopic) to Context-based Reasoning for generating appropriate responses.
The generic defeasible logics rule base for HeadacheTopic involves the following defeasible rules for relevant treatment recommendations: 1: 2: 3: 6: 4: ℎ 5: ℎ 5 > 3 and
= { ℎ ℎ ℎ ℎ ⇒ ⇒ ⇒ , , , _ _ _ User Profile }
According to the elderly’s profile, he suffers from frequent migraines and caffeine intolerance. Therefore, the following personalized rules are also considered: ⇒ , ¬
¬
In addition, the scenario involves a sleep sensor that monitors night sleep quality and provides an assessment every morning. The following defeater enriches context-based reasoning by fusing sleep quality information that overrides 1:
The rule base of the example (via SPINdle [11]) finally recommends that the user should take strong painkillers for his headache, since he suffers from migraines, overriding other plausible recommendations based on profile and sleep-related information. 6
In this paper we presented a framework for conversational awareness and conflict resolution in spoken dialogue systems combining ontologies and defeasible reasoning. OWL is used to model multimodal input and the semantics that underpin the conversational logic, while defeasible rules provide the non-monotonic semantics needed to deliver intuitive knowledge representation and advanced conflict resolution.
We are currently conducting pilots for collecting additional data and evaluating the framework with more use cases. In parallel, we are working towards further enrichment of the fusion and interpretation capabilities of the framework, so as to support additional use cases, e.g. taking into account emotions and facial expressions.
Acknowledgements. This work has been partially supported by the H2020-645012 project “KRISTINA: A Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities”.