In the CommRob project1 we are investigating Robot Assisted Shopping. We are considering the e ects on usability when allowing for mixed-initiative dialogue. It is noted that when adding a robotic assistant to a scenario that was previously involving only one agent, two new tasks are created: collaborative interaction, and learning an interface. Evaluation of mixed-initiative dialogue becomes complicated because it is not straightforward to separate the overall task performance from the attributes brought by mixed-initiative interaction.
Some use scenarios for which natural language user interfaces are being developed, draw on situations where two or more people naturally engage in collaborative communication (e.g., asking for time-table information, scheduling meetings etc). This is not the case in the CommRob project, where we are investigating how a robotic trolley can enhance shopping in a supermarket. The normal shopping scenario does not (usually) involve two agents solving the task using natural language. In our scenario the user enters a shopping list, e.g., by selecting products on a touch-screen, or by using speech commands. The robot then guides the user to the product locations allowing the user to scan the bar code and put the product in the trolley. During such this scenario we want to allow the initiative to shift back and forth between user and system based on the what these agents consider is bene cial for the collaborative task of assisted shopping. 1.1
Many approaches to robot interfaces assumes a xed-initiative, or
commandbased style of interaction, based on a controlled language centered around action
verbs that are directly translated to physical robot actions, e.g., \go forward" [
Creating a strategy for handling mixed initiative relies on information and actions from other types components. Selection of a strategy for handling mixedinitiative is a challenge, and we will only brie y list some of the components necessary to consider as suggested by [6]: { A natural language dialogue model: the system needs to handle natural language dialogue, manage turn-taking, and engage in sub-dialogues, only to mention a few things that are relevant for mixed-initiative. { A domain model de ning the task, agent roles and obligations. { A model of user attention to be able to decide when the user is possible to interrupt. { A strategy for managing initiative: based on the current status of the dialogue, the task, and the users' attentional state, the system should decide whether to take action to challenge the initiative?
User
Domainunderstanding
User
+
Domain- System understanding understanding
Robot
Mixedinitiative strategy User model
Natural Language capability Domain-model Given the research in the eld of social robotics [7] it is perhaps uncontroversial to assume that users and robotic agents form some kind of social relation when engaging in interaction. One way of understanding this social relation is to think of robots and humans in terms of roles. Thus the robot may be considered a helper, a facilitator, an information provider or even a sales-agent acting on behalf of the store owner. These roles can be performed through the employment of social communicative behaviour involving various interaction styles based on domain knowledge and task capabilities. By de ning social obligations [8] that comes with a role, and associate these with communicative goals is one approach to model initiative. For instance an initiative model should consider when is it alright for a robot or a computer to interrupt someone, given the social obligations of the role the robot plays in a situation [9]. Other social obligations may be inherited from the de nition of the role. Being a helper could by de nition entail that the robot should not ignore products on the shopping list or go to a product by another brand. Designing a initiative strategy therefore involves deciding what behaviours to engage dependent on what is believed to be an appropriate action that contributes to the joint task, taking the goals of the involved agent roles into account. In the shopping scenario this may involve the following conditions and resulting actions: { Robot passes a product that is in the shopping list on the route to another product ! Indicate products to user. { Planned route changes based on information gathered (including communication with other robots) ! o er another route. { When passage is too narrow, or way is blocked ! suggest a manual override of steering (our robot prototype has a haptic steering device). { User given preferences, e.g. concerning allergies, special diets ! suggest alternative or suited products.
{ In case of severe problems ! suggest to call sta or call sta Since the communicative framework [10] we are investigating in the CommRob project provides a general approach to representing interactive systems, initiative can be taken by the robot using several modalities, including GUI, speech output, robotic full-body gestures [10, 11]. 1.3
Until now the shopping scenario has been described as replacing one old and known task with another. But we might also understand robot assisted shopping two new, but related, tasks: collaborative, assisted shopping and learning the use of a multimodal user interface. In our view we need to this into consideration and evaluate the robot system along several dimensions such as overall usability, user experience, but also speci c components such as the mixed-initiative strategy. Two approaches to evaluation of mixed-initiative dialogue have been proposed by Guinn [12]: performance measures of how the system model ts descriptive data, i.e., a corpus of mixed-initiative interaction; analysis of the dialogues resulting from the initiative model to establish if they have the desired qualities.
As for the rst approach, we have already established that we cannot simply collect data on existing collaborative shopping to test our system against. One common approach for collecting data on interaction with a future system is to build a prototype using the Wizard-of-Oz technique [13, 14]. The prototyping involved in the Wizard-of-Oz method provides the means to try-out and evaluate mixed initiative strategies early on in the development process.
The second approach involves nding what the desirable qualities of a mixedinitiative dialogue for a robot is. This raises several challenges. First of all the robot needs to have some initial natural language capability that can support the mixed-initiative strategy, e.g., performance of speech recognizer may account for the overall performance of the system [12]. One approach to qualitative evaluation of dialogues is to use synthetic dialogues [14] whereby a designer constructs dialogues the system is intended to handle. This approach allows for evaluation towards what we may call a synthetic corpus. Doing this for a multimodal system is a challenging task, and should therefore be complemented with other methods for eliciting use data, such as Wizard-of-Oz.
From an experiential point of view we should also take into account that robot systems are physically embodied and that users may form (limited) social relations to them. Meaning that we need to evaluate to what extent the movements and anthropomorphic qualities of the robot a ect the way the mixed-initiative dialogue is experienced by users.
Another approach to qualitative evaluation of mixed-initiative strategies of robotic helpers is to use heuristic evaluation [15] using design guidelines [16] or maxims for communications [17]. Such guidelines should also be considered during the interaction design phase. 2
In this position paper we argue that introducing a robot into a scenario that was previously managed by a single user, introduces new tasks: collaborative mixedinitiative interaction with a robotic agent; and learning to handle a new interface. Evaluating this involves establishing evaluation of several components based on comparison with corpus data collected using hi- simulation or with early prototypes; or qualitative evaluation based on heuristics or synthetic dialogues.