We propose a framework for design and evaluation of mixed initiative robotic systems, focusing mostly on the robot initiative in the case of a robotic shopping trolley. Throughout, we consider the implications of the movement modality in the robot initiative and the communication that follows it. We illustrate our considerations with our experience in designing and evaluating mixed-initiative human-robot communication with a mock-up robot and subsequently an actual robot platform.
Constructing robots that propose own initiatives in a socially acceptable manner
is a challenging task. Multiple modalities can be used artfully to allow the robot
to indicate its initiative in ways that do not disturb the user, yet manage to
get the message across. In this position paper we examine the design space for
mixed initiative, in the case of a robotic shopping trolley that we are building in
the Commrob project (www.commrob.eu), which aims at developing techniques
for multimodal communication with robots and the safe navigation in crowded
human every-day environments. Furthermore we exemplify within our design
space points of interest related to a speci c modality, the robot movement, in
what is termed \movement as communication" [
Multi-modal, mixed initiative interaction is used by our robotic trolley to
help the customer to nd the desired products without extensive search in big
supermarkets and to relieve the customer from the burden to push the shopping
cart from his own force all the time, especially if the trolley is heavily loaded
or the customer is an elderly person. The interactive Behavior Operated Trolley
(InBOT, [
Our design space conceptualization proposes the focus of design to support robot initiative around the chronological phases of the human-robot communication: the robot initiative, the human perception of initiative, the human reaction, the robot detection of that reaction, and the robot response to the human reaction. With our focus on movement as a modality, we will use as running examples two applications of robot initiatives in the shopping case: Suggestion of product: The robot suggests an item that the user is probably interested in. This can be an item of the shopping list which is passed at the very moment. The robot can choose between the di erent modalities available to it. In this case slowing down and uttering a verbal comment (e.g. 'we are passing' followed by product name) can be most familiar to the user. This could be assisted by turning the robot in the direction of the item which would catch more attention. We are in the process of evaluating a design for this scenario. Suggestion of route: The robot suggests an alternative route which probably poses advantages for the user e.g. because it is shorter than the originally planned route or the planned route is crowded or blocked. Another interesting application can be leading a visually impaired person, who is holding on to the force sensitive handlebar, along a planned route and this way acting as an intelligent white cane. Here the most suitable modality might be slowing a down a little and slightly turning in the direction to be indicated while uttering a speech output. This provides the user with an extra modality to react to the robot's action: the resistance force on the handle bar. 2
We will now show how the design space for supporting robot initiative can be supported by considering the chronological communication phases and by examining each phase in turn. We also consider the role played by the phases in the evaluation of mixed initiative designs, and mention our experiences to date with such evaluations.
Machine initiative is the rst phase of the process. The design considerations for this phase are depending on the application, i.e. what the initiative is taken about. We have exempli ed above two such applications and the possible design decisions in regard to modalities used (movement, speech, GUI) and how to use them.
Human perception of machine initiative follows, after a certain time, the initial initiative. The time needed for the user to notice the initiative, as well as other aspects of the human perception are important factors to consider in a human-robot interaction set-up where human attention may be directed to other components of the environment. Further robot actions may come too early for the user to comprehend them in their logical relation to the initiative. Human response to the machine initiative can be of di erent natures. It can range from tacit perception whereby the user notices the initiative but does not react in any way that can be understood by the machine, to acknowledgement whereby the user reacts to the initiative without attempting to explicitly respond by interacting with the machine, to interactional response in which the user interacts with the machine to accept, strengthen, weaken, or reject the machine initiative. For example, for product suggestion, the user can acknowledge by slowing down, and/or looking at the product, without explicitly interacting with the trolley, yet both such acknowledging cues can help the trolley in shaping its further actions. An interactional response would occur when the user is stopping the trolley via a GUI touchscreen, stopping the trolley by scanning the o ered product. For route suggestion the user can amplify or reduce the robot turn using the robot's haptic handle.
It is important to consider whether the interaction is expected to take place in one of the modalities in which the initiative was presented or it can come in other modalities. For example, in the case of route suggestion, the user may choose to press a Stop button on the robot touchscreen GUI or more directly steer the robot in the opposite direction from where it turned to suggest another route.
In case of user acknowledgement without further interaction, a default
action may be de ned by design. In some cases acknowledgement without explicit
interaction may be enough to consider the robot initiative as accepted, while in
other cases it may only account as partial acceptance, thereby weakening the
initiative or canceling it alltogether. The di erence between acknowledgement
and explicit interactional response has played a role for us also in evaluating
the movement as communication designs in with mock-up robots [
Design plays a role also in facilitating interactional response. Not only the launching of the initiative can be designed but also the levers o ered to the user for responding to it can be made more available. In the case of a product o ering, the user could not only respond by speech or touchscreen GUI interaction, but also by scanning the o ered product with a portable scanner, which may be more handy than reaching for the touchscreen. To reject an o er, the user could also employ the movement modality by pushing the slowing-down trolley (in the case of product o ering) or turning it in the initial direction (in the case of route suggestion or product o ering via turn).
Such prompting for user interaction is also useful when the user has the initiative, e.g. when the user asks for the trolley to move to a destination, the trolley can start slowly and accelerate progressively, to allow for the user to stop the action or alter it.
Machine detection of human response is fundamental in providing a socially competent robot behavior. While explicit interactional response is straightforward to detect, passive acknowledgement may play an important role in the initiative interaction, yet it may be harder to detect. For example in the case of product o ering via slow-down, the user slowing down themselves can mean that the o er is acceptable (though not surely accepted), while the user not slowing down is almost certain to mean o er rejection or lack of notice about the offer. Similarly in our evaluation work (both with mock-up robot and with the InBOT) we have seen users looking at the o ered product but not following this up with any explicit interaction. This inspired us the idea that gaze detection can be a useful indicator to automatically detect acknowledgement, which can further help in continuing the slow-down robot action for an eventual explicit interaction response. Another detection of acknowledgement (and acceptance) is the posture of the user reaching out to the shelf to grab the product, which can be detected by image analysis. None of these acknowledgement cues represent certainties, yet probability-based techniques similar to those used in speech or gesture recognition can be employed to combine these cues in order to provide a more natural coordination between user and robot.
The product-o ering example thus poses at least three important sensing challenges: sensing user moving speed, sensing the user posture in relation to the shelf, and sensing the user gaze; all these are important challenges since detections need to take place while the robot itself moves. Furthermore this example is among the most illustrative to suggest that the interaction design of machine initiative does not end with the design of the initiative itself, but also includes detection of user response or uptake for the machine to be able to continue the interaction `properly'.
Machine reaction to human response involves the machine follow-up of the initiative based on the user response and its machine detection. Several cases need to be considered here: the user may be acknowledging, accepting, strengthening, weakening or rejecting the machine initiative. All these possibilities may lead to design decisions for a certain application. Furthermore, machine detection of acknowledgement can be followed by detection of acceptance or strengthening (or indeed weakening or rejection) so the machine can react to acknowledgement in a way that is consistent with the following reaction in case of subsequent explicit user interaction.
For example, in case of product o ering acceptance, we might want to design the trolley to stop, or to do even more by placing itself in a position which facilitates easier product moving from the shelf to the trolley. In case of sensing user acknowledgement without any detection of acceptance or rejection, we could decide to make the robot prolong its slowed-down course, to give more time to the user to decide on the machine initiative. 3
While initially [