This paper reports the design and evaluation following user-centered methods of a dancing robotic tablet prototype for co-creative human-robot interaction. An initial exploratory interview study served to obtain requirements for the design and implementation of a first prototype. This prototype was evaluated in a user study and subsequently improved. Two types of autonomous robot behavior were considered as creativity support and evaluated in a second user study. While imitation behavior was perceived as more intelligent; the generation behavior that attempted to challenge users and be different to the users' input led to a greater variety of gestures. Video recording analysis shows the potential of such autonomous behavior for the creative process, as users were inspired to some extent by the robot's input.
Tangible interactive systems have enabled new forms of creative expressions through
playful interactions in diverse areas of application such as education [
In this paper, we explore the design of an interactive robotic tablet prototype that allows the user to create a dance for it on a tabletop, intended as a creative and ludic 1 Authors contributed equally to this work. activity. We have designed two different co-creative strategies and carried out a study to understand how this collaboration between computer and human unfolds. We found that imitation behavior is perceived as more intelligent, while behavior that is notably different to the users’ input leads to a greater variety of gestures. The video recordings analysis showed that users were inspired to some extent by the robot’s autonomous input. Our observations contribute to get deeper insight into designing for future interactive co-creative systems.
The paper is structured as follows. Section 2 introduces the background and the work that inspired our research. Section 3 presents the design stages followed by the prototype implementation. Section 4 reports the user evaluation of the implemented co-creative strategies and Section 5 discusses the overall findings and observations. Finally, Section 6 concludes and introduces future work plans. 2
The area of co-creative systems pursues developing computer software to contribute
to creative processes in collaboration with humans [
A number of related papers have focused on technical aspects in humanoid dancing
robot systems; see [
Hence, an area of interest is that of interactive tabletop systems for creative play
performances or creative playful expression. An outstanding example is the Reactable
[
From the areas explored, we believe that to better examine the necessary
complicity and relationship of human and computer agents for the development of
future co-creative functions, we need a simple robot model that has only a few
degrees of freedom and that enables exploratory interaction as in the aforementioned
tabletop systems for creative expression. Moreover, an important remark by [
The development of the system followed a user-centered design approach and was carried out in three phases. First, an initial study was conducted, in order to get a general idea of how people would interact with a dancing robotic agent to order the dance movements. This study led to the design of a first prototype, which was evaluated in a user study. The results from this evaluation were then used to further improve the prototype. Ultimately, after implementing autonomous behavior, another study was performed to explore how people interact with the co-creative robot. All participants were students at our university, in their twenties, and from diverse disciplines ranging from Health Sciences and Psychology to Industrial Design or Computer Science. They self-reported that they dance “never” or “occasionally” in their free time. The interaction with the different robot prototypes was video-taped. Fig. 1 depicts the overview of the whole design process. The goal of the exploratory study was to gather initial user requirements concerning the robot’s design and the users’ expectations on the movements as a way to codesign the prototype. The study was carried out with eight participants. For the robot, a Pololu Zumo Robot for Arduino3 was used. Firstly, participants were given the time to explore the robot’s movement capability by controlling it through a mobile application which worked as a joystick. Secondly, participants were handed a cardboard prototype (see Fig. 2-a). They were asked to physically carry out movements with the cardboard proxy that they would imagine the robot to do, while thinking aloud. Third, participants were asked to draw gestures matching their previously performed movements on a tablet (see Fig. 2-b). Finally, they were asked how they would imagine the outer appearance of the robot. 2 Video of the final prototype: https://www.youtube.com/watch?v=A60fLKBpI7Y 3 Pololu Zumo Robot Specifications: www.pololu.com/product/2510
From the information elicited in this stage we observed that participants split the dancing performance into sequences of single gestures, and most of them were freeform gestures rather than symbolic commands (see Fig. 2-c for some samples). Typically, the free-form gestures were more complex and more than half of the gestures included a lot of curves and zig-zag movements. Sometimes participants indicated that they would like to be able to repeat movement steps. The participants’ input was used to develop the prototype for the next study, mainly resulting in (1) considering free-style touch input to indicate the dance instead of a predefined gesture vocabulary, (2) treating each gesture as a single but complete path that the robot should carry out, and (3) allowing users to repeat dancing steps. As for the appearance of the robot, most participants mentioned some need to add a more special and fancy case covering the wheels, with a curved shape, smooth trajectories and elements such as fabrics to create a less static look. As a result of these suggestions, we proceeded to add an oval plate with wheel protectors on the Pololu Zumo robot, to which a tablet Samsung Galaxy Tab A 7.0 SM-T280 and a skirt fabric can be attached to meet the users suggestions (see Fig. 3). Taking the physical design of the prototype in Fig. 3, we implemented an Android app to capture free-style dragging gestures to be transformed into robot movements. A gesture consists of a continuous drag without lifting the finger. The gesture drawn is transformed into a list of points. Then these points are transformed into a sequence of timed commands for the motor wheels, which will drive the movement in the physical robot. Fig. 4 shows the setup for user tests. Once the dance step has been completed, the drawing screen is displayed again awaiting new gestures. The coordinates’ list of each gesture is saved in a text file, to allow later analysis or re-enactment. This first prototype was evaluated in terms of usability with twelve participants, who did not participate in the previous study. During user testing, the participants were asked to create a dance by indicating movements on the tablet which would fit with the music being played in the background. Two types of background music (a Mozart sonata4 vs Gangnam Style5) were tested to investigate their influence on the creativity of the movements.
The usability of the prototype was evaluated by means of a questionnaire, based on
the Post-Study System Usability Questionnaire (PSSUQ) [
Reproduction: The nearly exact replication of a previous movement.
Duplication: A modified version of an already existing movement, which does retain the essential form. Rotations, small changes, deletion of parts of a movement and mirroring of previous gestures are considered duplications as they are duplicating the behavior with small variations.
Fabrication: The rearrangement, re-mixture, or combination of two or more gestures in a way that if you split the gesture again, you retain the original gestures to some degree. Both gestures should have already been made before.
Innovation: The creation of a new movement that retains the core essence of the original gesture but making a clear transformation. It looks different from all other previous made movements, but it is not perceived as original.
Generation or original creation: The creation of something entirely new, which is not related or limited to the previous gestures.
A total of 361 interactions were gathered, on which the creativity assessment
procedure described previously was carried out. The interrater-reliability for the
creativity-rating of the movements was calculated between every pair of raters using
Cohen’s kappa, obtaining k1,2=0.621, k1,3=0.638, k2,3=0.667, leading to an average
interrater-reliability of 0.642, which as indicated by Viera and Garret [
Taking into consideration the feedback gathered during the user studies, we refined the application to have the final functional design and related screens depicted in Fig. 6. The main visual difference is that we made the first screen clearly asymmetrical by adding a button robot movement in order to prevent confusion in identifying the head and tail of the robot; that button can be used to request a robot-generated movement. Furthermore, we added a different stop button (shown in the last screen) for when the robot is carrying out a robot-generated movement. This capacity related to the cocreative strategies is presented in the next section. Besides the improvements suggested in the previous section, two different kinds of autonomous behavior were implemented in the next version of the prototype, which will be tested in the study in Section 4. These strategies allow the robot to contribute to the dance and thereby make the activity co-creative.
During the interactive activity, the application evaluates every gesture based on a scoring metric to classify it in terms of length and edginess (i.e. number of edges). Fig. 7 shows sample gestures according to these two dimensions. Both parameters are mapped to be in a range between 1 and 10, and the ratio edges:length is considered as the final score of the input gesture. If the standard deviation of the last five gestures is lower than a fixed threshold of 4, which was established after a pilot testing phase, the system enters into the autonomous behavior mode, carrying out a gesture movement according to the strategies described as follows.
The first behavior, generation, aims at challenging the user, a feature suggested by
[
In the second behavior type, imitation, the robot imitates the user, by repeating the movement corresponding to the last gesture of the user. In both conditions, autonomous behavior is triggered when the user provides five similar gestures in a row (in terms of length and edginess). The user can also request autonomous behavior of the robot by pressing the robot movement button in the drawing interface.
Fig. 7. Example of gestures in the pool of the four categories possible in terms of Length and Edginess.
Because we are interested first in exploring the user understanding of the
cocreative strategies, we rely on a pool of pre-saved gesture movements taken from the
user testing stage. Implementing complex algorithms to produce the intended
cocreative movements, based for example on evolutionary/bioinspired techniques (e.g.
[
According to Stahl’s taxonomy, totally novel content corresponds to the most
creative input [
The second user study was intended to evaluate the participants’ appreciation of the autonomous input and to find out to what extent two autonomous robot behavior strategies can support the creative process. Nine participants who did not take part in previous tests participated in this evaluation. They were requested to carry out the same experimental task as in the first study, with the difference of having only the Mozart’s sonata as music background and using the final version of the designed application in two conditions corresponding to the implemented co-creative strategies. After the interactive task was completed, the users filled in a questionnaire on creativity support and perceived intelligence, which is explained in the next section. Furthermore, videos of the interaction with the robot were coded and analyzed qualitatively. All participants interacted with both of the autonomous behaviors. The order of the behaviors was switched after each participant to counterbalance order effect. 4.1
To assess the perceived intelligence (PI) of the robot, we used the corresponding part
of the Godspeed Questionnaire [
We hypothesize that a possible reason for this is that the behavior challenging the users, i.e. the generation strategy, was somehow perceived as sort of random. Some users might not have understood when or why the robot would perform an autonomous movement. This effect was less pronounced for the imitation behavior, because users could figure out more easily that the robot was simply imitating the gestures of the user.
Additionally, the number of movements initiated by the robot was on average lower in the generation condition (11 movements) compared to the imitation one (13.17 movements). Since the number of movements initiated by the robot is for a big part determined by the similarity of the gestures proposed by the human, this indicates that to some extent there was a higher variety of gestures made by the human in the generation condition. This suggests that users might have been influenced by the robot behavior to try out more different gestures, which is something to take into account in the future development of co-creative strategies as it is intended to favor diversity of ideas.
The Creativity Support Index (CSI) questionnaire [
In order to better understand the reported perceptions as well as how the interactions were performed, we reviewed the video recordings. We looked for relevant events such as user comments, pitfalls, and any identifiable visible pattern on interactions (e.g. stopping the robot movement). The review did not reveal remarkable differences between the imitation and generation conditions. Three out of nine participants spontaneously reacted when they recognized the autonomous behavior of the robot. For instance, one user exclaimed, “I didn’t do that. That’s its own movement”, when she first noticed that the robot was performing an autonomous movement, despite knowing that could happen during the performance.
In both the imitation and generation behavior conditions, the robot’s autonomous movements were stopped often. Only one participant never stopped the autonomous movements. The main reason for stopping the robot was because the robot was about to bump into the borders of the dance floor. The collisions with the dance floor borders were the result of technical limitations, as the prototype did not have border recognition implemented. Seven out of nine participants repositioned the robot when it first collided with the walls, so that it could continue its autonomous movement. However, most of the times participants stopped the autonomous behavior when the robot was bumping repeatedly during a particular movement.
Other causes for stopping the robot’s autonomous behavior could be observed in the interactions. Several users tried to refine their gestures because the execution by the robot did not result in the exact and accurate movement they really wanted. In this case, the users would repeat similar movements again and again. However, producing several similar movements after each other was the criterion for activating the autonomous behavior, which caused the robot to interrupt the performance with its own autonomous movements. In such cases, the users often stopped the robot’s autonomous behavior as it was interrupting their idea generation process, meaning that the co-creation was not always welcome.
A third cause for stopping the autonomous behavior was repetitious behavior of the robot or more generally, long duration of the robot’s movements (particularly, during one autonomous movement the robot would repeatedly drive in circles for more than 20 seconds). Participants would first watch the robot perform its autonomous movement and then stop it after a while. Interestingly, some of the users who stopped the circular movement took up the idea of moving in circles. Two participants drew a circular shape immediately after having stopped the robot’s autonomous circular movement. This clearly shows that participants did notice and took into account the input of the robot. Stopping the robot’s autonomous movement can therefore also be interpreted as a way to take the robot’s creative input in some cases.
Users also actively asked the robot for input. Seven out of nine participants used the button that called for autonomous behavior of the robot. Overall, the button was pressed at least once and not more than three times by each participant (mean= 2.14). Different ways of using the button can be observed. Four participants used the button to find out what the robot could do on its own. One participant used the button to reproduce the autonomous behavior of the robot that she had just discovered. Two participants stopped the autonomous behavior enacted by robot to carry it out themselves right afterwards. This type of behavior seems to reflect a desire of being in charge and determining at what time the robot may perform its autonomous behavior. 5
We have presented the design process in developing a co-creative dancing robotic tablet involving user-centric methods. We reported how participants reacted to the autonomous behavior of the robot in order to explore how it can support the cocreative activity. The preliminary findings indicate that the system can still be improved in several ways.
First, some technical aspects need to be improved. The inability of the robot to recognize the borders of the dance floor resulted in undesirable collisions of the robot with those borders. This had a disruptive effect on the performance as a whole. Furthermore, the length of the robot movements was not easy for users to foresee.
Second, we found that the current implementation of the autonomous behavior of the robot might not be optimally supporting the creative process, as the CSI scores suggest. Some users tried to create a specific movement they had in mind and thus drew similar movements repeatedly. The similarity was recognized by the robot and it reacted by proposing a movement that was totally different (generation condition) or simply a repetition of the participant’s movement (imitation condition). Thereby, the robot was interrupting the creative process of the user at the wrong moments. A possible improvement would be a change in the criterion for enacting autonomous movement, in such a way that the interactive development of an idea is not disrupted. A straightforward change would be to allow for more than five similar movements, or to start the autonomous behavior after a certain time of no input from the user. After all, users could always be allowed to request a co-creative action from the robot.
We suggest that the types of autonomous behavior presented by the robot should be improved as well. Interrupting the creative process of the user with an idea that is the total opposite of what he or she had been doing in terms of length and edginess might not lead to a positive experience of the interaction. The robot could have been perceived as not paying attention or as unaware of the user activity, instead of collaborating with the user, and therefore not fully co-creative. Similarly, repeating the previous movement of the user 1:1 does not add variance to the interaction, although it may facilitate recognition of what the robot is doing. In human-human collaboration or a co-creative activity it is important to work with each other’s ideas and elaborate on what someone else did. Thus, both behavior types must be combined at different levels, enabling the robot to take the movement proposed by the user and transform it into something new. The user should still be able to recognize his or her original work in the new movement. Gradually, the robot could perform movements that differ more from what the user is doing, thereby carefully providing its own input without disrupting the co-creative process. All these observations are relevant to guide the development of future co-creative interactive strategies, and in particular the development of generative strategies that fulfil the users’ expectations and their understanding while making the implementation cost-effective. 6
In this paper, we have explored the design of an interactive robotic tablet for a cocreative ludic activity, following a user-centered design approach. Two different creativity-support strategies have been implemented: a generation behavior, during which the robot challenges the user by performing movements that are different from the user’s last five inputs, and an imitation behavior during which the robot simply repeats the user’s last input. Although the imitation behavior was perceived as more intelligent, the generation behavior is worth exploring in terms of co-creativity as users tried out new ideas and showed a greater variety of inputs compared to the imitation behavior. Users asked the robot for its input several times and took the robot’s previous suggestion into account when developing their own gestures. However, the autonomous behavior as implemented in this prototype was not optimal, possibly having a disruptive effect on the creative process of some users.
Overall, the observations and findings along the design process can be transferable to design other co-creative interactive systems, especially concerning timing, recognition of robot contributions by users, using user centric methods and input at several stages to continue developing the interactive system iteratively. They also open directions for future work regarding implementation improvements and research. Firstly, we plan to include an autonomous mechanism for the robot to avoid bumping against the borders based on edge detection using an array of infrared reflectance sensors. This needs to be combined with additional visual feedback on screen to report the position of the robot with respect to the borders using a tracker.
On the co-creative strategies side, there are at least two aspects to address. One is the timing, which implies research exploring when the robots’ contributions should be triggered and for how long. The other is the generation process itself. Departing from the gathered gestures for movements, we can now perform a deeper structural analysis to identify patterns and features that can be used as chunks. Then they can be considered in a generative approach guided by an evolutionary computation algorithm to match the required degree of variation to still introduce some originality without damaging certain recognition of the user’s original input. The objective function could be parameterized to offer not only the two conditions we have been testing in the present paper, but to provide several levels in between. Finally, with the incorporated changes and improvements, we will carry out a pilot test and conduct a broader user study evaluating the new implemented generation methods.
We thank all participants their helpful collaboration. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 701991.