Mixed-Initiative Creative Interfaces for Collaborative Early-Stage Design

Abstract

This position paper outlines my ongoing research into how creativity unfolds in early stage design activities, and how such creativity can be supported. It considers the challenges posed by this context in terms of possible mixed-initiative creative interfaces; and poses questions for my own research, and for designers of mixed-initiative creativity support tools.

-

Copyright © 2017 for this paper is held by the author(s). Proceedings of MICI 2017: CHI Workshop on Mixed-Initiative Creative Interfaces.

Author Keywords

Mixed-initiative interaction; creativity support; codesign;

ACM Classification Keywords

H.5.m. Information interfaces and presentation (e.g., HCI):

Introduction

In the Creativity in Blended Interaction Spaces project at Aarhus University in Denmark, we are investigating the potential for integrating multiple digital devices and different analog materials into shared environments that support individual and group creativity [ 5 ]. This research typically studies creativity in early-stage design. We start from the perspective that tools and materials support the creative agency of human users, and that creative activities take place in complex situations.

However, A.I. is now a feature of commercial creativity support packages, e.g. generative design CAD tools [ 12,3 ]; conversational agents are a commonly used interaction method, e.g. in smartphones and social media [ 4 ]; and computing has become ubiquitous [ 1 ]. The way computational systems are used in creative practice changes. Understanding how this change unfolds, and the opportunities it presents, is an important part of our research. In this position paper I use Lawson and Loke’s framework for understanding the role of computers in design creativity [ 13 ] to pose some questions for my own research, which I hope are also relevant to others.

Mixed Initiative Creative Interfaces

This workshop is focused on those computational systems that are considered mixed-initiative creative interfaces (MICIs). This seems to be a useful category on the spectrum between tools that support human creativity and systems for autonomous computational creativity. To help understand how systems might be positioned on this spectrum, I take guidance from HCI research into mixed-initiative interaction e.g. [ 2,10,11 ]; and to help position them within my own area of study, I take guidance from design research into the roles computers might play in creative design activities [ 13 ].

Mixed Initiative Interaction

Mixed-initiative interaction aims to develop methods that enable computer systems to: “support an efficient, natural interleaving of contributions by people and computers, aimed at converging on solutions to problems” [ 11 ], and “where each agent can contribute to the task what it does best” [ 2 ]. Commonly it has been treated as a form of dialogue, in which agents dynamically adapt their initiative style, and use an interaction mode that supports human-style problem solving. Allen [ 2 ] identifies four levels of mixedinitiative interaction: 1. Unsolicited Reporting: The computer monitors work and if it identifies a problem notifies the user; but does not take or coordinate further action. 2. Subdialogue Initiative: The computer can initiate subdialogues, e.g. asking for clarification. Once clarified, initiative reverts to the user. 3. Fixed Subtask Initiative: The computer is responsible for particular tasks. The user sets a goal then the computer retains the initiative whilst working on this task. On completion initiative reverts to the user.

4. Negotiated Mixed Initiative: The computer monitors

the current subtask and assesses whether: it is able to, has the resources to, and is best qualified to coordinate interaction.

Horvitz [ 10 ] highlights the key decisions that mixedinitiative systems must take to support collaboration, which include: 1. When to engage users with a service 2. How to best contribute to solving a problem 3. When to pass control of problem solving back to users 4. When to query a user for additional information

Roles for Computers in Creative Design Processes

Lawson and Loke [ 13 ] imagined a CAD tool in which creativity support was provided through conversation between designer and system. They identify five roles that such a tool might adopt: 1. Computer as Learner: The computer absorbs and remembers. In conversation with a designer it records associations, and asks for an explanation of things it does not understand. 2. Computer as Informer: The computer answers queries, and provides information and examples in response to specific requests from the designer. 3. Computer as Critic: The computer checks and comments on the validity of ideas. It takes a critical stance, presents possible alternative views, perhaps warning about potential mistakes. 4. Computer as Collaborator: The computer builds on what others have said. It takes a positive and supportive stance, e.g. elaborating on ideas and extending metaphors. 5. Computer as Initiator: The computer develops new perspectives, suggests new directions for ideation when others have no more to say, and takes initiative in generative activities.

MICIs in Early-Stage Design

The call for participation in this workshop identifies procedural content generation for computer games as an example of how mixed-initiative interfaces are providing creativity support, e.g. [ 15 ]. Autodesk’s Dreamcatcher project [ 3 ] also seems to be an example of human and A.I. in creative collaboration. In simplistic terms, both these examples are based on a human designer setting parameters and an A.I. generating and partially evaluating large numbers of digital alternatives before presenting these back to the human user for further evaluation. In both cases, the model of creativity is based on searching a possible solution space. Does this represent a limitation in the scope of creative applications using mixed-initiative interfaces? Or do these systems offer an indication of future potential in other areas? Can mixed-initiative interfaces help us overcome some the issues raised by our research into other creative practices?

Early-stage Design Activities

Many of the creative practices we study within CIBIS are at the early stages of design processes, where the situation is not yet well understood and there is much ambiguity. These activities typically involve seeking and sharing information and insight, finding sources of inspiration, and framing inquiry.

Designers often use Post-It Notes to record, share and organise ideas, and through their use of Post-It Notes also develop and extend these ideas. The Post-It Notes help them to think about and manipulate their ideas, and construct semantic relationships that support longterm memory [ 6 ].

What might a mixed initiative interface that contributes to these processes be like? It seems probable that machine learning and natural language processing can play a role in making semantic connections between ideas, and machine vision might track individual Post-It Notes as they are manipulated through a design activity. A system that embodied Lawson and Loke’s [ 13 ] learner and informer roles might usefully augment designers’ Post It Note activities, but the question for a mixed-initiative interface would remain how and when to contribute appropriately. Perhaps this might be facilitated by the conventions, rules and structures that human participants typically follow, e.g. when brainstorming. Might these provide initial guidelines for how a system would make Horvitz’s [ 10 ] key decisions, and for selecting which of Allen’s levels [ 2 ] is most appropriate? When working with stakeholders during co-design workshops, we have found that activities such as making collages from photographs can help them interpret visualized data. These activities encourage participants to share their experiences and insights, and through this explore possible contexts in which data were generated [ 9 ]. This provides an important source of inspiration to support collaborative ideation. A mixed-initiative system that could work with participants interactively as they explore data would be extremely interesting to investigate, and search tools that utilise analogy or metaphor offer powerful sources inspiration e.g. [16]. However, the activities undertaken during co-design workshops typically aim to explore participants’ subjective experiences, and so any system should sensitively draw these out, and be aware of the possibility of priming responses too strongly. Could a mixed-initiative creative interface also play this type of role, i.e. computer as facilitator?

Supporting Reflective Practice

Our research group also develops tools and investigates methods to support designers’ reflective practice. For example we have investigated how revisiting projects to reflect on the way a design space changes increases awareness of the constraints introduced by particular design choices, qualifies understanding of how design activities filter the design space, and prompts reconsideration of disregarded opportunities [ 8 ]. This requires detailed design documentation, which can significantly add to overhead.

Systems that interactively record design activities, monitor them and learn about what might be important, and subsequently prompt designers’ critical reflection, could be of great benefit in this and similar contexts. For example, machine learning might be used to extract key moments, or uncover patterns and make connections between different concepts in design conversations. A system that embodies each of Lawson and Loke’s learner, informer and critic roles [ 13 ] might be a useful addition to designers’ reflective practice.

Engaging with MICIs

The tough question for mixed-initiative interaction remains how and when computational systems should interject, engage users, and take initiative. Familiar instances, such as spellcheck and grammar checking in word processing software, struggle to solve this satisfactorily; and conversational agents can be frustrating [ 14 ]. This is likely to be further complicated in situations where groups of human collaborators interact with ecologies of interactive artifacts and intelligent agents.

A survey of UX practitioners working with machine learning [ 7 ] surfaced a number of challenges designers face working on the type of systems likely to play a leading role in mixed-initiative creativity support for the areas discussed here. The danger of systems that monitor activity appearing creepy was highlighted as an important UX concern, and the probability that systems require ground truth from large amounts of data challenged typical approaches to prototyping. Other difficulties designers raise, which might be indicative of some of the challenges MICIs will face, included: the implication that “learning” means the system and data will change over time, and be dynamic at a large scale; and that statistical correlations lack common-sense, can appear simplistic and stupid, and therefore false negatives or false positives can be hard to assimilate. The wider issues designers face working with intelligent systems are likely to be increasingly prominent in systems that aim for Negotiated Mixed Initiative [ 2 ], and where the computer is the Initiator [ 13 ].

Acknowledgements

This research is funded by The Danish Innovation Foundation grant 1311-00001B. CIBIS

References

1. Gregory D. Abowd . "What next, ubicomp?: celebrating an intellectual disappearing act." In Proceedings of the 2012 ACM Conference on Ubiquitous Computing , pp. 31 - 40 . ACM, 2012 .

2. James F Allen. 1999 . Mixed Initiative Interaction . In IEEE Intelligent Systems and Their Applications . 14 , 5 : 14 - 16

3. Autodesk . Project Dreamcatcher. Retrieved January 27th 2016 from https://autodeskresearch.com/projects/dreamcatch er

4. John Brownlee . 2015 . Apple finaly learns AI is the new UI . Retrieved January 27th , 2017 from http://www.fastcodesign.com/3047199/applefinally-learns -ai-is-the-new-ui

5. CIBIS. http://cavi.au.dk/projects/creativity -inblended-interaction-spaces/

6. Graham Dove , Sille Julie Jøhnk Abildgaard, Michael Mose Biskjaer, Nicolai Brodersen Hansen, Bo T. Christensen, & Kim Halskov. 2017 . Grouping Notes Through Nodes: The Functions of Post-It Notes in Design Team Cognition . To Appear In Design Studies, Special Issue on Designing in the Wild (Papers from Design Thinking Research Symposium DTRS11) . Elsevier.

7. Graham Dove , Jodi Forlizzi, Kim Halskov, & John Zimmerman . 2017 . UX Design Innovation: Challenges for Working with Machine Learning as a

8. Graham Dove , Nicolai Brodersen Hansen, & Kim Halskov. 2016 . An Argument For Design Space Reflection . In Proceedings of the 9th Nordic Conference on Human-Computer Interaction , (pp. 17 - 26 ). ACM.

9. Graham Dove , & Sara Jones. 2014 . Using Information Visualization to Support Creativity in Service Design Workshops . In ServDes.2014 Service Future , Proceedings of the fourth Service Design and Service Innovation Conference , (pp. 281 - 290 ), Linköping University Electronic Press

10. Eric Horvitz . 1999 . Principles of Mixed-Initiative User Interfaces . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. (CHI 99) : 159 - 166 .

11. Eric Horvitz . 2007 . Reflections on Challenges and promises of Mixed-Initiative Interaction . In AI Magazine . 28 , 2 : 19 - 22

12. Paul Keskeys . 2016 . How Generative Design Will Change Architecture Forever . Architizer. Retrieved January 26th from http://architizer.com/blog/howgenerative -design-will-change-architecture-forever

13. Bryan Lawson and Shee Ming Loke. 1997 . Computers, Words and Pictures . In Design Studies, 18 ,2: 171 - 183

14. Ewa Luger , and Abigail Sellen . 2016 . Like having a really bad PA . In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI'16) : 5286 - 5297

15. Gillian Smith , Jim Whitehead and Michael Mateas . 2011 . Tangara: Reactive Planning and Constraint Solving for Mixed-Initiative Level Design . In IEEE Transactions on Computational Intelligence and AI in Games 3 , 3 : 201 - 215 .