Mixed-Initiative Creative Interfaces for Collaborative Early-Stage Design Graham Dove Abstract CAVI, Aarhus University This position paper outlines my ongoing research into Aarhus, 8200, Denmark how creativity unfolds in early stage design activities, graham.dove@cc.au.dk and how such creativity can be supported. It considers the challenges posed by this context in terms of possible mixed-initiative creative interfaces; and poses questions for my own research, and for designers of mixed-initiative creativity support tools. Author Keywords Mixed-initiative interaction; creativity support; co- design; ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI): Introduction Copyright © 2017 for this paper is held by the author(s). In the Creativity in Blended Interaction Spaces project Proceedings of MICI 2017: CHI Workshop on Mixed-Initiative Creative at Aarhus University in Denmark, we are investigating Interfaces. the potential for integrating multiple digital devices and different analog materials into shared environments that support individual and group creativity [5]. This research typically studies creativity in early-stage design. We start from the perspective that tools and materials support the creative agency of human users, and that creative activities take place in complex situations. However, A.I. is now a feature of commercial creativity dynamically adapt their initiative style, and use an support packages, e.g. generative design CAD tools interaction mode that supports human-style problem [12,3]; conversational agents are a commonly used solving. Allen [2] identifies four levels of mixed- interaction method, e.g. in smartphones and social initiative interaction: media [4]; and computing has become ubiquitous [1]. The way computational systems are used in creative 1. Unsolicited Reporting: The computer monitors work practice changes. Understanding how this change and if it identifies a problem notifies the user; but unfolds, and the opportunities it presents, is an does not take or coordinate further action. important part of our research. In this position paper I 2. Subdialogue Initiative: The computer can initiate use Lawson and Loke’s framework for understanding subdialogues, e.g. asking for clarification. Once the role of computers in design creativity [13] to pose clarified, initiative reverts to the user. some questions for my own research, which I hope are 3. Fixed Subtask Initiative: The computer is also relevant to others. responsible for particular tasks. The user sets a goal then the computer retains the initiative whilst Mixed Initiative Creative Interfaces working on this task. On completion initiative This workshop is focused on those computational reverts to the user. systems that are considered mixed-initiative creative 4. Negotiated Mixed Initiative: The computer monitors interfaces (MICIs). This seems to be a useful category the current subtask and assesses whether: it is on the spectrum between tools that support human able to, has the resources to, and is best qualified creativity and systems for autonomous computational to coordinate interaction. creativity. To help understand how systems might be positioned on this spectrum, I take guidance from HCI Horvitz [10] highlights the key decisions that mixed- research into mixed-initiative interaction e.g. initiative systems must take to support collaboration, [2,10,11]; and to help position them within my own which include: area of study, I take guidance from design research into the roles computers might play in creative design 1. When to engage users with a service activities [13]. 2. How to best contribute to solving a problem 3. When to pass control of problem solving back to Mixed Initiative Interaction users Mixed-initiative interaction aims to develop methods 4. When to query a user for additional information that enable computer systems to: “support an efficient, natural interleaving of contributions by people and Roles for Computers in Creative Design Processes computers, aimed at converging on solutions to Lawson and Loke [13] imagined a CAD tool in which problems” [11], and “where each agent can contribute creativity support was provided through conversation to the task what it does best” [2]. Commonly it has between designer and system. They identify five roles been treated as a form of dialogue, in which agents that such a tool might adopt: 1. Computer as Learner: The computer absorbs and future potential in other areas? Can mixed-initiative remembers. In conversation with a designer it interfaces help us overcome some the issues raised by records associations, and asks for an explanation of our research into other creative practices? things it does not understand. 2. Computer as Informer: The computer answers Early-stage Design Activities queries, and provides information and examples in Many of the creative practices we study within CIBIS response to specific requests from the designer. are at the early stages of design processes, where the 3. Computer as Critic: The computer checks and situation is not yet well understood and there is much comments on the validity of ideas. It takes a ambiguity. These activities typically involve seeking and critical stance, presents possible alternative views, sharing information and insight, finding sources of perhaps warning about potential mistakes. inspiration, and framing inquiry. 4. Computer as Collaborator: The computer builds on what others have said. It takes a positive and Designers often use Post-It Notes to record, share and supportive stance, e.g. elaborating on ideas and organise ideas, and through their use of Post-It Notes extending metaphors. also develop and extend these ideas. The Post-It Notes 5. Computer as Initiator: The computer develops new help them to think about and manipulate their ideas, perspectives, suggests new directions for ideation and construct semantic relationships that support long- when others have no more to say, and takes term memory [6]. initiative in generative activities. What might a mixed initiative interface that contributes MICIs in Early-Stage Design to these processes be like? It seems probable that The call for participation in this workshop identifies machine learning and natural language processing can procedural content generation for computer games as play a role in making semantic connections between an example of how mixed-initiative interfaces are ideas, and machine vision might track individual Post-It providing creativity support, e.g. [15]. Autodesk’s Notes as they are manipulated through a design Dreamcatcher project [3] also seems to be an example activity. A system that embodied Lawson and Loke’s of human and A.I. in creative collaboration. In [13] learner and informer roles might usefully augment simplistic terms, both these examples are based on a designers’ Post It Note activities, but the question for a human designer setting parameters and an A.I. mixed-initiative interface would remain how and when generating and partially evaluating large numbers of to contribute appropriately. Perhaps this might be digital alternatives before presenting these back to the facilitated by the conventions, rules and structures that human user for further evaluation. In both cases, the human participants typically follow, e.g. when model of creativity is based on searching a possible brainstorming. Might these provide initial guidelines for solution space. Does this represent a limitation in the how a system would make Horvitz’s [10] key decisions, scope of creative applications using mixed-initiative and for selecting which of Allen’s levels [2] is most interfaces? Or do these systems offer an indication of appropriate? When working with stakeholders during co-design contexts. For example, machine learning might be used workshops, we have found that activities such as to extract key moments, or uncover patterns and make making collages from photographs can help them connections between different concepts in design interpret visualized data. These activities encourage conversations. A system that embodies each of Lawson participants to share their experiences and insights, and Loke’s learner, informer and critic roles [13] might and through this explore possible contexts in which be a useful addition to designers’ reflective practice. data were generated [9]. This provides an important source of inspiration to support collaborative ideation. A Engaging with MICIs mixed-initiative system that could work with The tough question for mixed-initiative interaction participants interactively as they explore data would be remains how and when computational systems should extremely interesting to investigate, and search tools interject, engage users, and take initiative. Familiar that utilise analogy or metaphor offer powerful sources instances, such as spellcheck and grammar checking in inspiration e.g. [16]. However, the activities word processing software, struggle to solve this undertaken during co-design workshops typically aim to satisfactorily; and conversational agents can be explore participants’ subjective experiences, and so any frustrating [14]. This is likely to be further complicated system should sensitively draw these out, and be aware in situations where groups of human collaborators of the possibility of priming responses too strongly. interact with ecologies of interactive artifacts and Could a mixed-initiative creative interface also play this intelligent agents. type of role, i.e. computer as facilitator? A survey of UX practitioners working with machine Supporting Reflective Practice learning [7] surfaced a number of challenges designers Our research group also develops tools and investigates face working on the type of systems likely to play a methods to support designers’ reflective practice. For leading role in mixed-initiative creativity support for the example we have investigated how revisiting projects areas discussed here. The danger of systems that to reflect on the way a design space changes increases monitor activity appearing creepy was highlighted as an awareness of the constraints introduced by particular important UX concern, and the probability that systems design choices, qualifies understanding of how design require ground truth from large amounts of data activities filter the design space, and prompts challenged typical approaches to prototyping. Other reconsideration of disregarded opportunities [8]. This difficulties designers raise, which might be indicative of requires detailed design documentation, which can some of the challenges MICIs will face, included: the significantly add to overhead. implication that “learning” means the system and data will change over time, and be dynamic at a large scale; Systems that interactively record design activities, and that statistical correlations lack common-sense, monitor them and learn about what might be can appear simplistic and stupid, and therefore false important, and subsequently prompt designers’ critical negatives or false positives can be hard to assimilate. reflection, could be of great benefit in this and similar The wider issues designers face working with intelligent systems are likely to be increasingly prominent in Design Material. Accepted for 2017 CHI Conference systems that aim for Negotiated Mixed Initiative [2], on Human Factors in Computing Systems. ACM. and where the computer is the Initiator [13]. 8. Graham Dove, Nicolai Brodersen Hansen, & Kim Halskov. 2016. An Argument For Design Space Acknowledgements Reflection. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction, (pp. This research is funded by The Danish Innovation 17-26). ACM. Foundation grant 1311-00001B. CIBIS 9. Graham Dove, & Sara Jones. 2014. Using Information Visualization to Support Creativity in References Service Design Workshops. In ServDes.2014 1. Gregory D. Abowd. "What next, ubicomp?: Service Future, Proceedings of the fourth Service celebrating an intellectual disappearing act." Design and Service Innovation Conference, (pp. In Proceedings of the 2012 ACM Conference on 281-290), Linköping University Electronic Press Ubiquitous Computing, pp. 31-40. ACM, 2012. 10. Eric Horvitz. 1999. Principles of Mixed-Initiative 2. James F Allen. 1999. Mixed Initiative Interaction. User Interfaces. In Proceedings of the SIGCHI In IEEE Intelligent Systems and Their Applications. Conference on Human Factors in Computing 14,5: 14-16 Systems. (CHI 99): 159-166. 3. Autodesk. Project Dreamcatcher. Retrieved January 11. Eric Horvitz. 2007. Reflections on Challenges and 27th 2016 from promises of Mixed-Initiative Interaction. In AI https://autodeskresearch.com/projects/dreamcatch Magazine. 28,2: 19-22 er 12. Paul Keskeys. 2016. How Generative Design Will 4. John Brownlee. 2015. Apple finaly learns AI is the Change Architecture Forever. Architizer. Retrieved new UI. Retrieved January 27th, 2017 from January 26th from http://architizer.com/blog/how- http://www.fastcodesign.com/3047199/apple- generative-design-will-change-architecture-forever finally-learns-ai-is-the-new-ui 13. Bryan Lawson and Shee Ming Loke. 1997. 5. CIBIS. http://cavi.au.dk/projects/creativity-in- Computers, Words and Pictures. In Design Studies, blended-interaction-spaces/ 18,2: 171-183 6. Graham Dove, Sille Julie Jøhnk Abildgaard, Michael 14. Ewa Luger, and Abigail Sellen. 2016. Like having a Mose Biskjær, Nicolai Brodersen Hansen, Bo T. really bad PA. In Proceedings of the 2016 CHI Christensen, & Kim Halskov. 2017. Grouping Notes Conference on Human Factors in Computing Through Nodes: The Functions of Post-It Notes in Systems (CHI’16): 5286-5297 Design Team Cognition. To Appear In Design Studies, Special Issue on Designing in the Wild 15. Gillian Smith, Jim Whitehead and Michael Mateas. (Papers from Design Thinking Research Symposium 2011. Tangara: Reactive Planning and Constraint DTRS11). Elsevier. Solving for Mixed-Initiative Level Design. In IEEE Transactions on Computational Intelligence and AI 7. Graham Dove, Jodi Forlizzi, Kim Halskov, & John in Games 3,3: 201-215. Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a 16. http:/yossarian.co