Sketchplorer: A Mixed-Initiative Tool for Sketching and Exploring Interactive Layout Designs Kashyap Todi1,2 Abstract kashyap.todi@gmail.com This workshop paper discusses our mixed-initiative ap- Daryl Weir1 proach that enables designers to rapidly sketch and explore darylweir@gmail.com interactive layout designs. Although optimisation meth- Antti Oulasvirta1 ods can attack very complex design problems, their insis- antti.oulasvirta@aalto.fi tence on precise objectives and a point optimum is a poor 1 Aalto University, Finland fit with sketching practices. Typical optimisation tools also 2 Hasselt University, Belgium fail to incorporate the human in the loop. Sketchplorer is a mixed-initiative sketching tool that uses a real-time lay- out optimiser. It automatically infers the designer’s task to search for both local improvements to the current design and global (radical) alternatives. Using predictive models of sensorimotor performance and perception, it generates suggestions that interactively steer the designer towards more usable and aesthetic layouts without overriding them or demanding extensive input. While this position paper summarises our work from the mixed-initiative perspective, further details can be found in the original publication [4]. Author Keywords Sketching; Model-based optimisation; Visual Layouts; Mixed- To cite this work, please refer to the original publication [4] in Proceedings of the ACM SIGCHI Conference on Designing Interactive Systems (DIS 2016). Initiative; Design Tools Copyright © 2017 for this paper is held by the author(s). Proceedings of MICI 2017: CHI Workshop on Mixed-Initiative Creative Interfaces. ACM Classification Keywords H.5.2. [Information Interfaces and Presentation]: User Inter- faces Real-Time Optimiser Sketchplorer Design Tool Predictive Models Explore Timeline + Layout Elements Global Canvas Optimiser Search Space Local Optimiser Current Design Design Space Figure 1: Sketchplorer is a mixed-initiative interactive sketching tool that uses a novel approach to real-time layout optimisation. While the designer sketches, the optimiser infers the design task and searches for local improvements (recolour ; fix) and global alternatives (explore). It uses predictive models of performance and perception to generate new designs and suggests these to the designer. Introduction quirements for controlling an optimiser. As the designer Sketching is a powerful visual thinking tool, employed by sketches, the optimiser infers the design task automatically. designers to explore different solutions to a given design It then searches for both local (small changes) and global problem. It is a very fluid process, and tends to be ambigu- (large changes) alternatives. Importantly, our optimiser de- ous and uncertain [2]. Optimisation, on the other hand, can ploys several predictive models of user performance and efficiently attack very complex design problems, and sys- perception, adapted from literature. This allows it to make tematically improve both usability and aesthetics of designs. informed suggestions that “pull” designers toward usable Traditionally, optimisation is a rigid process, and demands and aesthetic designs. precise input. These contradicting requirements inhibit de- signers from employing powerful optimisation techniques Sketchplorer is a mixed-initiative design tool that show- during sketching and ideation phases of design. cases the first investigation of this concept, focusing on interactive layouts familiar from GUIs, web pages, menus, Sketchploration is a novel concept that brings real-time and dialogs. It provides designers with a multitouch envi- optimisation to visual design tools. It couples a designer ronment for sketching, and for exploring interactive layout and an optimiser in a new “serendipitous” way, support- designs generated computationally. ing the natural flow of sketching and ideation. Technically, sketchploration (Figure 1) extends model-based interface optimisation [1, 3] to real-time design exploration under ill-specified and changing design goals. It relaxes the re- Walkthrough: Designing a Blog Page Exploring global alternatives: By abstracting from the This walkthrough illustrates the use of Sketchplorer, our current design, the global optimiser retrieves unique de- design tool based on the sketchploration concept, from a signs, and returns them to the designer. An explore panel, Canvas designer’s perspective. residing on the left edge of the display (Figure 2d), is peri- odically updated with these designs. The designer browses Sketching the initial layout: Sketchplorer initially presents through the list of alternatives, and finds two interesting al- the designer with an empty canvas. The designers starts ternatives. She adds the first to her list of saved designs, sketching by creating a structure for her design (Figure 2a). and drags the second onto the canvas, to continue working Importance Drawn elements can be moved around, or resized, at any Levels on it. time. She ambiguously sketches out boxes, serving as proxies for the elements of her blog page. Sketchplorer in- In a short duration, the designer’s collection of saved lay- fers the design task, and the optimiser starts computing outs is populated with several feasible alternatives—some both local and global suggestions in the background. sketched by the designer, and the others with the aid of the optimiser. While the above phases appear to be linear, in Inspector Refining and adding details: An inspector panel sits on practice, sketching and exploration phases are intertwined. the right-edge of the display, and can be pulled out at any Timeline time. This can be used to specify details such as element Sketchploration: Mixed-Initiative Layout Design type, colour, and importance (Figure 2b). The designer As is evident from the workflow, Sketchplorer relies heavily now takes a moment to add these details to some of the on both the designer and the machine to iterate over layout elements. For instance, she indicates that her page has a designs. In our work, we identified that such layout design Recolour/Fix heading and a paragraph element, and marks them as be- tasks require both—human and machine knowledge—for ing of high importance. Satisfied with the first version of the successful completion of the task in a short timespan. The design, she taps the save (‘+’) button. This adds the current mixed-initiative nature of the tool is critical to its success. Explore design to the designer’s timeline, and provides a preview (Figure 2c). Saved designs can be retrieved, and edited, at Designers are good at quickly sketching out a design task, any later time. and at using their internal design knowledge to identify fea- sible designs. On the other hand, computational methods Making local changes (fix and recolour): While the de- can explore a large number of solutions in a short amount signer sketches and refines, the system continuously streams of time. By employing predictive models of human perfor- the description of the current design to the optimiser. The mance and aesthetics, they can optimise for certain criteria, local optimiser uses the current design as a starting point and objectively improve designs for the user population. to suggest fixes and recoloured designs. Pulsating icons in However, since machines do not have complete knowledge the inspector panel indicate the availability of these sugges- of the design tasks, and might not be aware of aspects Figure 2: A Walkthrough. The tions (Figure 2c). The designer now refers to these sugges- such as semantics of the layout, we rely on designers to designer (a) ambiguously sketches tions, and realises that by using a recolouring suggestion, finally evaluate the suggestions, select feasible solutions, layout designs, and (b) refines and she can make the paragraph of text stand out. She selects and optionally refine them further. By employing this ap- add details. They have immediate access to a timeline of saved this, and continues working on the sketch. designs. The optimiser suggests (c) local (fix and recolour ) changes, and (d) global (explore) design alternatives. proach, we can enable rapid as well as systematic explo- can be selected and manipulated at the same time. This ration of the design space, to arrive at feasible solutions for inferred hierarchy is also essential for the internal repre- the given design task. sentation of a layout in the optimiser. A colour picker allows designers to select the hue–saturation combination, and ad- Sketchplorer divides the task of creating layout designs as just the brightness. Double-tapping on an element reveals follows: an in-place pop-over, and allows adding details without hav- Designer: ing to move to the inspector. An element’s importance can 1. Specify the design task by sketching on the canvas. be adjusted in the inspector panel. Alternatively, an over- 2. Select feasible solutions from a larger set of sugges- lay can be enabled, that displays the importance of every tions. element, and allows batch adjustments (Figure 2b). 3. Further iterate over these suggestions to arrive at one Saved versions and timeline: In Sketchplorer, the current or several final designs. design can be added to the timeline at any time. This time- line provides an overview of all saved designs. Designers Optimiser: can non-linearly continue iterating over any saved design by 1. Interpret the design task from the sketch. dragging them back to the canvas. Thus, the timeline allows 2. Exploit and explore the design space to generate comparing of designs, provides an overview of the evolution suggestions. of designs, and enables iteration over sketches. 3. Periodically return a set of feasible designs that im- Interacting with the optimiser: To the designer, the opti- prove objective aspects of the design. misation appears as a two-pronged approach, consisting of local and global optimisation. The local optimiser ob- To fluidly support these sketching and exploration activities, serves every change in the design. It suggests recolouring Sketchplorer provides designers with a set of interactions, and fixes that maintain the overall composition of the orig- as highlighted in the following section. inal sketch, but improve certain aspects. It also provides harmonious recolouring suggestions, which improve aes- Overview of Interactions thetics and performance. In contrast, the global optimiser Sketchplorer is designed for a multitouch environment using listens exclusively to changes in design tasks, and acts a large display, and uses touch gestures for all controls. upon them. It abstracts away from exact details, allowing it to explore the entire design space, and generate unique Sketching and Refinement: Sketchplorer allows design- designs. It performs exploration in real-time and periodically ers to either sketch ambiguous bounding boxes for layout returns improved results, displayed to the designer in the elements, or pick out a specific element type and place it on expandable explore panel. The global optimiser focuses on the canvas. It takes care to accurately order every element creating unique and improved solutions, not on refining or on the canvas, without designer intervention. Each time an polishing a solution to make them perfect. element is changed, the hierarchy is inferred, and the order- ing dynamically adapted. Hierarchical groups of elements As the optimiser generates these suggestions, they are 1 5 2 revealed to the designer in the tool (Figure 1). Designers To evaluate Sketchplorer as a design tool, we conducted a 3 can cycle through local improvements (recolour or fixes) study with experienced designers, where they were given 4 by tapping on the respective icons. Global improvements sample design tasks, and asked to create multiple feasible (exploratory suggestions) appear as thumbnails, and the designs. We found that designers referred to both local and designer can scroll through them, save select designs to global suggestions while creating there designs. 9 out of Visual clutter the timeline, refine them further on the canvas, or reject 10 designers took advantage of optimised suggestions, 1 2 5 them. to achieve their final designs. We also observed that by 4 using these design suggestions, designers were able to 3 Predictive Models and Objective Function create a larger number of feasible results in a short amount Whereas previous work has used heuristics and focused of time. This is especially encouraging since the purpose of on visual search and motor performance [1, 3], Sketch- sketching designs is to explore the feasible space, to detect plorer addresses spatial and colour aspects of layouts. To a variety of promising solutions. Visual search achieve this, we use the following predictive models: 1. Visual Clutter: Rosenholtz Model Lessons for Mixed-Initiative Creative Interfaces 3 4 Sketchplorer is a system that enables designers to rapidly 2 2. Visual Search: Kieras-Hornof Model 1 generate several design ideas for a given layout design 3. Target Acquisition: Fitts’ Law problem, and employs predictive models and optimisa- 5 4. Grid Quality: Balinsky’s Symmetry Measure tion techniques to assist designers in their task. During 5. Colour Harmony: Cohen’s Harmonic Colour Sets the design and implementation of our system, we learnt Target acquisition some valuable lessons that we feel can benefit other mixed- Figure 3 illustrates optimisation results for individual ob- initiative tools. Some of these are listed as follows: 3 2 jectives, for a sample task with five elements. We define a 1 multi-objective task where we seek to minimise a weighted 1. Minimise Control Requirements: Sketchplorer in- combination of the outputs of these models: 5 4 fers the design task by abstracting from designers’ 5 X concrete sketches. It does not force designers to ex- Grid quality U= wi Si , (1) plicitly specify the task, or additional details, and al- i=1 lows them to focus on the sketching task. Although 4 where the weights wi sum to 1 and the individual objectives additional details can help the optimiser find better Si are normalised to the range [0, 1]. solutions, they distract from the primary objectives of 1 5 the user, and hence are not desirable. 3 Evaluation 2 We evaluated our model-based optimisation approach 2. Appropriate Division of Labour: The system does Colour harmony through a user study, which compared participants’ per- not attempt to replace the designer. Instead, the de- formance with a baseline design to that with an optimised signer and the computer complement each others’ Figure 3: Outcomes when skills. While the designer is good at recognising good designs. Quantitative results favoured our approach. Fur- optimising for a single objective. designs and understanding semantics of a layout, ther details can be found in our full paper. Sketchplorer combines these into multi-objective search. the computer can efficiently generate several good approach to interface design enables the designer and opti- solutions that objectively improve performance and miser to complement each other, to arrive at good solutions aesthetic criteria. Thus, the system rapidly gener- in a short span of time. We have conducted user studies ates several solutions that it deems to be objectively to validate our optimisation technique, and the interactions good, and relies upon the designer to select a subset supported by Sketchplorer. The results from these are pos- of these that semantically make the most sense in itive, but a full discussion is outside the scope of this ab- fulfilling the design requirements. stract, and we invite readers to refer to [4] for further details. 3. Support Local and Global Improvements: Sketch- References plorer attempts to both exploit and explore the design [1] Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sab- space to improve designs. Designers can thus get rina Hoppe. 2013. MenuOptimizer: Interactive Op- support to make minor improvements as well as ma- timization of Menu Systems. In Proceedings of the jor redesigns. This approach can thus be employed 26th Annual ACM Symposium on User Interface Soft- to generate new solutions at early stages, as well as ware and Technology (UIST ’13). ACM, New York, NY, fine-tune existing solutions. USA, 331–342. DOI:http://dx.doi.org/10.1145/ 2501988.2502024 4. Provide Quick and Plenty Solutions: While the de- [2] Bill Buxton. 2007. Sketching User Experiences: Get- signer sketches out an idea, the system continually ting the Design Right and the Right Design. Morgan analyses the sketches and searches for improve- Kaufmann Publishers Inc., San Francisco, CA, USA. ments. Since it does not have a complete under- [3] Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: standing of the design task, it tries to generate sev- Automatically Generating User Interfaces. In Proceed- eral new solutions, and returns them to the designer. ings of the 9th International Conference on Intelligent By presenting them as a non-disruptive overview, it User Interfaces (IUI ’04). ACM, New York, NY, USA, allows designers to quickly glance through them and 93–100. DOI:http://dx.doi.org/10.1145/964442. discard undesirable solutions, and pick out one or 964461 more suitable designs. [4] Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and Explore with a Layout Op- Conclusion timiser. In Proceedings of the 2016 ACM Confer- By integrating real-time optimisation into a sketching tool, ence on Designing Interactive Systems (DIS ’16). sketchploration opens up opportunities for designers to effi- ACM, New York, NY, USA, 543–555. DOI:http: ciently explore alternative designs, and improve the usabil- //dx.doi.org/10.1145/2901790.2901817 ity and aesthetics of outcome designs. Our mixed-initiative