Pen and paper is still the best tool for sketching GUIs. However, sketches cannot be executed, at best we have facilitated or animated scenarios. The Advanced User Interaction Environment facilitates turning hand-drawn sketches into executable prototypes. Graphical user interfaces (GUIs) have two important, independent aspects: appearance and a ordances (i.e., visuals vs. behavior). Existing techniques focus mostly on visual appearances, providing tools for a limited scope of visual delity (e.g., from GUI-builders at the high end via drawing tools such as Photoshop, Visio or PowerPoint, to sketching tools like Balsamic at the low end). The simplest possible tool for creating sketches of the visual appearance of a UI is, of course, pen and paper (PaP). It turns out, that PaP is hard to beat in terms of usability and cost/bene t ratio; thus it is our gold standard of drawing. On the other hand, there is the behavioral aspect of GUIs. Most tools completely abstract from this aspect, restricting designers to simple mock-ups of individual scenarios made from hyperlinked pictures, or manually facilitated paper prototypes. A notable exception is Flowella (see http://www.youtube.com/ watch?v=xmuJwKYjiW0). An early approach of combining rough sketches with interactive executability (to a degree) is the Silk/Denim line of work by Landay et al. [3,2], where users would input a sketch with a digital pen. Both Flowella and Silk/Denim are limited to very small UIs as the complexity of designs grows dramatically with the number of behavioral variants and details added. AIDE aspires to overcome this limitation by a number of mechanisms, most notably using hierarchical state machines with concurrent substates, and syntactic layers.1 Also, AIDE allows inputting UI designs by PaP, thus allowing to include diverse audiences in the creation process and reducing overhead and extraneous load caused by inadequate tooling. AIDE allows to complement rough sketches (i.e., PaP input) by more elaborate input in the form of XUL (XML User Interface Language, see https://developer.mozilla.org/en/XUL). PaP and XUL may be mixed freely, allowing scalable delity. AIDE provides the following advantages.
{ Inclusive Technology We all learn to use pen and paper from early
childhood, so it is safe to assume everyone has su cient dexterity in sketching;
artistic skills are not required. In contrast, operating a computer-based tool
often demands substantial expertise and/or training, which end users and
domain experts may lack. Thus, using simple pen-and-paper sketches as AIDE
does allows us to include virtually everybody in the UI design process.
{ Continuous Work ow Graphic designers appreciate sketching tools: their
low viscosity makes them ideal for exploring the design space (cf. [
Consider an example Fig. 1. It shows a small portion of an interface design from a teaching example, the Library Management System (LMS). This toy case study is about searching for media in a library, lending them, prolonging, them, and and so on. Fig. 1 shows an interface design for the process of issuing a new reader card. The numbers in red dots are not part of the notation but have been added to improve the presentation here.
First, a dialog for entering some data appears. It contains a text eld, two groups of radio buttons with two choices each, and two buttons to proceed and abort. The user inputs a reader's name, selects a few options, and eventually A Sketch & Explore create initial design as a pen-and-paper storyboard explore design space by e.g. group dynamic exercise elicit interface requirements by e.g. task analysis or reverse engineering WED LMS: Issue New Reader Card
IssueNewReaderCard Reader Proceed
FirstCard ReplacementCard Pickupatlibrary
Mailtohomeaddress H* Abort pressed \Proceed". If the data validation is successful, the user must acknowledges or aborts. If the process is aborted or the validation was not successful, the user may either revise the inputs or terminate the whole process. Using the right mouse button on the window \Issue New Reader Card" will open a pop-up menu with four options.
States UI widgets like text boxes or buttons are represented as simple states. Groups of widgets and complete dialogs are modeled as composite states. As a default, only one widget or dialog can be active at any time, which in UML maps to sequential composite states. In order to achieve di erent behavior, concurrent composite may be used. Not all states need be visible in a design. For instance, grouping radio buttons together can be achieved by an invisible compound state. The same applies for layout elements such as vertical boxes.
Triggers Positioning a pointer over a WED element is interpreted as putting the focus on that element. Technically, the corresponding state con guration tree is activated. Any user events issued subsequently will be interpreted by that tree, bottom up. For instance, positioning the mouse pointer over the \Abort" button and pushing the left mouse button (a) issues the event single left click in the state \IssueNewReaderCard.Abort" and triggers the transition emanating that state. Likewise, moving the pointer over \Issue New Reader Card" and pressing the Escape key (b) will reset the corresponding state. Any (user) actions the UI a ords may be used as triggers.
Guards Transitions may carry a guard that enforces the respective condition to be true before the transition is taken. Guards may refer to environment variables that may be used to represent a hidden UI state such as a mode.
E ects Then, the e ect of the transition (modeling a UI action) is executed and its target state is entered. E ects may be described by plain text (a), code snippets, invocation of library functions, or maybe visualized by an icon (b). Probably the most common e ects are opening a new window (a), closing one (b), or opening a modal dialog (c).
Entering States When entering a composite state C for the rst time, the substate to be entered is determined by the initial state. Reentering C will reset its state con guration unless a history state is added to C, which restores the state con guration in e ect at the time of exiting C. Exit Points (and and Entry Points) help achieving a modular design (this is regular UML 2.2 syntax). Exiting a state (or state machine) automatically exits all sub states, i.e., corresponding windows are closed by reaching a nal node.
Secondary Notation Annotations and comment boxes may be used freely; they are represented as UML PseudoStates.
There is no semantic di erence between UML 2 state machines and a UI design in this form: every construct in a UI design may be mapped to a UML state machine construct. These mappings are typically very straightforward, but have to be added manually in the current version of AIDE.
SM
LMS: Issue New Reader Card
Issue New Reader Card
key(tab) key(tab)
key(tab) key(tab)
key(esc) mouse(l) [credentials ok] /spawnWindow()
mouse(l) / switchWindow()
Confirm [credentials not ok] /spawnWindow() Error: Card Can Not Be Issued mouse(l) / spawnWindow()
mouse(l) / issueRC() mouse(l) /beep() mouse(l) / closeWindow() mouse(r) / openModalWindow() mouse(l) / closeModalWindow()
Fig. 2. This UML state machine is yielded by stripping all appearance cues and elaborating e ects to procedure calls. The Advanced Interaction Design Environment (AIDE) is a highly modular platform independent direct interaction tool set for creating WEDs, re ning and elaborating them in a methodical fashion, supporting distributed concurrent group work, and generating working prototypes from WEDs. AIDE has been under development since 2006, with 25 students at Innsbruck University, Munich University, and the Technical University of Denmark contributing a total of approximately 10,000 work hours to it. Step B in Fig. 1 is actually a screenshot of the AIDE tool.
AIDE is created using pure Java, using Piccolo2D, jEdit, JGoodies Looks, the Tango Iconset, VLDocking, and JAXB for persistency. AIDE follows a strict separation of logic and presentation. Extensibility of AIDE is ensured by a cartridge mechanism that encapsulates the visual appearance of elements and functions associated with them. Cartridges may be dynamically loaded or unloaded. Apart from the basic cartridges of UML state machines and annotations, there are currently cartridges for the XML User Interface Language (XUL), and for importing hand-drawn storyboards. XUL is used e.g., for de ning the UIs of Mozilla projects such as Firefox and Thunderbird. AIDE provides XUL export and integrated simulation. Finally, AIDE also o ers unbounded Undo/Redo, user de nable roll-back points, tear-o -menus to support very large screens, a locator map, sophisticated zoom and scrolling functions, and multiple views on elements.