Multitouch-based Annotation Boards for Face-to-Face Discussions Jessica Rubart OWL University of Applied Sciences An der Wilhelmshöhe 44 37671 Höxter, Germany jessica.rubart@hs-owl.de ABSTRACT This paper presents a hybrid approach to annotations for face-to- This paper presents a hybrid approach to annotations, which face settings. Firstly, multitouch-based cooperative interaction for focuses on face-to-face cooperation and large displays is used to support group discussions with many different structures. Secondly, augmented notes are used to  combines multitouch-based cooperative interaction for integrate paper-based notes in electronic structures. large advanced interactive displays with CCS Concepts  Augmented notes that integrate paper-based notes in • Human-centered computing ➝ Human computer interaction electronic structures. (HCI) ➝ Interaction paradigms; • Human-centered computing With advanced interactive displays we mean display hardware, ➝ Human computer interaction (HCI) ➝ Interaction devices; which supports multitouch technology and delivers good • Human-centered computing ➝ Collaborative and social responsiveness and scalability. The number of supported computing ➝ Computer supported cooperative work; • Applied touchpoints as well as the quality of its responsiveness and computing ➝ Document management and text processing ➝ scalability shall be based on the number of users who can Document preparation ➝ Annotation; physically interact with the display at the same time. In the following, this paper presents related work and describes Keywords the proposed approach in the context of different application Multitouch; augmented notes; annotation; discussion; cooperative scenarios. interaction 2. RELATED WORK 1. INTRODUCTION Touch and tabletop technology has been proposed for the Annotations are a very useful means to support knowledge application areas education, e.g. [4], computer games, e.g. [8], and building processes. For example, scribbling, adding comments art design, e.g. [11]. Compared to traditional user interfaces and references, or highlighting media are very useful annotation richer user experience is pointed out. Another trend is to use mechanisms supporting discussions. Many annotation approaches surface technologies for analysis work [2]. Visualization apply the paper-and-pen paradigm to the digital world, which can techniques and exploring huge amounts of data are in the focus. result in some shortcomings [6]. For example, multiple annotation systems are used for different document formats or The annotation systems used in schools are often limited. Just one media, such as paper-based documents, PDF files, or documents user at one point in time can interact with the device. In recent of a word processing system. Another example for such years large displays providing multitouch functionality have shortcomings is missing functionality to capture relations between become affordable so that more solutions providing simultaneous documents to reflect lateral reading. A third example is very interaction should become available. limited sharing capabilities of annotations among a group of users, In the Hypertext community, tools for making notes, structuring in particular with respect to paper-based documents. A repository and sharing ideas have been worked on in several directions. In for different document formats and annotations on them is NoteCards [5], for example, notes are represented as cards proposed, which gets replicated to cooperating users. In this way, holding text and images, and being interconnected with typed diverse users could get access to different annotations in specific links. Fileboxes are a kind of composite notes useful to organize documents [6]. large collections of notecards. In gIBIS [3], for instance, Ted Nelson often argues that due to the simulation of paper in the hypertext networks based on the Issue-Based Information System electronic world, structure is limited to hierarchy [10]. He usually (IBIS) approach can be created in a group. IBIS is an argues for more sophisticated structures in the electronic world, argumentation-based approach focusing on problem solving. such as ZigZag® [9]. Cooperative hypermedia approaches, such as in [13], focus on collaboration support – both asynchronously as well as synchronously with fine-grained notifications of other users’ interactions. In the context of hypertext narrative there is Workshop “The Future of Computer Annotation” at the ACM Tinderbox [1], for example. Tinderbox is a tool for making, conference on Hypertext and Social Media, July 10, 2016, Halifax, analyzing, and sharing notes focusing on spatial hypertext, Canada informal semantics, and web collage. All these tools use the Copyright is held by the owner/author(s). strength of hypertext to make information more structured and meaningful. The Digital Desk [14] is early work to integrate paper on a physical desk with the virtual world. In this article, we combine a multitouch-based cooperative annotation board with paper-based notes. 3. MULTITOUCH-BASED ANNOTATION BOARDS A multitouch-based annotation board, proposed in this article, supports the annotation of different document formats and media either individually or in a group. For the latter, we are currently focusing on face-to-face settings. We are using widespread gestures, known from smartphones and tablets, and apply them to the simultaneous annotation and discussion of different media, such as text, images, or video, on large displays. Gestures support moving, resizing, and rotating an artifact, and can also be executed by multiple users, e. g. for enlarging a document. Users Figure 2: Individual Multitouch-based Learning can show documents to each other, annotate them cooperatively, Figure 3 shows a group of students annotating and discussing and store the discussion results in group-related knowledge learning material on a multitouch table. repositories. Onscreen keyboards are provided to edit text simultaneously. In addition, this approach utilizes augmented notes [12] by integrating a camera in the environment. Figure 1 shows a camera setup, which we are currently using above a multitouch table in order to take pictures from the display triggered by a specific gesture. In this way, a paper-based document can be put on the table and automatically integrated in the electronic hypermedia. Figure 3: Cooperative Multitouch-based Learning Figure 4 shows an augmented note on a multitouch table. By a specific gesture the camera is triggered and an electronic version of the note is integrated in the learning process. Figure 1: Camera Setup for Multitouch Table The camera is connected to the wireless LAN and runs the Android operating system. Through a specifically developed mobile app we can remotely control the camera. Currently, we are experimenting with optical character recognition to identify text. If the handwritten text cannot be recognized, we integrate the electronic version of the document as an image. We have developed multitouch-based solutions for supporting Figure 4: Augmented Note different usage scenarios. Two of them are described in the following. 3.2 Industry 4.0 In the context of the German manufacturing industry new 3.1 Interactive and Cooperative Learning developments towards “Industry 4.0” are carried out [7]. We are In our university, we are using several multitouch-based currently discussing innovative control panels for the applications to get more attention from students in exercises as manufacturing industry, which visualize data mining results from well as to support self-learning. Students can individually analyzing production data. The data analysis focuses on navigate through learning material or cooperatively in groups identifying causes for quality problems. The visualization and using larger multitouch-enabled displays. They can use intuitive interaction focuses on multitouch-based annotation boards. We gestures and annotation functionality, such as scribbling or discuss the usage of multitouch screens in order to support highlighting. In this way, learning material can be discussed problem solving in a group. Incident management is handled effectively. Annotated artifacts can be saved and linked in the similar to case management. Information and knowledge about learning process. Figure 2 shows an individual student using and the current situation is visualized and can be interacted with and annotating learning material on a mid-size multitouch-enabled explored further cooperatively. display. Figure 5 shows a current prototype for this application scenario. the 1st international conference on Tangible and embedded Reports are visualized as interactive documents, which can be interaction, 87-90. annotated cooperatively to support discussions. [5] Halasz, F.G. 2001: Reflections on NoteCards: seven issues for the next generation of hypermedia systems. In: ACM Journal on Computer Documentation (New York, NY, USA: ACM Press) 25 (3), pp. 71–87. doi:10.1145/507317.507321, reprint. [6] Hoff, C. and Rothkugel, S. 2008: Shortcomings in Computer- based Annotation Systems. In: Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, pp. 3715-3720, ISBN 978-1-880094-66-2. [7] Kagermann, H., Wahlster, W., and Helbig, J., Eds. 2013: “Securing the Future of German Manufacturing Industry: Recommendations for Implementing the Strategic Initiative Figure 5: Prototype for "Industrie 4.0" INDUSTRIE 4.0,” Final Report of the Industrie 4.0 Working Group. Forschungsunion im Stifterverband für die Deutsche Wirtschaft e.V., Berlin. 4. CONCLUSIONS AND FUTURE WORK [8] Magerkurth, C., Memisoglu, M., Engelke, T., and Streitz, N. This paper has proposed a hybrid approach to annotations for 2004: Towards the next generation of tabletop gaming face-to-face settings. Multitouch-based cooperative interaction experiences. In Proceedings of Graphics Interface, 73-80. for large displays is combined with augmented notes. In this way, both paradigms – rich hypermedia-based structures as well as [9] Nelson, T.H. 2001: Tech Briefung: ZigZag. In: Proceedings paper-based notes and annotations – can be used and integrated in of Hypertext 2001, p. 261-262, ACM Press. the knowledge media. The feedback from our students was very [10] Nelson, T.H. 2003: Structure, Tradition and Possibility positive. In particular, the cooperative multitouch-based learning (Keynote). In: Proceedings of Hypertext 2003, p. 1, ACM has been considered useful to support the knowledge building Press. process. In our future work, we will do more evaluations and continue on the “Industrie 4.0” scenario. [11] Patten, J., Recht, B., and Ishii, H. 2006: Interaction techniques for musical performance with tabletop tangible 5. ACKNOWLEDGMENTS interfaces. In Proceedings of the int. conference on Advances Thanks to Benjamin Lietzau for implementing different in computer entertainment technology, Article No. 27. cooperative multitouch-based prototype solutions. [12] Rubart, J. 2015: Augmented Notes: Supporting Paper-based Notes on Multitouch Tabletops. In: SIGWEB Newsletter, 6. REFERENCES Summer 2015, ACM Press. [1] Bernstein, M. 2003: Collage, Composites, Construction. In: Proceedings of Hypertext’03, ACM Press. [13] Rubart; J., Haake, J.M., Titze, D.A., and Wang, W. 2001: Organizing Shared Enterprise Workspaces Using [2] Brown, J., Wilson, J., Gossage, S., Hack, C., and Biddle, R. Component-Based Cooperative Hypermedia. In: Proceedings 2013: Surface Computing and Collaborative Analysis Work. of Hypertext’01, ACM Press. Morgan & Claypool Publishers. [14] Wellner, P. Interacting with Paper on the Digital Desk 1993. [3] Conklin, J., and Begemann, M.L. gIBIS 1988: A Hypertext In: Communications of the ACM, 36 (7), ACM Press, pp. 86- Tool for Exploratory Policy Discussion. In: Proceedings of 96. Computer Supported Cooperative Work (CSCW), ACM Press, pp. 140-152. [4] Döring, T., and Beckhaus, S. 2007: The card box at hand: exploring the potentials of a paper-based tangible interface for education and research in art history. In Proceedings of