Multitouch-based Annotation Boards for Face-to-Face Discussions

Abstract

This paper presents a hybrid approach to annotations for face-toface settings. Firstly, multitouch-based cooperative interaction for large displays is used to support group discussions with many different structures. Secondly, augmented notes are used to integrate paper-based notes in electronic structures.

Keywords

• eol>Multitouch
• augmented notes
• annotation
• discussion
• cooperative interaction

1. INTRODUCTION

Annotations are a very useful means to support knowledge building processes. For example, scribbling, adding comments and references, or highlighting media are very useful annotation mechanisms supporting discussions. Many annotation approaches apply the paper-and-pen paradigm to the digital world, which can result in some shortcomings [ 6 ]. For example, multiple annotation systems are used for different document formats or media, such as paper-based documents, PDF files, or documents of a word processing system. Another example for such shortcomings is missing functionality to capture relations between documents to reflect lateral reading. A third example is very limited sharing capabilities of annotations among a group of users, in particular with respect to paper-based documents. A repository for different document formats and annotations on them is proposed, which gets replicated to cooperating users. In this way, diverse users could get access to different annotations in specific documents [ 6 ].

Ted Nelson often argues that due to the simulation of paper in the electronic world, structure is limited to hierarchy [ 10 ]. He usually argues for more sophisticated structures in the electronic world, such as ZigZag® [ 9 ].

This paper presents a hybrid approach to annotations, which focuses on face-to-face cooperation and combines multitouch-based cooperative interaction for large advanced interactive displays with Augmented notes that integrate paper-based notes in electronic structures.

With advanced interactive displays we mean display hardware, which supports multitouch technology and delivers good responsiveness and scalability. The number of supported touchpoints as well as the quality of its responsiveness and scalability shall be based on the number of users who can physically interact with the display at the same time.

In the following, this paper presents related work and describes the proposed approach in the context of different application scenarios.

2. RELATED WORK

Touch and tabletop technology has been proposed for the application areas education, e.g. [ 4 ], computer games, e.g. [8], and art design, e.g. [ 11 ]. Compared to traditional user interfaces richer user experience is pointed out. Another trend is to use surface technologies for analysis work [ 2 ]. Visualization techniques and exploring huge amounts of data are in the focus. The annotation systems used in schools are often limited. Just one user at one point in time can interact with the device. In recent years large displays providing multitouch functionality have become affordable so that more solutions providing simultaneous interaction should become available.

In the Hypertext community, tools for making notes, structuring and sharing ideas have been worked on in several directions. In NoteCards [ 5 ], for example, notes are represented as cards holding text and images, and being interconnected with typed links. Fileboxes are a kind of composite notes useful to organize large collections of notecards. In gIBIS [ 3 ], for instance, hypertext networks based on the Issue-Based Information System (IBIS) approach can be created in a group. IBIS is an argumentation-based approach focusing on problem solving. Cooperative hypermedia approaches, such as in [ 13 ], focus on collaboration support – both asynchronously as well as synchronously with fine-grained notifications of other users’ interactions. In the context of hypertext narrative there is Tinderbox [ 1 ], for example. Tinderbox is a tool for making, analyzing, and sharing notes focusing on spatial hypertext, informal semantics, and web collage. All these tools use the strength of hypertext to make information more structured and meaningful. The Digital Desk [ 14 ] is early work to integrate paper on a physical desk with the virtual world.

In this article, we combine a multitouch-based cooperative annotation board with paper-based notes.

3. MULTITOUCH-BASED ANNOTATION

BOARDS

A multitouch-based annotation board, proposed in this article, supports the annotation of different document formats and media either individually or in a group. For the latter, we are currently focusing on face-to-face settings. We are using widespread gestures, known from smartphones and tablets, and apply them to the simultaneous annotation and discussion of different media, such as text, images, or video, on large displays. Gestures support moving, resizing, and rotating an artifact, and can also be executed by multiple users, e. g. for enlarging a document. Users can show documents to each other, annotate them cooperatively, and store the discussion results in group-related knowledge repositories. Onscreen keyboards are provided to edit text simultaneously.

In addition, this approach utilizes augmented notes [ 12 ] by integrating a camera in the environment. Figure 1 shows a camera setup, which we are currently using above a multitouch table in order to take pictures from the display triggered by a specific gesture. In this way, a paper-based document can be put on the table and automatically integrated in the electronic hypermedia. The camera is connected to the wireless LAN and runs the Android operating system. Through a specifically developed mobile app we can remotely control the camera. Currently, we are experimenting with optical character recognition to identify text. If the handwritten text cannot be recognized, we integrate the electronic version of the document as an image.

We have developed multitouch-based solutions for supporting different usage scenarios. Two of them are described in the following.

3.1 Interactive and Cooperative Learning

In our university, we are using several multitouch-based applications to get more attention from students in exercises as well as to support self-learning. Students can individually navigate through learning material or cooperatively in groups using larger multitouch-enabled displays. They can use intuitive gestures and annotation functionality, such as scribbling or highlighting. In this way, learning material can be discussed effectively. Annotated artifacts can be saved and linked in the learning process. Figure 2 shows an individual student using and annotating learning material on a mid-size multitouch-enabled display. 3.2 Industry 4.0 In the context of the German manufacturing industry new developments towards “Industry 4.0” are carried out [ 7 ]. We are currently discussing innovative control panels for the manufacturing industry, which visualize data mining results from analyzing production data. The data analysis focuses on identifying causes for quality problems. The visualization and interaction focuses on multitouch-based annotation boards. We discuss the usage of multitouch screens in order to support problem solving in a group. Incident management is handled similar to case management. Information and knowledge about the current situation is visualized and can be interacted with and explored further cooperatively.

Figure 5 shows a current prototype for this application scenario. Reports are visualized as interactive documents, which can be annotated cooperatively to support discussions.

4. CONCLUSIONS AND FUTURE WORK

This paper has proposed a hybrid approach to annotations for face-to-face settings. Multitouch-based cooperative interaction for large displays is combined with augmented notes. In this way, both paradigms – rich hypermedia-based structures as well as paper-based notes and annotations – can be used and integrated in the knowledge media. The feedback from our students was very positive. In particular, the cooperative multitouch-based learning has been considered useful to support the knowledge building process. In our future work, we will do more evaluations and continue on the “Industrie 4.0” scenario.

5. ACKNOWLEDGMENTS

Thanks to Benjamin Lietzau for implementing cooperative multitouch-based prototype solutions. different the 1st international conference on Tangible and embedded interaction, 87-90.

References

[1] Bernstein , M. 2003 : Collage, Composites, Construction. In: Proceedings of Hypertext'03 , ACM Press.

[2] Brown , J. , Wilson, J. , Gossage , S. , Hack , C. , and Biddle , R. 2013 : Surface Computing and Collaborative Analysis Work . Morgan & Claypool Publishers.

[3] Conklin , J. , and Begemann , M.L. gIBIS 1988 : A Hypertext Tool for Exploratory Policy Discussion . In: Proceedings of Computer Supported Cooperative Work (CSCW) , ACM Press, pp. 140 - 152 .

[4] Döring , T. , and Beckhaus , S. 2007 : The card box at hand: exploring the potentials of a paper-based tangible interface for education and research in art history . In Proceedings of

[5] Halasz , F.G. 2001 : Reflections on NoteCards: seven issues for the next generation of hypermedia systems . In: ACM Journal on Computer Documentation (New York, NY, USA: ACM Press) 25 ( 3 ), pp. 71 - 87 . doi: 10 .1145/507317.507321, reprint.

[6] Hoff , C. and Rothkugel , S. 2008 : Shortcomings in Computerbased Annotation Systems . In: Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, pp. 3715 - 3720 , ISBN 978-1- 880094 -66-2.

[7] Kagermann , H. , Wahlster , W. , and Helbig , J., Eds. 2013 : “Securing the Future of German Manufacturing Industry: Recommendations for Implementing the Strategic Initiative INDUSTRIE 4.0,” Final Report of the Industrie 4 .0 Working Group. Forschungsunion im Stifterverband für die Deutsche Wirtschaft e .V., Berlin.

2004: Towards the next generation of tabletop gaming experiences . In Proceedings of Graphics Interface , 73 - 80 .

[9] Nelson , T.H. 2001 : Tech Briefung: ZigZag . In: Proceedings of Hypertext 2001 , p. 261 - 262 , ACM Press.

[10] Nelson , T.H. 2003 : Structure, Tradition and Possibility (Keynote) . In: Proceedings of Hypertext 2003 , p. 1 , ACM Press.

[11] Patten , J. , Recht , B. , and Ishii , H. 2006 : Interaction techniques for musical performance with tabletop tangible interfaces . In Proceedings of the int. conference on Advances in computer entertainment technology, Article No. 27.

[12] Rubart , J. 2015 : Augmented Notes: Supporting Paper-based Notes on Multitouch Tabletops . In: SIGWEB Newsletter, Summer 2015 , ACM Press.

[13] Rubart ; J., Haake , J.M. , Titze , D.A. , and Wang , W. 2001 : Organizing Shared Enterprise Workspaces Using Component-Based Cooperative Hypermedia . In: Proceedings of Hypertext'01 , ACM Press.

[14] Wellner , P. Interacting with Paper on the Digital Desk 1993 . In: Communications of the ACM , 36 ( 7 ), ACM Press, pp. 86 - 96 .