=Paper=
{{Paper
|id=None
|storemode=property
|title=InkChat: a collaboration tool for mathematics
|pdfUrl=https://ceur-ws.org/Vol-1010/paper-14.pdf
|volume=Vol-1010
|dblpUrl=https://dblp.org/rec/conf/mkm/Hu13
}}
==InkChat: a collaboration tool for mathematics==
https://ceur-ws.org/Vol-1010/paper-14.pdf
InkChat: A Collaboration Tool for Mathematics
Rui Hu and Stephen M. Watt
The University of Western Ontario
London Ontario, Canada N6A 5B7
{rhu8,Stephen.Watt}@uwo.ca
Abstract. We investigate the question of how multimodality can be
used in computer-mediated mathematical collaboration. To demonstrate
our ideas, we present InkChat, a whiteboard application, which can be
used to conduct collaborative sessions on a shared canvas. It allows par-
ticipants to use voice and digital ink independently and simultaneously,
which has been found useful in mathematical collaboration.
Keywords: Pen computing, multimodal computing, computer aided
collaboration, InkML
1 Introduction
A number of computer applications have been developed over the past years
to accommodate the needs of collaboration. One general category of these is
whiteboard systems, where multiple users can interact over a shared canvas using
a particular input method. For example, the use of pen input allows participants
to write or draw naturally, which has great potential to increase productivity,
especially in mathematics [1]. Existing systems, however, typically allow only one
input method to be used at one time. For example, in Microsoft OneNote [2], one
can either type or draw, but not simultaneously. This places strong limitations
on what can be done. For example, it becomes quite awkward to explain an
activity while it is being performed.
Mathematical collaboration software can be most useful when it supports
input from multiple modalities, as it allows collaborators to interact more richly
and participants to use the input methods that are most suitable. For example, in
mathematics, some points can be most efficiently made through the spoken word
while others can best be communicated by a hand-drawn diagram or equation.
Software for collaboration also should allow users to perform various editing
operations to revise or tidy up jointly created drawings, text and so on. Finally,
collaborative work usually includes exploring ideas in discussions that can have
false starts and dead ends so the ability to record and roll-back the discussion
to prior points is important.
The primary objective of this article is to analyze the design issues in in-
corporating multimodal interactions in this kind of mathematical collaboration.
Considerable related work has been conducted, some of which we highlight here.
In the 1990s, QuickSet [3] was a multimodal framework used by the US Navy
2 InkChat: A Collaboration Tool for Mathematics
and US Marine Corps to set up training scenarios and to control virtual envi-
ronments. It accepted voice and pen input, communicating via a wireless LAN
through an agent architecture to a number of systems. The system could rec-
ognize voice input with certain responses. If voice interaction was not feasible,
it could still analyze digital ink and then give several possible interpretations.
This demonstrated that multimodal interactions could enable efficient commu-
nication. Classroom 2000 was an application [4] whose primary purpose was
to create an environment to capture as many activities as possible from the
classroom experience. It included tools to automate the production of lecture
notes and to assist students in reliving lectures. This application did not support
real-time distributed collaboration, however. In 2004, the InkBoard [5] network-
shared whiteboard application was released. It allowed graphical collaboration
and design, including network-shared ink strokes and audio/video conferencing.
Because it integrated the Microsoft Conference XP technology, it was limited to
Windows platforms. In the same year, Electronic Chalkboard [6] was developed.
Its goal was to integrate distance education tools with the traditional blackboard
experience. It could load images and interactive programs from a file system or
the Internet and could interact with computer algebra systems and display com-
putation results. Because content was saved as images, it was not possible to
later edit or perform semantic operations on saved sessions.
To explore ideas for mathematical collaboration, Regmi and Watt [7] devel-
oped a whiteboard application that provided collaborative sessions with synchro-
nized voice and digital ink on a shared canvas. This system could save sessions
with the digital ink in InkML [8] format and the voice as an MP3 sound file.
A significant drawback, however, was that the client interface implementation
varied from platform to platform: The client for Windows was implemented in
C#, using the .NET framework, while the client for Linux and Mac OS X was
implemented in Python. Although Python supports cross-platform portability,
the client was constructed using Linux-specific and Mac OS X-specific APIs and
thus could not be ported to other platforms.
To address the portability issue, Hu, Mazalov, and Watt [9] proposed a
streaming digital ink framework for multi-party collaboration. The framework
was portable across multiple platforms and consisted of a number of exten-
sions, These extensions could work independently and simultaneously, serving
as plug-ins for the host collaboration software. It is portable across multiple
platforms including Windows, Linux, and Mac OS X. It currently uses the pop-
ular Skype and Google Talk services as the backbone to deliver data streams,
but other transport mechanisms could be used. The digital ink data is repre-
sented in InkML, allowing flexible manipulations for different content types, such
as mathematics and diagrams. The collaborative sessions can be recorded and
stored for playback, analysis or annotation. InkChat is available for download
at http://www.orcca.on.ca/InkChat/.
The present article explores support for collaboration in this framework. It
is based on the same InkChat software infrastructure as [9], but where that
article focused on the handling of streaming digital ink, we now focus on the
InkChat: A Collaboration Tool for Mathematics 3
multimodal aspects. The remainder of the article is organized as follows. In
Section 2, we examine how multimodal input can best be used in collaborative
environments to improve efficiency. Section 3 recalls InkChat, the whiteboard
application for multimodal collaboration. In Section 4, we explain how InkChat
supports multimodality. Section 5 describes the collaborative aspects of InkChat.
In Section 6, we conclude the article.
2 Multimodal Collaboration
Multimodal input is useful as it provides versatile means for users to interact
with computers. These input modalities include keyboard, mouse, voice, pen,
and video and so on. We will focus on the voice input and pen input in this
section, and explore their capabilities in mathematical collaboration.
Voice Input Voice communication is a fast and natural way to interact with
other people and computers. Most people speak faster than they can type or
manipulate a mouse. Notably, certain people with physical disabilities prefer
operating their computers simply by speaking. Voice input is hands-free, which
is useful if one is driving. Also, voice input is flexible. One does not have to sit in
front of a computer: it is possible to use voice input while active or while sitting,
standing, or reclining. Most modern devices come with built-in microphones.
Handwriting Input With the widespread availability of pen-based devices such
as Tablet PCs, PDAs and even cell phones, pen input starts playing an important
role in human computer interaction. Pen input is a natural and powerful input
modality since everyone learns to write in school. It is versatile as it provides
more gestures and motions available compared to mouse and keyboard input.
Many devices without pens support touch input, which may also be used to
capture handwriting using a finger, but at a lower resolution.
Handwritten input is expressive. Modern devices capture digital ink traces
in a two dimensional writing plane, and may support pressures angles and non-
contact pen height. This may be used to capture mathematics, as most mathe-
matical notations are two dimensional, with similarities to both text and draw-
ing. Mathematical formulae are hard to understand by means of voice, keyboard
or mouse, but can be expressed easily in handwriting.
Voice and Handwriting Multimodal Input We chose voice and pen as
the input modalities for collaboration. Together they provide more than either
individually, and indeed we find in this combination the whole is more than
the sum of its parts. The advantages of voice and pen multimodal collaboration
include:
Portability Most computing platforms support both voice and pen input.
Ease of use Writing and voice are familiar and require little training to use.
4 InkChat: A Collaboration Tool for Mathematics
Fig. 1: InkChat user interface
Complementarity Both input modalities can work independently and simulta-
neously. Speaking and writing at the same time gives two communication chan-
nels, allowing one to be used to explain or amplify the other. Either may give
the main message, with the other supporting it, or both be equally important.
Two channel communication avoids the clutter and confusion that arises when
two related messages are multiplexed through one modality. For example, text
with footnotes or parenthetical remarks or formulae with many annotations.
3 InkChat
As stated earlier, we have developed a platform-independent version of InkChat
to evaluate and demonstrate our ideas. This is built on top of the portable
framework presented in [9], whose primary purpose is to collect digital ink across
a variety of platforms and to provide a platform-independent, consistent interface
for digital ink applications. As a result, InkChat is available on these platforms
and can process ink data without knowing the underlying details. To support
digital ink data portability, InkChat uses InkML to represent digital ink data as
it provides digital ink streaming and archival support independent of platforms.
This allows flexible interchange of digital ink data in collaborative environments
and, in addition, allows cut and paste of digital ink between applications, e.g.
between Microsoft Office 2010 and InkChat.
Figure 1 shows the user interface of InkChat. A number of control buttons are
located at the top of the canvas and grouped together to minimize the distance
pen movement. InkChat provides a set of pre-defined colors and a palette to
create new colors, if needed. To accommodate the needs of different writing
activities, we have developed a few brush types. For example, one can choose
the pen or pencil for diagramming and the tear drop brush for digital painting
or calligraphy. Editing is also supported. This includes redo, undo, and select,
cut, copy, and paste of different kinds of content, such as images, typed text,
and digital ink.
InkChat: A Collaboration Tool for Mathematics 5
4 InkChat Support for Multimodality
InkChat is a multi-year ongoing project. Its primary design objective is to en-
hance mathematical collaboration by incorporating multiple modalities. This
allows participants to flexibly choose the input methods that are most suitable
for a particular topic. Below we describe the modalities that are supported by
InkChat.
Ink Traces Handwriting is one of the most natural ways to input mathematics,
as most mathematical notations are two dimensional, with elements of both
writing and drawing. InkChat captures handwriting as ink traces and exchanges
the data with other clients using InkML.
Voice InkChat also supports voice input. The voice stream is paired with the ink
stream to improve the efficiency of collaboration. For example, one can verbally
explain the underlying meaning of a complex diagram while drawing it on a
shared canvas. This avoids the clutter and confusion that may arise when either
input method is used individually.
Floating Pointer To support collaboration, InkChat provides users with float-
ing pointers that can be used to point at target objects on the shared canvas
without leaving any ink mark. Together with the voice channel, participants can
point to and discuss aspects of the common canvas.
5 InkChat Support for Collaboration
Communication The primary goal of InkChat is to allow users on different
computers to collaborate on a shared canvas. It currently uses the popular Skype
and Google Talk services for the communication channel, but other transport
mechanisms could be used. The primary design principle is to give users the
freedom to choose the communication mechanism without too much configura-
tion. If one service is not available in a particular location, it is easy to switch
to another. Conference mode is supported, where more than two participants
can be involved in one conversation. Depending on the chosen underlying com-
munication service, InkChat adopts different mechanism to exchange data with
other participants. For example, when a P2P backbone is used, the conference is
initiated by the host that has a connection with every other participant. Digital
ink routing shares the same mechanism as audio routing, each piece of ink stroke
will be broadcast by the host to all participants except the initiator.
Page Navigation InkChat also supports page navigation. This has been found
useful when participants wish to cover multiple topics in one session or to load
previous work in the middle of a conversation. In both cases, the current page
will first be saved to the file system as an InkML file. Then the Ink Canvas will
send a page request to the file system to check if the next page already exists. If
so, the Ink Canvas will parse the InkML file and load the content so that users
can continue to work on that page. Otherwise, a blank page will be created.
Figure 2 illustrates the communications used in page navigation.
6 InkChat: A Collaboration Tool for Mathematics
Fig. 2: InkChat page model
Ink Editing In collaboration it is useful to edit or modify a work in progress,
and in order to edit, it is necessary to be able to erase digital ink. InkChat
provides two ways to erase ink: either by erasing whole strokes or parts of strokes.
We call these “stroke-wise” and “point-wise” erasing. Stroke-wise erasing uses
a hit testing method to detect whether a particular stroke is selected. If so,
it removes the stroke from the canvas and re-renders other strokes that may
be affected. Point-wise erasing erases part of a stroke instead of removing the
whole from the canvas. A stroke may be split into pieces when using point-wise
erasing, and this requires the application to detect where the stroke is broken
up. Point-wise erasing uses a hit testing method as well, and this returns a
collection of ink points that need to be removed from the target stroke. It then
groups the remaining ink points into new strokes and calculates the properties
for each, including starting time and duration. The new strokes are then placed
in sequence by starting time.
Drag and Drop InkChat allows existing ink to be moved on the canvas us-
ing drag and drop. This uses a special lasso cursor to select the content to be
moved. This lasso is a free selection tool that allows users to create a selection
by encircling a region with a pen. The Lasso is useful in mathematical domains
as users may often wish to select a portion of a mathematical expression. Figure
3 shows an example of using Lasso. Notably, as the bounding boxes of character
“a” and “+” overlap, a rectangular selection is not suitable for this operation.
Real-Time Mirroring InkChat is able to animate the drawing of ink strokes,
and uses this to render the ink of collaborators as it is being written. To avoid
jarring and distraction that are caused by large visual changes, InkChat splits
long ink traces into small pieces and send each individually. This is to allow
smooth rendering on each participant’s canvas. The representation of digital ink
data is the key to the success of this animation. Collaboration sessions often take
place in heterogeneous environments, where participants may work on different
platforms and use various pen devices. These pen devices typically have different
settings such as sampling rate, sensitivity, channel properties and so on, and
consequently output digital ink data with different characteristics. This requires
digital ink data to be represented in a flexible, platform- and vendor-independent
InkChat: A Collaboration Tool for Mathematics 7
(a) (b)
Fig. 3: An example of using Lasso in InkChat. (a) Selection (b) Drag and Drop
format so that the animation is possible across different platforms. Meanwhile,
ink strokes must be organized in time order in order to support smooth rendering
and synchronization with other modalities.
We have found InkML suitable for these animation purposes. It is platform-
and vendor-independent and allows complete and accurate representation of digi-
tal ink by capturing and recording information such as the device characteristics,
pen tilt, pen pressure and so on. Most importantly, it provides a wide range of
features to support smooth rendering and synchronization.
Session Recording and Playback Collaboration sessions may be recorded
and stored for later playback, analysis or annotation. InkChat stores digital ink
in InkML archival style which keeps the contextual information and ink traces
separately in order to achieve compact representation. When playback is de-
sired, the digital ink data can be efficiently converted into streaming style which
organizes ink strokes along with contextual information in time order [10]. In
addition, each ink trace and its constituent ink points can be timestamped in
order to support accurate synchronization with content input by other modali-
ties. Figure 4 shows an example of playback in InkChat.
6 Conclusion and Future Work
We have shown how InkChat has been made to support multimodal interac-
tion and communication. We have found, informally, that these features greatly
enhance the InkChat’s effectiveness for collaboration. This seems to arise pri-
marily by separating the creation and manipulation of the objects of discourse
(diagrams, equations, and so on) from the discussion about the objects and
the manipulations. It is an ongoing question of investigation to quantify these
findings.
We would like thank Michael Friesen, Vadim Mazalov, Amit Regmi, and
Coby Viner for their contributions to the implementation of InkChat. We would
also like to thank James Wake for investigating how InkChat may be integrated
in other environments, including Google Hangouts.
8 InkChat: A Collaboration Tool for Mathematics
Fig. 4: InkChat animation
References
1. Anthony, L., Yang, J., Koedinger, K.R.: Evaluation of multimodal input for en-
tering mathematical equations on the computer. In: CHI ’05 Extended Abstracts
on Human Factors in Computing Systems. CHI EA ’05, ACM (2005) 1184–1187
2. Microsoft Inc.: Onenote 2010
3. Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L.,
Clow, J.: QuickSet: Multimodal Interaction for Distributed Applications. In: Pro-
ceedings of the fifth ACM international conference on Multimedia. MULTIMEDIA
’97, ACM (1997) 31–40
4. Abowd, G.D., Brotherton, J., Bhalodia, J.: Classroom 2000: A system for capturing
and accessing multimedia classroom experiences. In: CHI’98: CHI 98 conference
summary on Human factors in computing systems, ACM (1998) 20–21
5. Ning, H., Williams, J.R., Slocum, A.H., Sanchez, A.: InkBoard - Tablet PC En-
abled Design Oriented Learning. In: Proc. of the 7th International Conference on
Computers and Advanced Technology in Education. CATE 2004 154–160
6. Friedland, G., Knipping, L., Rojas, R., Tapia, E.: Teaching with an intelligent elec-
tronic chalkboard. In: ETP’04: Proceedings of the 2004 ACM SIGMM workshop
on Effective telepresence, ACM (2004) 16–23
7. Regmi, A., Watt, S.M.: A Collaborative Interface for Multimodal Ink and Audio
Documents. In: Proceedings of the 2009 10th International Conference on Docu-
ment Analysis and Recognition. ICDAR ’09 (2009) 901–905
8. Watt, S.M., Underhill, T. (Editors): Ink Markup Language (InkML) W3C Rec-
ommendation. http://www.w3.org/TR/InkML/ (September 2011)
9. Hu, R., Mazalov, V., Watt, S.M.: A Streaming Digital Ink Framework for Multi-
Party Collaboration. In: Proceedings of the 11th international conference on In-
telligent Computer Mathematics. CICM’12 (2012) 81–95
10. Keshari, B., Watt, S.M.: Streaming-archival inkml conversion. In: Proc. 2007
International Conference on Document Analysis and Recognition, (ICDAR 2007),
Curitiba, Brazil (September 23-26 2007) pp. 1253–1257