We investigate the question of how multimodality can be used in computer-mediated mathematical collaboration. To demonstrate our ideas, we present InkChat, a whiteboard application, which can be used to conduct collaborative sessions on a shared canvas. It allows participants to use voice and digital ink independently and simultaneously, which has been found useful in mathematical collaboration.
A number of computer applications have been developed over the past years
to accommodate the needs of collaboration. One general category of these is
whiteboard systems, where multiple users can interact over a shared canvas using
a particular input method. For example, the use of pen input allows participants
to write or draw naturally, which has great potential to increase productivity,
especially in mathematics [
Mathematical collaboration software can be most useful when it supports input from multiple modalities, as it allows collaborators to interact more richly and participants to use the input methods that are most suitable. For example, in mathematics, some points can be most efficiently made through the spoken word while others can best be communicated by a hand-drawn diagram or equation. Software for collaboration also should allow users to perform various editing operations to revise or tidy up jointly created drawings, text and so on. Finally, collaborative work usually includes exploring ideas in discussions that can have false starts and dead ends so the ability to record and roll-back the discussion to prior points is important.
The primary objective of this article is to analyze the design issues in
incorporating multimodal interactions in this kind of mathematical collaboration.
Considerable related work has been conducted, some of which we highlight here.
In the 1990s, QuickSet [
To explore ideas for mathematical collaboration, Regmi and Watt [
To address the portability issue, Hu, Mazalov, and Watt [
The present article explores support for collaboration in this framework. It
is based on the same InkChat software infrastructure as [
Multimodal input is useful as it provides versatile means for users to interact with computers. These input modalities include keyboard, mouse, voice, pen, and video and so on. We will focus on the voice input and pen input in this section, and explore their capabilities in mathematical collaboration. Voice Input Voice communication is a fast and natural way to interact with other people and computers. Most people speak faster than they can type or manipulate a mouse. Notably, certain people with physical disabilities prefer operating their computers simply by speaking. Voice input is hands-free, which is useful if one is driving. Also, voice input is flexible. One does not have to sit in front of a computer: it is possible to use voice input while active or while sitting, standing, or reclining. Most modern devices come with built-in microphones. Handwriting Input With the widespread availability of pen-based devices such as Tablet PCs, PDAs and even cell phones, pen input starts playing an important role in human computer interaction. Pen input is a natural and powerful input modality since everyone learns to write in school. It is versatile as it provides more gestures and motions available compared to mouse and keyboard input. Many devices without pens support touch input, which may also be used to capture handwriting using a finger, but at a lower resolution.
Handwritten input is expressive. Modern devices capture digital ink traces in a two dimensional writing plane, and may support pressures angles and noncontact pen height. This may be used to capture mathematics, as most mathematical notations are two dimensional, with similarities to both text and drawing. Mathematical formulae are hard to understand by means of voice, keyboard or mouse, but can be expressed easily in handwriting.
Voice and Handwriting Multimodal Input We chose voice and pen as the input modalities for collaboration. Together they provide more than either individually, and indeed we find in this combination the whole is more than the sum of its parts. The advantages of voice and pen multimodal collaboration include: Portability Most computing platforms support both voice and pen input. Ease of use Writing and voice are familiar and require little training to use. Complementarity Both input modalities can work independently and simultaneously. Speaking and writing at the same time gives two communication channels, allowing one to be used to explain or amplify the other. Either may give the main message, with the other supporting it, or both be equally important. Two channel communication avoids the clutter and confusion that arises when two related messages are multiplexed through one modality. For example, text with footnotes or parenthetical remarks or formulae with many annotations. 3
As stated earlier, we have developed a platform-independent version of InkChat
to evaluate and demonstrate our ideas. This is built on top of the portable
framework presented in [
Figure 1 shows the user interface of InkChat. A number of control buttons are located at the top of the canvas and grouped together to minimize the distance pen movement. InkChat provides a set of pre-defined colors and a palette to create new colors, if needed. To accommodate the needs of different writing activities, we have developed a few brush types. For example, one can choose the pen or pencil for diagramming and the tear drop brush for digital painting or calligraphy. Editing is also supported. This includes redo, undo, and select, cut, copy, and paste of different kinds of content, such as images, typed text, and digital ink.
InkChat is a multi-year ongoing project. Its primary design objective is to enhance mathematical collaboration by incorporating multiple modalities. This allows participants to flexibly choose the input methods that are most suitable for a particular topic. Below we describe the modalities that are supported by InkChat.
Ink Traces Handwriting is one of the most natural ways to input mathematics, as most mathematical notations are two dimensional, with elements of both writing and drawing. InkChat captures handwriting as ink traces and exchanges the data with other clients using InkML.
Voice InkChat also supports voice input. The voice stream is paired with the ink stream to improve the efficiency of collaboration. For example, one can verbally explain the underlying meaning of a complex diagram while drawing it on a shared canvas. This avoids the clutter and confusion that may arise when either input method is used individually.
Floating Pointer To support collaboration, InkChat provides users with floating pointers that can be used to point at target objects on the shared canvas without leaving any ink mark. Together with the voice channel, participants can point to and discuss aspects of the common canvas. 5
Communication The primary goal of InkChat is to allow users on different computers to collaborate on a shared canvas. It currently uses the popular Skype and Google Talk services for the communication channel, but other transport mechanisms could be used. The primary design principle is to give users the freedom to choose the communication mechanism without too much configuration. If one service is not available in a particular location, it is easy to switch to another. Conference mode is supported, where more than two participants can be involved in one conversation. Depending on the chosen underlying communication service, InkChat adopts different mechanism to exchange data with other participants. For example, when a P2P backbone is used, the conference is initiated by the host that has a connection with every other participant. Digital ink routing shares the same mechanism as audio routing, each piece of ink stroke will be broadcast by the host to all participants except the initiator. Page Navigation InkChat also supports page navigation. This has been found useful when participants wish to cover multiple topics in one session or to load previous work in the middle of a conversation. In both cases, the current page will first be saved to the file system as an InkML file. Then the Ink Canvas will send a page request to the file system to check if the next page already exists. If so, the Ink Canvas will parse the InkML file and load the content so that users can continue to work on that page. Otherwise, a blank page will be created. Figure 2 illustrates the communications used in page navigation. Ink Editing In collaboration it is useful to edit or modify a work in progress, and in order to edit, it is necessary to be able to erase digital ink. InkChat provides two ways to erase ink: either by erasing whole strokes or parts of strokes. We call these “stroke-wise” and “point-wise” erasing. Stroke-wise erasing uses a hit testing method to detect whether a particular stroke is selected. If so, it removes the stroke from the canvas and re-renders other strokes that may be affected. Point-wise erasing erases part of a stroke instead of removing the whole from the canvas. A stroke may be split into pieces when using point-wise erasing, and this requires the application to detect where the stroke is broken up. Point-wise erasing uses a hit testing method as well, and this returns a collection of ink points that need to be removed from the target stroke. It then groups the remaining ink points into new strokes and calculates the properties for each, including starting time and duration. The new strokes are then placed in sequence by starting time.
Drag and Drop InkChat allows existing ink to be moved on the canvas using drag and drop. This uses a special lasso cursor to select the content to be moved. This lasso is a free selection tool that allows users to create a selection by encircling a region with a pen. The Lasso is useful in mathematical domains as users may often wish to select a portion of a mathematical expression. Figure 3 shows an example of using Lasso. Notably, as the bounding boxes of character “a” and “+” overlap, a rectangular selection is not suitable for this operation. Real-Time Mirroring InkChat is able to animate the drawing of ink strokes, and uses this to render the ink of collaborators as it is being written. To avoid jarring and distraction that are caused by large visual changes, InkChat splits long ink traces into small pieces and send each individually. This is to allow smooth rendering on each participant’s canvas. The representation of digital ink data is the key to the success of this animation. Collaboration sessions often take place in heterogeneous environments, where participants may work on different platforms and use various pen devices. These pen devices typically have different settings such as sampling rate, sensitivity, channel properties and so on, and consequently output digital ink data with different characteristics. This requires digital ink data to be represented in a flexible, platform- and vendor-independent format so that the animation is possible across different platforms. Meanwhile, ink strokes must be organized in time order in order to support smooth rendering and synchronization with other modalities.
We have found InkML suitable for these animation purposes. It is platformand vendor-independent and allows complete and accurate representation of digital ink by capturing and recording information such as the device characteristics, pen tilt, pen pressure and so on. Most importantly, it provides a wide range of features to support smooth rendering and synchronization.
Session Recording and Playback Collaboration sessions may be recorded
and stored for later playback, analysis or annotation. InkChat stores digital ink
in InkML archival style which keeps the contextual information and ink traces
separately in order to achieve compact representation. When playback is
desired, the digital ink data can be efficiently converted into streaming style which
organizes ink strokes along with contextual information in time order [
We have shown how InkChat has been made to support multimodal interaction and communication. We have found, informally, that these features greatly enhance the InkChat’s effectiveness for collaboration. This seems to arise primarily by separating the creation and manipulation of the objects of discourse (diagrams, equations, and so on) from the discussion about the objects and the manipulations. It is an ongoing question of investigation to quantify these findings.
We would like thank Michael Friesen, Vadim Mazalov, Amit Regmi, and Coby Viner for their contributions to the implementation of InkChat. We would also like to thank James Wake for investigating how InkChat may be integrated in other environments, including Google Hangouts.