Rapid growth in mobile computing has given rise to a dramatic evolution in communication tools that have the potential to transform the educational experience. Because of the complexities of mathematical communication, however, progress towards the realization of that potential has been particularly slow in the mathematical sciences. We are addressing this by developing iCE (interface for Collaborative Equations), a multimodal mathematical communication environment for postsecondary mathematics and statistics courses. Based on our experiences with this, we discuss the user requirements and di culties in building a universal interface { one that allows users, ranging from mathematical novices to experts, to intuitively communicate mathematics in real-time using a broad range of computing devices.
Mathematical reasoning can rarely sustain itself in the mind alone. Just as
Archimedes drew \geometrical gures in the ashes" [
With the revolution in communication technologies, however, the face of post-secondary education is changing. Flipped classrooms, Massive Open Online Courses (MOOCs), hybrid and fully online courses use the Internet to provide students with access to learning materials from anywhere at any time.
Access to those learning materials is, of course, only one facet of the
learning experience. While great libraries have existed for thousands of years,
classroom education has ourished because of the importance of student-to-teacher,
teacher-to-student and student-to-student communication. For example,
outsidethe-classroom student-teacher contact (e.g., o ce hours, e-mail) correlates
positively with key educational indicators such as academic performance, student
retention, and student satisfaction [
To remain relevant in the distance education market, it is therefore
important for universities to be more than mere content providers { they must embrace
the communication revolution to nd ways to engage geographically-dispersed
students, with greater immediacy and increased e ciency on large MOOC-like
scales. There is evidence that higher education has fallen short in terms of the
use of mobile devices for out-of-classroom contact. A recent study [
Anecdotal evidence suggests that virtual communication in mathematics lags
signi cantly behind that in other disciplines. One reason is likely the lack of
availability of user-friendly math-enabled communication tools. That is to say,
while real-time communication tools such as instant messaging, chat-rooms and
twitter are increasingly used by instructors to enhance the student experience,
those tools are text-based, so their usefulness in mathematics is severely limited.
Yet, the potential bene ts of making mathematics communication more readily
available are clear, for it has been demonstrated that online o ce hours can be
very e ective and e cient in mathematics courses, resulting in high levels of
attendance and student engagement [
In this paper, we will discuss the user-interface challenges in creating a realtime communication environment for mathematics { one that is universal in that it makes casual communication, i.e. without the user needing to pre-install software or learn the interface in advance, between experts and novices alike as easy and intuitive as possible, regardless of whether they are using a more standard computing platform (Windows/Mac/Linux) or a mobile device such as a tablet or smartphone. 2
To achieve the goal of having mathematical formulae be composed and distributed in real-time, there are several issues that must be overcome when designing the user-interface and software.
First, there is the symbol problem. Mathematics makes use of far more symbols than can be conveniently laid out on a keyboard. Expansion of what a keyboard can do through the use of key combinations using the \alt", \ctrl" or \command" keys is di cult to achieve in a platform- or device-independent way (e.g., these key-sequences are not available on mobile platforms). Even where this expansion can occur, its use is di cult for a novice to learn.
Second, there is the layout problem. Many mathematical expressions are written in a 2-dimensional way, with symbols appearing above and below other symbols, and where not all symbols are of the same size. The relative positioning of the symbols is often important: altering those positions can change their meaning in a critical way.
Third, real-time input of mathematics also presents a problem with sequencing. The order in which mathematical symbols are transferred to paper or screen often does not re ect the left-to-right order of the nal expression.
Fourth, while typos in real-time text-based communications are mostly harmless (the reader can typically gure out the intended meaning from the surrounding text) mathematics is a very precise language in which typos lead to errors in mathematical expressions or to confusion among users. 3
Recent years have seen the fragmentation of platform monoculture. There is, for example, a trend in many workplaces away from the support of a corporate standard towards encouraging employees to `bring your own device.' A similar trend can be seen in post-secondary education, where many universities that once provided a standard notebook computer to every student, now facilitated by the advent of campus-wide wi , expect students to bring their own.
A decade ago, it was a safe bet that every student had access to a desktop/notebook running Internet Explorer. Today, the landscape is more varied, with devices, such as smartphones, tablets (some with a stylus; some without) and a resurgence of Macbooks { a wide assortment of available devices that have vastly di erent user interfaces, screen sizes and computational power. With the introduction of Chrome, Firefox and mobile browsers, the market share for Internet Explorer has decreased signi cantly.
For a mathematical communication interface that is meant for novice users { for example, students wishing to only casually drop into an online o ce hour { the requirement of special software or hardware, or the need to download and install a specialized interface, may represent a barrier. A browser-based interface is, therefore, a natural choice, provided it allows for intuitive behaviour on multiple browsers. While there has been a proliferation of computing platforms, many mainstream applications, such as Google Docs and O ce 365, are already browser-based and the web browser is quickly becoming a standard interface for full-featured applications, especially collaborative ones. Rich browser-based applications used to rely on Java or Flash, but these languages are no longer well-supported and are not viable options.
Xpress [
To date, few communication tools have been developed speci cally for mathematics communication. Two freely available examples that we have experience with are MathIM and enVision: MathIM: MathIM is a typical Javascript/HTML-based scrolling chat with the added ability to insert TEX images by including TEX code between $...$ delimiters. There are a small number of template buttons that insert some TEXcode into the chat input line to assist with input. (See Figure 1).
Our experience is that TEX input is very di cult for even advanced mathematics undergraduates and is rarely used by students. Thus students using MathIM tend to write their expressions descriptively using only text, which can lead to many ambiguities. The software also only allows one line at a time, with no ability to edit previous expressions. The ability to interact with and edit expressions would speed up communication (e.g., a professor at a chalkboard erasing symbols that are cancelled in an expression without rewriting it). enVision: enVision is a typical Java-based shared whiteboard application that includes a chat window for textual conversation. Its whiteboard is modelled after a raster-based paint program which would allow users to either draw with a pen or drag-and-drop or insert resizable mathematics symbols anywhere on the shared canvas. An audio communication mode has been produced as a separate add-on. (See Figure 2).
Our experience with this application is that students are able to communicate relatively easily with this type of interface. However, since it is pixel-based, deleting is accomplished by making use of an eraser brush, and symbols on the canvas cannot be easily selected and moved like in a vector-based diagram editor. More importantly, Java is not supported on newer devices, such as tablets, and hence enVision is not a universal option.
Other technologies, including Skype and screen sharing could be used for mathematical communication, but these approaches do not directly solve the problem of joint work on shared equations and do require the pre-installation of software which might represent a barrier for a student who has a simple immediate question. 5
There are four distinct User Interface (UI) models for the creation of equations
on a computer or device:
Structure-based: Graphical structure-based editors are the most common form
of mathematical input UI. These editors typically separate mathematical
structures, such as fractions, from the individual symbol elements. To enter a
complicated expression requires the user to rst set up an appropriate structure of
nested boxes, which may then be populated with numbers or symbols (see Figure
3). These structures can be di cult to navigate and manipulate, and there are
inconsistencies between editors [
Pen-based: A robust pen-based mathematical expression input system that allows one to input expressions as easily as one can on paper could be described as the \holy grail" of mathematical equation input research. Unfortunately, the challenge of recognizing handwritten mathematical expressions is much more complex than that of recognizing handwritten text. While characters in handwritten words have a one-dimensional layout (i.e., are arranged linearly in order), mathematical expressions often have a two-dimensional layout (e.g., fractions, matrices). Mathematical symbols also tend to be very similar, so determining the di erences between 1:2 and 1 2, between < and h, or distinguishing among the symbols ; ; O; o; 0; ; ; ; ; ; ;; ; ; ; ; ; : : : ; is a very di cult task { one of many UI challenges remaining with mathematical pen input.
For the purposes of creating a universal user interface model, pen-input alone
is not a su cient approach. Although pen-based devices have been available since
the 1980s [
Text-based: Mathematical content can certainly be communicated using text
only, but plain text communication is a poor option for mathematics beyond
the simplest expressions. An option that is sometimes used to address this is
TEX, the technical writing language created by Donald Knuth in 1978. TEX was
designed for \the creation of beautiful books" [
Unconstrained Editor: Xpress [
It has been shown that with this unconstrained approach, students tend to
write expressions in the same order that they would on a piece of paper and
with greater speed than with a structure-based editor [
The goal of iCE is to develop a multi-user communication interface that will run on a wide range of platforms and that will allow users with di erent degrees of mathematical experience to communicate with each other in a casual (but accurate) manner. We will consider a mathematical UI model based on an unconstrained approach, such as with enVision/Xpress, but it will include as many elements as possible from other approaches with which a user might be familiar, including pen, structural and TEX.
Until recently, the most common platform for accessing the Internet was a personal computer equipped with a keyboard and mouse (or equivalent pointing device). In recent years, with the advent of mobile computing, touch devices, such as smartphones and tablets, are becoming ubiquitous. These devices typically have small screen sizes and rely on virtual onscreen keyboards. Some smartphones and tablets have a built-in digital pen, but others, such as iOS devices, have no built-in pen. Even when a digital pen is available, these styli are blunt and do not provide a high degree of accuracy on a small screen. Thus, to be usable on all platforms, a universal interface must consider the UI models for keyboard, mouse, pen, and nger touches. There are three basic interactions in the iCE model: a text mode, a symbol mode, and a TEX mode.
Text Mode: As iCE is designed as an extension of the diagram editor UI model, it includes a text mode which can be engaged from the mode selection panel. The text mode allows a text box to be placed anywhere on the screen with text that can assume various font attributes: style, size, italics, bold, colour, etc. It behaves similar to the text mode found for labelling diagrams in a typical diagram editor.
TEX Mode: TEX Mode is similar to Text Mode in that text is entered into a text box. However, in this case, once the text entry is complete, the TEX is compiled into SVG using MathJaX via a web-service call. At this time, the TEX input cannot be edited after it is compiled, but the resulting visual expression can be edited using the iCE drag-and-drop UI model.
Symbol Mode: This UI mode is unique to the iCE/Xpress model. In the iCE model, when in almost any mode, there is a cursor on the screen. Whenever a user clicks or taps on a blank section of a canvas, the symbol mode cursor is placed at that point. Cursor keys also allow the cursor to be moved freely in any direction in fractional steps. Once a key is pressed on the keyboard, that key's corresponding symbol is placed in the cursor location. With repeated presses of the key, however, the placed symbol cycles through several logical alternatives. For example, \<" cycles through \<", \ ", \ ", \\", while pressing \A" cycles through \A," \ ," \8," \^," \@," \\." If the symbol mapping is not intuitively obvious, a symbol could be inserted by clicking on a symbol from a palette.
In addition to these keyboard modes for the shared SVG document, there is also a chat pane. In our model, the shared SVG document is like a physical chalkboard where individuals are engaging, while the chat pane is for the bulk of the conversation. While this is primarily text based, it will support symbol input from palettes as well as TEX for expert users as MathIM does.
Text interactions work well on a desktop or laptop computer. However, touch devices typically have virtual keyboards. While a custom virtual keyboard could solve the symbol problem by o ering alternate key selections that have math symbols, this can be problematic: Android allows the user to install custom keyboards, but iOS does not. However, custom Javascript keyboards can still be implemented in a web browser.
Virtual keyboards also have several drawbacks. First, they can occupy about half of the screen when engaged. This is further compounded by the fact that browser applications are embedded in browsers, which typically have their own UI components that can occupy their own screen space. Second, because precision in determining touch location is low, it is easy to make errors on touch keyboards. Text methods for input on touch devices often include correction { for example, dictionary matching { to increase accuracy. For mathematical expression input, the very broad range of correct mathematical expressions makes robust corrective algorithms extremely di cult to implement. Third, the keyboards can be very distracting as they repeatedly appear/disappear as the user switches between text input and a touch interaction, such as a click or drag.
For touch devices, the iCE model currently uses a combination of OS virtual keyboards (text mode or chat window) and a basic custom javascript virtual keyboard (symbol mode). More research needs to be conducted concerning the best way to handle text input of mathematics on a virtual keyboard. For example, the iOS virtual keyboard is not well suited for inputting TEX, as symbols, such as {, }, $, \, _, are not grouped together or found on the same screen. 6.2
The mouse/pointer interaction in iCE is modelled after that of a standard diagram editor, where objects may be placed anywhere on a canvas, moved and resized. In the case of iCE, instead of objects being restricted to diagram elements, such as boxes and arrows, mathematical symbols may be placed anywhere on the canvas. These symbols may be manipulated as follows: Moving: Any individual symbol may be moved by just dragging it. Selection: Any group of symbols may be selected by dragging out a selection box starting from any blank spot on the canvas (see Figure 5). When completed, any objects falling inside the box will be selected. A selected group can be moved by dragging the group. By clicking/tapping on a blank selection of the canvas, the group is unselected. Currently, iCE implements rectangular selection for all devices, although our tests indicate that, particularly for touch devices, a lassostyle selection of elements, in which users circle the group of elements that they want to select with their nger, may also be an e ective possibility. Resizing: If an object is clicked on or tapped on, resizing widgets appear which, when dragged, resize the object. These resize widgets will automatically be larger on touch devices to account for the lower click/tap precision. (See Figure 6.) 6.3
A selected group of symbols may also be converted into TEX typeset output form for high quality visual display by having the user click on the \convert to TEX" button on the mode panel, using a similar approach to the expression recognition found in Xpress.
iCE also has a free sketch mode, where a user may draw freely in a layer of digital ink. In the future, once a group of pen-drawn symbols is selected, the group might be passed to a handwriting recognition algorithm to have the entire expression to convert into TEX typeset output.
At the moment, expressions are only recognized for presentation structure. In the future, it might be possible to try to recognize mathematical expressions for the purpose of eliciting mathematical content structure. Once recognized, the extracted content could be passed to a Computer Algebra System to assist with intermediary steps in order to speed up communication.
Inline expression recognition could also be valuable for assisting users in forming expressions. In its current form, iCE has a \snap" feature, which, when dropping or resizing symbols, forces the symbols to stick to some invisible lines. As a possible extension, an inline expression recognition algorithm could, as an expression is being created, determine the most likely locations for additional symbols to be appended, and, by way of a magnetic attraction, provide the user with subtle hints for where the additional symbols should be placed. Analogous to the auto-complete feature now becoming ubiquitous in many text applications, these suggestions could then be used or ignored { the user would still be free to place the symbols anywhere on the canvas. This approach can also be thought of as a hybrid between an unconstrained editor like Xpress and a more structurebased editor. 7
In this paper, we have considered a model for real-time communication that combines UI elements from palette-based structural editors, keyboard-based TEX input, whiteboard UIs and text messaging panels. The goal is that each user would be able to use the model or models that best suit their experience and hardware limitations, making communication as easy as possible.
Our experience is that this model works very well on desktops/laptops. The drag-and-drop portion of the model also works well on tablets; virtual keyboard improvements would still need to be explored. Because of its limited screen size, the smartphone provides a challenge for inputting equations quickly and a robust zoom in/out feature will likely have to be developed. However, the high resolution of smartphone screens means this model allows the user to easily view the discussion and provide occasional input. It should also be noted that, if the trend towards increasing smartphone screen sizes continues, these concerns may be mitigated.
When developing a mathematical communication interface, careful consideration must also be given to the issue of con ict resolution. Is the canvas truly shared, as in a free-for-all, or should there be a method of managing which user has the canvas? How should the editing of other users' contributions be managed? What about sequencing { if, say, a user arrives late, can he/she play back a portion of the conversation? These questions, while important, are beyond the scope of this paper.
There is undoubtedly a need for the development of communication environments designed speci cally for mathematics { a task made more di cult by the growing range of user devices. We hope that, by exploring this issue, we are taking the rst steps towards building a universal interface for the real-time communication of mathematics.