Intuitive user interfaces need a good understanding of the respective users and practices. One important mathematical practice consists in using mathematical expressions, which evolved as concise and precise representations of mathematical knowledge and as such guide and inform mathematical thinking. We present an exploratory eye-tracking study in which we investigate how humans perceive and understand mathematical expressions. The study confirms that math-oriented and not math-oriented users approach them differently and reveals implicit mathematical practices in the decoding and understanding processes. Further experiments are needed to confirm and study them. We discuss how these practices could be used to improve mathematical user interfaces.
Copyright c by the paper’s authors. Copying permitted for private and academic purposes.
GRAY AND TALL note in [5, p. 1] about mathematical expressions that their “ambiguity of notation allows the successful thinker the flexibility in thought to move between the process to carry out a mathematical task and the concept to be mentally manipulated as part of a wider mental schema”. For example, the very simple mathematical expression “1=2” can be looked at as a the process of dividing 1 by 2 or as the product of this division, i.e., 0:5. Such expressions can be very elaborate and intricate, e.g. look at the complex formula Expression 3 given in Fig. 1. As mental models [8] shape how people interpret and experience the world, they can be very powerful.
Mathematical expressions provide a trained person with a specific mental model. They serve – as mathematical tool-ofthe-trade – the distinct purpose to enable mathematicians to proceptualize, that is to provide focus on either the process (or procedure) of obtaining an information chunk, or on the conceptualization (or the idea) of the information chunk itself; see [5]. ANDRA ET AL. found in [2] that a formula condenses its information in a shorter but more implicit inscription. They note that a “formula constitutes both a procedure and a symbolic narrative” [2, p. 242, original emphasis].
As mathematical expressions can neither be considered text nor image, it was previously suggested that they form a separate category (see e.g., [6]) which humans perceive differently. This perception was coined by ARCAVI “symbol sense”, i.e., a “complex and multifaceted ‘feel’ for symbols [. . . ] a quick or accurate appreciation, understanding, or instinct regarding symbols” [3, p. 31]. In 2005 W. SCHNOTZ presented a study proving that “comprehension is highly dependent on what kind of information is presented and how it is presented” [12, p. 73]. Even though he used only text and image information chunks, it is suspected that mathematical expressions build a category that enables math-oriented persons to understand math in their own way. The “formula shock” [13] effect is well-known, but how exactly do math-oriented people read mathematical expressions differently? If we can get a deeper understanding, then we might get new insights for the design of math search engines.
In [
In [9] KAMALI AND TOMPA studied the retrieval for content in mathematical documents. Their empirical research also indicates that mathematical expressions are special. They conclude that mathematical expressions should neither be used as conventional document fragments nor with a too exact retrieval algorithm as both result in very poor search results. Instead they showed that algorithms that are based on making use of the Content MathML representation of the document corpus fare much better. So we know now that math search research should take the specifics of mathematical expressions into account. But we still don’t know why and how exactly. In our experiments we approached the answer in a very broad and exploratory way, so that we get clues for the answer. Our results should be viewed as hypotheses about the process of understanding mathematical expressions. These need to be confirmed or rejected in future research.
In Section 2 we first describe the set-up of our eye-tracking experiment. Next we present our findings and discuss them to develop first hypotheses for mathematical practices with respect to mathematical expressions. Section 3 concludes the paper. 2
Ultimately, we are interested in the potentially distinct user groups of technical documents in a broad sense and want to create fitting mathUIs. Here, our goal consists in the comprehension of the intuitive use of mathematical expressions. One strength of mathematical expressions is the exact and concise formalization of mathematical information. Therefore we wanted to understand the process of reading and understanding mathematical expressions: How are they read and what are the mathematical intuitions of use? To get a better understanding of the relevance of mathematical expressions and user group behaviour w.r.t. them we set up an eye-tracking study.
With an eye-tracker a subject’s eye movements can be followed and measured while she is performing a task. According to e.g. [7], semantically relevant information increases the number of fixations. Hence, we were interested in the specific spots in a mathematical expression fixated the most or the longest and the order in which this was done by our users. The location where people look at first is the one from which they decode the meaning of the expression. In turn, this might indicate what they are trying to comprehend first and thus, what is the most relevant ‘object’ when searching for the mathematical expression. 2.1
Set-Up We invited 23 participants to look at concrete mathematical expressions, particularly the three expressions seen in Fig. 1 in the order of numbering. The first expression represents a very simple equation system, the second one is slightly more complex as the variables are not on the same side of the equation symbol, and the third one is a rather complex mathematical expression. After Expression 1 and 2 the participants were asked to select the correct answer.
To differentiate between mathematical practice and typical reading behaviour, we invited math-oriented and not math-oriented persons to participate in the test. They were invited via e-mail and were asked to enrol online, not knowing what kind of test to expect. This information was withheld to avoid losing participants that dislike mathematics in general. Before starting the test, the test subjects where asked to indicate their mathematical abilities and interest in mathematics on a range from low, medium and high. Based on this self-assessment and our judgement based on the correctness of their answers given as solutions to math problems in the test the subjects were grouped by “math-affinity” into MATH and NO-MATH participants respectively (see Fig. 2.1). As we were interested in the specific spots in a mathematical expression fixated the most or the longest over time, the visualization of our results in this respect in form of a heatmap or a gaze opacity map seemed the most suitable. Heatmaps as the ones in Fig. 5 were originally used to visualize the heat distribution on a surface for a set period of time, where typically the color “red” indicates the hottest transitioning over yellow, green and light blue to the color “deep blue” representing the coldest area. In an eye-tracking study the longest and most fixated areas are the hottest, the completely ignored spots the coldest. A gaze opacity map, used for example in Fig. 3, works like a mask based on a heatmap: The most looked at areas are the transparent, the ignored ones are the opaque areas of the mask. Therefore, with a gaze opacity map one can only see those areas that the participant group has looked at and thus, seen. Everything else is blacked out. To get a better understanding of the order in which objects in mathematical expression are looked at, we also used the visualization in form of a gazeplot, in which fixations are represented by dots that are connected and numbered according to their accumulated occurrence in a given time frame and whose size indicates the length of the gaze.
Let us first have a look at the result for the very simple Expression 1 in Fig. 1. We gathered the gaze opacity maps after the indicated time for the MATH and NO-MATH participant group in Fig. 3. Human eyes are adapted to fast and efficient information processing. In particular, there are certain visual patterns like vertical lines that are registered by optical cells specialized for the perception of this kind of visual pattern. If nothing strikes the eye, then a typical grasp of the input given starts in the centre. Thus participants in the NO-MATH group began to cope with the input in the center of the mathematical expression as expected. Interestingly, the MATH members did not. Instead they started by looking at the first equation symbol. We hypothesize that
If this could be confirmed, then this might have broad consequences. For instance, the formula shock which many people suffer could be overcome by new visual training methods. Also, math search engines could offer new kinds of query interfaces according to such visual patterns and they could be another facet for sorting search results.
The gaze opacity map taken after 400ms is also revealing. Here, NO-MATH members start to explore the symbols in the mathematical expression near the center, whereas MATH members seem to gather information about the variable name “b” and the operator “+”. This behavior is continued over the next milliseconds. After 1000ms the MATH participants have discovered the last equation symbol, only afterwards they are interested in the actual numbers. In contrast, the NOMATH participants searched for another occurrence of the element “b” and its concrete value “11”. We noted earlier that mathematical expressions have a procedural and a symbolic/calculational aspect at the same time. Using the procedural aspect of the equation system, one has to gather information about the procedural structure first. That is, one observes the kind of mathematical expression first (here after 100ms “equation”), then one looks for the major variable, i.e., the one that is essential to understand the task, and the operator (here after 400ms “b” and “+”), and finally, one is interested in the concrete values (here after 20,000ms “11” and “16”). Note that
Mathematical Practice 2: “The decomposition of a mathematical expression is organized along its procedural character.” The second variable “a” in the second line of the equation system is not looked at at all: soon after 20,000ms all participants of the MATH group progressed to the solution task. We suspect that the math-oriented subjects know that the plus-operator is binary, so the name of the minor variable is not relevant at this point.
Expression 2 was the second mathematical expression shown to our suspects (see Fig. 4) and already more complex than the first one because of its unusual structure. Note that the first clear fixation could already be noted after 100ms in contrast to the first for Expression 1 which took 200ms. Our participants didn’t know before the test what kind of test to expect. We therefore suspect that this time difference of fixations for Expression 1 and 2 is due to an adjustment to the test. Now everyone was aware that some mathematical expression was to be expected.
It is quite stunning, how the strategy of members of the two groups differed from each other for Expression 2 in Fig. 4. If we look at the first two screenshots taken at 100 and 400ms respectively, we notice that the numbers are singled out by the NO-MATH participants whereas the variables were in focus by the MATH members. This means the grasp of these two distinct types of objects (literals and variables) can be easily done as it is the first means to comprehend Expression 2 and maybe generally mathematical expression. So we hypothesize:
Mathematical Practice 3: “Literals and variables can be easily differentiated by most people (with an educated background).” If we look back at our subjects’ eye movements in Expression 1 in Fig. 3, we notice that the MATH members started with focusing on the first equation symbol. According to Math Practice 1 they might do so because they perceive a specific visual pattern. This pattern could be the one of an “equation system”, e.g. three columns with the middle one containing “=”-symbols. Once this is established, as for example can be seen in the screenshot in Fig. 3 at 400ms, they look for the variables (and arguably operators). The interpretation of “b” being perceived as variable in Fig. 3 and not just as surrounding ‘text’ as in the screenshot at 400ms for the NO-MATH users is based on the contemporaneous gaze at the operator symbol. The Expression 2 was shown to participants directly after having been presented Expression 1. Therefore this first phase may have been dropped when reading Expression 2 and the subjects jump right in by looking for variables in Fig. 4. We observe that the variable “a” in Expression 1 is not fixated at the beginning, whereas both variables “x” and “y” are in Expression 2.
Note that the unusual form of Expression 2 makes the variable “y” into a suspected second major variable, whereas the variable “a” in Expression 1 was a minor one in the sense that it was just the other argument for the binary “+” -operator as can be seen in Fig. 1. Not only are variables placeholders for values, their names are also “holding a place” for operator arguments. This specific placeholder role of variable names can be extended to simple mathematical expressions. As this seems to be integrated in the decomposition of mathematical expressions, we hypothesize:
Mathematical Practice 4: “Simple mathematical expressions can be treated as placeholders for argument positions and therefore as neglectable in the mathematical expression decomposition process.” Subexpressions are treated like variables in the perception of mathematical expressions just as if they were placeholders for syntactic constituents that make a formula functionally well-formed.
Now we might ask, why do the MATH members look at the variables first, that is, why is this the second phase in the decomposition of a mathematical expression? We suggest that math-oriented persons want to grasp the task first. So here they make out the “equation system” first, then they try to understand what the distinct elements of the task are. We noticed the priorization with Math Practice 4, but we can also observe that variable names as signifiers for variables are important for the task set-up. We argue that the MATH subjects’ focus on the variables in Fig. 3 can be attributed to their specific placeholder role in a mathematical expression. Thus, as they were singled out so markedly, we like to note:
This placeholder role of variables is already made use of in many math search engines. Concretely, content MathML is used to allow the user to input arbitrary variable names, but search for the generic expression in the underlying corpus. Our hypothesis rather confirms that math-aware search engines need the semantic capture of math knowledge to be intuitive. It is a question whether the exchangeability of variable names as argument placeholders (and not just value placeholders) has yet been considered. We observed that the name is not looked at if the operator is so well known that the existence of this variable can be assumed, which we framed as their characteristic of being “minor” or “major”. This hypothesis may be used to differentiate importance among substitutions in search results.
The difference of accessing the complex mathematical expression between math-oriented and not math-oriented subjects in the respective heatmaps after one minute of watching – presented in Fig. 5 – is striking. The focus on the centre of the expression by NO-MATH members supports Math Practice 1. We suspect that they also didn’t even try to comprehend the formula, so they retreated to answer a question on the meta-level “What kind of mathematical expression is this?” with “an inequality”. This is corroborated by the absence of the concepts “lower/upper bound (of an integral)” in the think-aloud protocol by NO-MATH participants.
Interestingly, neither the integral symbols nor their limits were glanced at by NO-MATH subjects, whereas even the integral symbol itself was fixated by some MATH ones. Note that the precision of the hotspots in the heatmaps for Expression 3 suffered especially as all participants changed their seating position due to the complexity of the mathematical expression. As a consequence the calibration of the eye-tracker to the individual pair of eyes deteriorated accordingly. Assuming that MATH members really looked at the integration symbol itself, we may hypothesize:
Mathematical Practice 6: “In the decomposition of a mathematical expression, some symbols (e.g. the integral sign) carry structural information, which is read independently from its functional information. The structural information does not include parameter values.” Math Practice 5 strengthens this statement, as major variables help to understand the structure, whereas minor ones the function. For math search interfaces this might mean that LATEX is not a superior input language for search queries after all, as it requires syntactic correctness, which according to Math Practice 6 might be too strict.
In Fig. 6 it is obvious that the MATH participants started out from the left to decode the mathematical expression. One could think that this is due to the natural reading direction in Western societies and indeed, the very first deciphering of the mathematical expression has to be expected to be on the very left. But it is not evident that they continued to do so until the pow exp
exp x z1 s
Figure 7: Functional Structure of Expression 3 in Fig. 1 c1 pow first meaningful sub-expression, i.e., the lower limit, was completed (compare with the overview scanning behaviour from left to right by the NO-MATH members). For the MATH participants the entire decoding process starting after the grasp of the first sub-expression contained back and forth jumps to comprehend the intra-relations of the expression.
Moreover, from Fig. 5 we can gather that they looked at each part of the formula separately, whereas the NO-MATH subjects only understood it on the meta-level, that is the mere structure of the formula being an inequality that tells us something about a function a(x). From this finding we can hypothesize that
Mathematical Practice 7: “The decoding of a mathematical expression starts from the left until a first meaningful subexpression is grasped. Further comprehension is chunked into understanding sub-expressions and their relations.” This makes it very probable that they remember the parts in a formula that are located to the left better than ones to the right as these sub-expressions will tend to be the recognition anchors. A potential consequence of this hypothesis consists therefore in the suggestion that the ranking of search results has to include a factor that is concerned with the position of the found fragment where more to the left should rank higher than one to the right. 2.3
Results and Discussion with respect to Proceptualization To assess whether the MATH members scan the mathematical expression using proceptualization, we have to find out whether the participants switch between its procedural and conceptual aspect. Instead of conducting a visual cluster analysis for the data elicited in our study, here we had to turn to a visual inspection of each participant’s data in Expression 3, particularly the order in which he or she deconstructed the complex formula. We exemplify this with one math-oriented subject, but we verified this with all.
Let us first look at the functional structure of the complex formula, that is the content tree. Fig. 7 shows a a tree representation of Expression 3 in Fig. 1, where the tokens (variables and symbols) are at the leaves and the applicative/binding structure is given by internal nodes labeled with “@” (application) and “ ” (binding). The top of the tree is an application of the “ ”-relation to two products (apply “ ” to multiple arguments) and so on down the tree. The subtree rooted and the node “ ” is the representation of the integral R (x) z1s(x) ds. The binding construction has three arguments: a binding operator R (x), the bound variable s and the body z1s(x) .
Now if we superimpose the sequence of fixations identified in our eye-tracking experiment (see Fig. 8), we obtain the situation in Fig. 9 – we restrict our attention on the first/left argument of @ the inequality. 1
We interpret the first two steps (1,2) as formula segmentation process: the c1 @ @ cp1ro(b(axn)d)loo1k)etdoaittslalregfte, laenfdt bthraecnkfeotufnodr othrieenmtaatticohni,nigdreingthi fitebdratchkeefit.rsSttefapcsto3r, pow @ 8 @9 7 2exp 0 moves to the integral symbol and step 4 fixes the lower bound – the upper 3 6 bpaosusneds tdootehsenbootdsyeeomfthtoe irnetceegirvael mbeufcohrebsytetphe6 dMisAcToHvegrrsotuhpa.t tFheinianltleyg,rsatlehpa5s x 1 @ 4 s5 @ the exponential function applied to it. Then the attention shifts to the first R @ / @ s factor again and explores its base and exponent (steps 7 and 8).
This exploration of the left-hand-side of the inequality is followed by an x z1 s orientation towards the right-hand-side via the two “ ” symbols, and then Figure 9: Exploration of Expression 3 an exploration of the right hand side that is very similar to the one detailed in Fig. 9. We observe even though the constant c1 had not been fixated on the left –presumably, since it is very simple, the corresponding c2 was – presumably the reader noticed the contrast. The final part of the formula exploration by the MATH group consisted in a systematic comparison of the left and right sides of the inequality: the fixations jump between the corresponding subtrees (c1 vs. c2, 1 vs. 2, and z1 vs. z2, but (x) used uniformly).
Those patterns of visual exploration can be detected in recordings of multiple MATH participants which supports Math Practice 7 and moreover, leads to
Mathematical Practice 8: “Proceptualization is not only used when using mathematical expressions but also when decoding.”
In contrast to this the NO-MATH subjects explored the inequality in two essentially left-to-right sweeps. The only significant reversals in this are the decoding of the two infix “ ” symbols in the middle (see the second frame on the right in Fig. 8) and some consideration for the large brackets on the left (see the third frame on the right in Fig. 8). After these two sweeps, structured exploration of the inequality seems to cease: the sequencing of fixations seems essentially random. We also observe that the not math-oriented group seems to have less fixations and those that are identified seem less intensive, even though the overall time spent looking at the inequality is the same. 3
Mathematical expressions are widely used in technical documents, therefore fitting mathUIs have to be created that empower their users to better utilise those tools of the trade. With our eye-tracking experiment we have identified a set of mathematical practices when decoding a math expression.
We observed that math-oriented participants have visual patterns for math detection and are able to decompose mathematical expressions with clearly organized procedures. Literals and variables are easily distinguished by most readers, whereas variable names are not the pivotal point essential for problem solving. This might be different with mathematical symbols, that do not necessarily include all parameters of their information. The comprehension of mathematical expression by math-oriented participants is chunked into understanding sub-expressions and their relations. They start decoding the mathematical expression from left to right, until a first meaningful sub-expression is grasped. Those six hypotheses stated, enable a deeper understanding of the way mathematical expression are perceived and show that further research is needed.
This set of math practices is still hypothetical but nevertheless give rise to interesting new perspectives for the design of mathUIs, particularly math search engines. We admit that the observed practices are not yet orthogonal and that we have to refine the primitives which we feel obliged to do in the near future. Moreover, it would be interesting to understand what mathematicians memorize when having looked at mathematical expressions and what parts they indeed recall. Keith Rayner. “Eye Movements in Reading and Information Processing: 20 Years of Research”. English. In: 124.3 (1998), pp. 372–422.
Wolfgang Schnotz. “An Integrated Model of Text and Picture Comprehension”. In: The Cambridge Handbook of Multimedia Learning. Ed. by R. E. Mayer. Cambridge University Press, 2005 (2014), 49–69 (72–103). Alexander Strahl, Julian Grobe, and Rainer M’´uller. “Was schreckt bei Formeln ab? - Untersuchung zur Darstellung von Formeln”. In: PhyDid B - Didaktik der Physik - Beitr’´age zur DPG-Fr’´uhjahrstagung 0.0 (2010). URL: http: //phydid.physik.fu-berlin.de/index.php/phydid-b/article/view/169.