An infra-red finger tracking system used in the assessment and remediation of “graph-as-picture” misconceptions

Abstract

The workshop presentation will describe a specialized application of Lee's “Wiimote Whiteboard” [7] an infra-red camera based tracking system which uses the Nintendo Wii wireless remote control unit and Bluetooth. Young students wear a very small infra-red LED on their index finger with a forefinger/thumb operated micro-switch for producing “mouse clicks”. This system is combined with a vertically mounted data projector or a horizontally mounted regular computer LCD display, creating a cost-effective large interactive touch surface. The system has a fast response time and has been used with primary school students in diagrammatic knowledge (graphicacy) assessment [4] and in interactive dynalinked diagrammatic applications [5]. These applications were designed to investigate the “graph-as-picture” misconception and they will be described and demonstrated at the workshop.

Keywords

• interactive environments
• graphicacy
• touch-screen technology

-

For this workshop presentation, we will demonstrate how Lee’s [ 7 ] “Wiimote Whiteboard” (an infra-red tracking system) was adapted to create a very affordable and fast-responding interactive surface from a data-projector’s image or a horizontally mounted regular computer monitor. The purpose of adapting Lee’s system was to create an effective interactive surface for use in an interactive learning environment appropriate for use with young students.

Large touch screen displays have been in the market for a few years. However, not all them are ready to support learning environments and many do not offer fast-enough response latencies for optimal user-experience in interactive learning applications. They are also very costly, physically large and heavy.

Two different applications of the system are presented consisting of a discrimination task and a racing car activity designed, respectively, to assess and remediate the “graph-as-picture” misconception in young students. The graphas-picture misconception occurs when the student interprets a graph (e.g. a line graph) as the picture of an object (e.g. a mountain). The infra-red tracking system has to accomplish two objectives. For the discrimination task it should enhance research by supporting the collection of spatial movement and response latency data. In the case of the racing car task it should enhance learning by providing rich, enjoyable and embodied natural interactions with the learning system. In the racing task, the child “drives” a car along a track while a speed/distance graph is plotted alongside concurrently. The racing car activity is designed to help students overcome the graph-as-picture misconception and it is particularly important for our research that participants could use their natural movements and receive kinaesthetic and proprioceptive feedback from their actions. In this case, the role of the technology is to provide an enactive interface that supports an embodied approach to learning without mediating artifacts such as computer mice, keyboards interposed between the learner and the material being learned - in other words to provide an embodied-interaction (EI) learning environment [ 12 ].

The Wiimote Whiteboard system allows for less constrained interactions than a touch screen monitor in that the user is not constrained to physically touch the screen surface - very light levels of contact and non-contact gestures above the display can be made. As Nogueira de Lima and Tall [ 10 ] point out, in arithmetic, the addition of whole numbers corresponds to physical actions on objects. Beichner [ 1 ] also suggests that this type of intervention could be especially useful to overcome the graph-as-picture misconception because the interaction creates an event-graph unit, which it is easy to manage in working-memory and to keep in long-term memory as a single entity (p.804-805). Mokros and Tinker [ 9 ] observe benefits from using what they call microcomputer based-labs (MBL) too because they provide a “...real-time link between a concrete experience and the symbolic representation of that experience... (in) Piagetian theory, MBL may be a bridge between concrete and formal operations...” (p.381) and they add that MBL is “...a genuine scientific experience for students: students gather and analize real data...” (p.381). Thus, our aim is to attempt to preserve the relationship between physical actions and concept acquisition in the context of a more abstract subject domain (speed/distance graphs) so that motor memories of actions (fine and gross motor movements of the arm, hand and fingers) and kinaesthetic feedback from friction of the fingertip against the screen become associated in the student’s mind with changes in speed of the racing car being “driven” by the student. The task is not a user-controllable animated diagram (e.g. [ 8 ]), rather the student generates data directly from his or her embodied operations and actions which are plotted in real time alongside the activity. We share the goal of Trninic and Abrahamson [ 12 ] of “...availing of novel technologies to engineer learning environments wherein students craft embodied artifacts in pursuit of mathematical competence” (p. 285).

The Wiimote Whiteboard system is cheap to build and use. It costs less than a hundred dollars compared to several thousand dollars for a large interactive touchscreen monitor. Unlike proprietary touch screen systems, the infra-red tracking system also has considerable potential to meet a variety of diagram research needs because the Wiimote project is open-source and extendible. We found that the system was more responsive than an expensive touch screen system.3 When used with a data projector it has the advantage of allowing the usable interactive screen area to be scaled to any size required.

The student-system interactions are logged and use in conjunction with video recordings of the computer screen and of the learner. Using multiple sources of data gives a rich picture of students’ learning trajectories and exemplifies technology-enhanced research [ 2 ].

The rest of the paper is organised as follows. We begin with a detailed description of the infra-red finger tracking device. Next two examples of how this device was used are presented and finally we discuss how motion-sensitive technologies such as the Wiimote Whiteboard, Kinect, and iPhone can be used to innovative methodologically and to design embodied interactive learning systems. 2

Infra-Red Finger Tracking Device

As mentioned earlier, the system presented in this document is a specialized application of Lee’s “Wiimote Whiteboard” [ 7 ]. The Wiimote uses the infra-red camera of the Nintendo Wii hand-held controller (Wiimote) to track infra-red light. Its 1024x768 infra-red camera has built-in hardware for infra-red tracking of up to 4 infra-red sources at 100Hz. Communication between the Wii controller and the computer is achieved through Bluetooth and the tracked infra-red light is translated to the computer as mouse input.

An infra-red finger tracking (IRFT) system was constructed comprised of a tiny high energy infra-red LED4 and a small micro-switch arranged in series in a simple DC circuit with power from a single “AAA” battery. The LED was positioned on the nail of the index finger of the student’s dominant hand by a small piece of surgical tape and the micro-switch was operated by the student using his forefinger and thumb producing events that were interpreted by the system as mouse clicks (Figure 1). The battery was enclosed in a battery holder sewn into an armband worn by the student. The armband was a modified version of one designed for joggers to hold keys and money. The tiny infra-red light and switch provided the inputs tracked by the Wii controller, which in turn, send data via Bluetooth to the Wiimote Whiteboard PC software which interpreted it as mouse movements and clicks.

The IRFT set-up was used with a vertically or horizontally mounted data projector or a vertically or horizontally mounted regular computer LCD display (Figure 2). Setting up the system involved defining and calibrating the interactive area via mouse clicks in the four corners of the projected image or on-screen display. Note that the participant was not required to learn particular gestures to control the system (except for activating the tiny “mouse click” button).

Next we present two different examples of how this technology was used for research and learning. Each of the following two sections contains a brief descrip3 40 NEC Touch Screen MultiSync LCD4020 4 e.g. www.digikey.com.au/product-detail/en/TSSS2600/751-1233-ND/1681368 tion of the task and describes how the IRFT device was integrated with other research methodology tools such as data logging, video recording and screen recording in order to richly characterise and record the student’s behaviour.

Diagrammatic Knowledge Assessment

We designed a graphical decision task to identify students’ “graph-as-picture” misconceptions (GAPM) and general diagrammatic knowledge (graphicacy) [ 4 ]. The aims were (1) to identify if the GAPM occurs in representational forms other than the line graphs originally studied by Janvier et al. and (2) to improve upon paper-and-pencil methods of assessing the GAPM. A modified version of a card-sort task [ 3 ] was used to assess participants’ graphical knowledge. Cox and Grawemeyer [ 3 ] asked participants to classify various representational forms and to name them. They found that subjects who were better at using diagrams for problem solving and reasoning tasks gave more accurate names to their classifications and organized them according to their semantics rather than superficial similarities (p.95, [ 3 ]). The six diagrammatic categories that were explored were: tables, bar charts, pie charts, line graphs, hierarchies/network diagrams, and set diagrams. The original paper and pencil task was computerised in a way that preserved the look and feel of the original desk-based card sort activity. Images of “cards” were therefore presented on the display and a participant could move them with his finger across the interactive surface in a sorting activity. The IRFT system was particularly effective in this application. Other touch-screen systems we tried could not be positioned horizontally over a desk or their screen-size were too small that some items’ details could not be clearly seen. The IRFT system addressed those two issues. It allowed the use of a large horizontal table sized display with a large number of “cards” visible to the student. 3.1

Brief Description of the Task and the Procedure

Discrimination Task. Students were presented with 72 unique pairs of items (one pair per trial). Each pair of items consisted of a picture and a diagram, a diagram and a diagram or a picture and a picture. On each trial a pair of items appeared in the centre of the screen and two green areas were displayed on the left and right sides of the display. At the top of the left side area was written “more like diagrams, charts, graphs” and on the top of the right side “more like pictures”. The participant sorted the pairs presented on the screen by sliding the cards on the interactive display. The student then clicked a “done” button and the next stimulus pair was presented. The activity continued until either 5 minutes had elapsed or the student had completed the task.

Equipment. The IRFT system was used as described in the previous section. The Wii controller was mounted about one meter above the student and it tracked the pupil’s finger movement as he interacted with the stimuli. A horizontally mounted LED 24” computer monitor was positioned at desk height. Participants and Procedure. Participants were primary (elementary) school students between third, fourth, fifth and sixth school years (age 7 to 11 years). They were interviewed individually in a room next to the classroom. They were first shown how to use the IRFT device and were given a few minutes practice using a painting application. Then the student was asked to sort each card by dragging it using his finger on the touch screen (that is, using the IRFT system) into one of the two green areas. The experimenter demonstrated the task and the student had some practice-trials before starting. 3.2

Example of Technology-Enhanced Research

Figure 3 presents an example of how one participant sorted a pair of cards in one of the trials. The technology was used to observe the student’s decisions at different levels of granularity.

The bottom part of figure 3 shows some of the student’s utterances and the top part shows an image obtained from the video-recording. The utterances represent the student’s explanation of why the two items (a network diagram and a picture) were positioned within the “more like pictures” area. The video recording works together with the on-screen recording. Student actions which are unclear on video-recording were triangulated with the screen recording and the IRTS data-logs. Response latencies were longer when the video-recording showed the student pointing to a particular card but then changing his mind. 3.3

Lessons Learned

The triangulation of data allowed using technology for research innovation. In particular, the IRFT device worked well in this task. The large interactive display area allowed large numbers of items to be presented and provided the student with a wider area for “grabbing” cards. The system tracking speeds were very good and kept pace with students’ speed of card moving. In addition, young students were really keen and seemed motivated to use it. However the IRFT system had a couple of weaknesses. Some students noted that the IRFT device could be used without touching the screen (i.e. with finger hovered above screen surface level) and they had to be reminded to keep their finger close to the display. Secondly the Wiimote software occasionally failed during the display of multiple frames. However, this did not happen frequently and it was quickly resolved by a “minimize and maximize screen” action. The amount of data loss caused by this was small though compared with that endured using other touchscreen systems such as a 23” Hewlett Packard TouchSmart interactive touchscreen PC which crashed and had to be rebooted in the middle of an experimental session causing the student to withdraw and in comparison to the very slow response latencies of an expensive NEC MultiSync 40” LCD touch-screen which rendered it unusable for research. 4

Dynalinked Interactive Diagrams - the Racing Car activity

Students who were identified in the previous task as having a graph-as-picture misconception were invited to try an interactive racing car activity designed as a remedial intervention[ 5 ]. In the racing activity the student “drove” a car around variously shaped racing tracks while a speed/distance graph was plotted concurrently alongside the track. The activity was based on previous research on interactive environments designed to help students understand abstractions (e.g. [ 11 ], [ 9 ]) and it aimed to improve Janvier’s paper-and-pencil tasks [ 6 ].

Students were given a few trial tracks to race before starting the task proper. Following the practice sessions, students were allowed to play with six different racing tracks. The tracks varied in complexity and form and each student was permitted to freely experiment and race the car at his will. The participant was questioned about his plans and observations during the activities and he was encourage to “think aloud” during his performance. Additionally, at the end of each trial, a replay of the student’s experiments was presented. During the replay, the student was asked to recall his experiences (retrospective debriefing).

The track and graph were “dynamically linked” [ 11 ] or dynalinked. The dynalinking took several forms. First the track was coloured according to speed behind the car as it progressed around the track and corresponding colours on function graph axis reflected this. In addition a “speedometer” was displayed on car’s roof in order to make more salient the relationship between the car’s speed and features on the plotted graph. Each race was logged in order to allow for immediate replay and the student was encouraged to reflect on his actions and their effect on the graph during the replay. 4.1

Example of Technology-Enhanced Learning

Figure 4 shows an example of how data was collected and how data from various sources was triangulated. The data analysis software allowed a detailed view of what the student was visualizing at different points of time and video-recording enabled us to observe student’s gestures and utterances.

During the race the student had to divide his attention between the race track and the graph, but during the replay the learner could devote full attention to the effect of his car racing actions upon the graph. Thus our research set-up allowed us to combine embodied actions (student movements) and reinforced connections between physical actions and visually perceived graph forms [ 5 ]. Compared to the assessment task, the IRFT device had a more critical function in the racing activity. It had to be highly accurate in order to allow the student “grabbing” the car and it had to respond with minimal latency to allow the student to race the car at quite fast speeds. For the youngest students (7-8 years old) “grabbing” the car proved slightly challenging, but older students had less difficulties in this aspect. Additionally, the response time of the IRFT for dragging the car was acceptable, but some limitations were clear when fast gestures did not produced the expected output. One of the factors that caused some difficulties was the student covering the infra-red LED with part of his hand, in those cases, a reminder on keeping the finger extended had to be done. In spite of some initial difficulties, the students’ experience using the IRFT device was much smoother (in terms of response latency) than was for a group of students who performed the same task on a NEC MultiSync 40” LCD touchscreen display. Thus a system that cost around $50 to build for our purposes outperformed a piece of equipment costing several thousand dollars. 5

Conclusions

Two examples of interactive diagrammatic systems that were administrated to young students using an IRFT device showed how technology could be used to innovate methodologically and enhance learning and research. In particular, the IRFT device proved cost-effective and it was adaptable to the needs of the research. In spite of some difficulties for manipulating very small items, the fast-response experience observed in the system outperformed some of the existing touch-screen displays. Table 1 presents a summary of some IRFT device’s advantages and limitations.

The IRFT device provided a useful low-cost basis for the design of an embodiedinteractive diagram learning system and also provided rich research data streams. Students appeared to enjoy using the system and their comments on the device were very positive.

Advantages ⋆ Acceptable “grabbing” of large

items. ⋆ Acceptable responsivity to tracking

of large items. ⋆ Adaptable to different surfaces and

sizes. ⋆ Affordable. ⋆ Extensible for multi-touch purposes. ⋆ Young students (9 year-olds) could

use use it satisfactorily. ⋆ Students were keen to use it.

Limitations × Some difficulties to “grab” small

items. × Difficulties to track very fast ges

tures. × The Wiimote software could interfere

when using multiple frames. × Covering the infra-red light stops the

tracking.

Acknowledgements

We would like to thank the two anonymous reviewers for their valuable comments and suggestions.

References

1. Beichner , R.J.: The effect of simultaneous motion presentation and graph generation in a kinematics lab . Journal of Research in Science Teaching , 27 ( 8 ), 803 - 815 ( 1990 )

2. Cox , R : Technology-enhanced research: Educational ICT systems as research instruments . Technology, Pedagogy & Education , 16 ( 3 ), 337 - 356 ( 2007 )

3. Cox R. , Grawemeyer B. : The mental organisation of external representations . In: Schmalhofer, F. , Young , R. , Katz , G . (eds.), European Cognitive Science Conference (EuroCogSci - joint Cognitive Science Society and German Cognitive Science Society conference) . Proceedings of EuroCogSci03 (1st European Cognitive Science Conference) , 91 - 96 . Lawrence Erlbaum Associates ( 2003 )

4. Garcia Garcia , G. , Cox , R.: “ Graph-as-Picture” Misconceptions in Young Students . In: Goel, A.K. , Jamnik , M. , Narayanan , N.H. (eds.) Diagrams 2010 . LNCS, vol. 6170 , 310 - 312 . Springer ( 2010 )

5. Garcia Garcia , G. , Cox , R.: An Interactive Educational Diagrammatic System for Assessing and Remediating the Graph-as-Picture Misconception . In: Aleven, V. , Kay , J. , Mostow , J . (eds.) Intelligent Tutoring Systems 2010. LNCS , vol. 6095 , 224 - 226 . Springer ( 2010 )

6. Janvier C. : The Interpretation of Complex Cartesian Graphs Representing Situations - Studies and Teaching Experiments . PhD thesis , University of Nottingham ( 1978 )

7. Lee , J.C. : Projects - Wii. http://johnnylee.net/projects/wii/

8. Lowe , R.: Animation and learning: Selective processing of information in dynamic graphics . Learning and Instruction , 13 , 157 - 176 ( 2003 )

9. Mokros , J.R. , Tinker R.F. : The impact of microcomputer-based labs on children's ability to interpret graphs . Journal of Research in Science Teaching , 24 ( 4 ), 369 - 383 ( 1987 )

10. Nogueira de Lima, R., Tall , D. : Procedural embodiment and magic in linear equations . Educational Studies in Mathematics , 67 ( 1 ), 3 - 18 ( 2008 )

11. Rogers Y. , Scaife M. , Aldrich F. , Price S. : Improving children's understanding of formalisms through interacting with multimedia . Technical report , University of Sussex. CSRP Technical Report No. 559 ( 2003 )

12. Trninic , D. , Abrahamson , D. : Embodied artifacts and conceptual performances . In: J. v. Aalst, K. Thompson , M. J. Jacobson , P. Reimann (eds.) Proceedings of the International Conference of the Learning Sciences: Future of Learning (ICLS 2012) , vol. 1 , 283 - 290 . Sydney: University of Sydney / ISLS ( 2012 )