=Paper=
{{Paper
|id=None
|storemode=property
|title=An infra-red finger tracking system used in the assessment and remediation of "graph-as-picture" misconceptions
|pdfUrl=https://ceur-ws.org/Vol-887/GarciaGarciaandCox.pdf
|volume=Vol-887
}}
==An infra-red finger tracking system used in the assessment and remediation of "graph-as-picture" misconceptions==
https://ceur-ws.org/Vol-887/GarciaGarciaandCox.pdf
An infra-red finger tracking system used in the
assessment and remediation of
“graph-as-picture” misconceptions
Grecia Garcia Garcia1 and Richard Cox2
1
University of Sussex, Brighton, BN19RH, UK
gg44@sussex.ac.uk
2
The University of Edinburgh, Edinburgh, EH8 9AB, UK
rcox@inf.ed.ac.uk
Abstract. The workshop presentation will describe a specialized ap-
plication of Lee’s “Wiimote Whiteboard” [7] an infra-red camera based
tracking system which uses the Nintendo Wii wireless remote control unit
and Bluetooth. Young students wear a very small infra-red LED on their
index finger with a forefinger/thumb operated micro-switch for produ-
cing “mouse clicks”. This system is combined with a vertically mounted
data projector or a horizontally mounted regular computer LCD display,
creating a cost-effective large interactive touch surface. The system has
a fast response time and has been used with primary school students in
diagrammatic knowledge (graphicacy) assessment [4] and in interactive
dynalinked diagrammatic applications [5]. These applications were de-
signed to investigate the “graph-as-picture” misconception and they will
be described and demonstrated at the workshop.
Keywords: interactive environments, graphicacy, touch-screen techno-
logy
1 Introduction
For this workshop presentation, we will demonstrate how Lee’s [7] “Wiimote
Whiteboard” (an infra-red tracking system) was adapted to create a very af-
fordable and fast-responding interactive surface from a data-projector’s image
or a horizontally mounted regular computer monitor. The purpose of adapting
Lee’s system was to create an effective interactive surface for use in an interactive
learning environment appropriate for use with young students.
Large touch screen displays have been in the market for a few years. However,
not all them are ready to support learning environments and many do not offer
fast-enough response latencies for optimal user-experience in interactive learning
applications. They are also very costly, physically large and heavy.
Two different applications of the system are presented consisting of a dis-
crimination task and a racing car activity designed, respectively, to assess and
remediate the “graph-as-picture” misconception in young students. The graph-
as-picture misconception occurs when the student interprets a graph (e.g. a line
graph) as the picture of an object (e.g. a mountain). The infra-red tracking
system has to accomplish two objectives. For the discrimination task it should
enhance research by supporting the collection of spatial movement and response
latency data. In the case of the racing car task it should enhance learning
by providing rich, enjoyable and embodied natural interactions with the lear-
ning system. In the racing task, the child “drives” a car along a track while a
speed/distance graph is plotted alongside concurrently. The racing car activity
is designed to help students overcome the graph-as-picture misconception and
it is particularly important for our research that participants could use their
natural movements and receive kinaesthetic and proprioceptive feedback from
their actions. In this case, the role of the technology is to provide an enactive
interface that supports an embodied approach to learning without mediating
artifacts such as computer mice, keyboards interposed between the learner and
the material being learned - in other words to provide an embodied-interaction
(EI) learning environment [12].
The Wiimote Whiteboard system allows for less constrained interactions than
a touch screen monitor in that the user is not constrained to physically touch the
screen surface - very light levels of contact and non-contact gestures above the
display can be made. As Nogueira de Lima and Tall [10] point out, in arithmetic,
the addition of whole numbers corresponds to physical actions on objects. Beich-
ner [1] also suggests that this type of intervention could be especially useful to
overcome the graph-as-picture misconception because the interaction creates an
event-graph unit, which it is easy to manage in working-memory and to keep in
long-term memory as a single entity (p.804-805). Mokros and Tinker [9] observe
benefits from using what they call microcomputer based-labs (MBL) too because
they provide a “...real-time link between a concrete experience and the symbolic
representation of that experience... (in) Piagetian theory, MBL may be a bridge
between concrete and formal operations...” (p.381) and they add that MBL is
“...a genuine scientific experience for students: students gather and analize real
data...” (p.381). Thus, our aim is to attempt to preserve the relationship be-
tween physical actions and concept acquisition in the context of a more abstract
subject domain (speed/distance graphs) so that motor memories of actions (fine
and gross motor movements of the arm, hand and fingers) and kinaesthetic feed-
back from friction of the fingertip against the screen become associated in the
student’s mind with changes in speed of the racing car being “driven” by the
student. The task is not a user-controllable animated diagram (e.g. [8]), rather
the student generates data directly from his or her embodied operations and
actions which are plotted in real time alongside the activity. We share the goal
of Trninic and Abrahamson [12] of “...availing of novel technologies to engineer
learning environments wherein students craft embodied artifacts in pursuit of
mathematical competence” (p. 285).
The Wiimote Whiteboard system is cheap to build and use. It costs less
than a hundred dollars compared to several thousand dollars for a large interac-
tive touchscreen monitor. Unlike proprietary touch screen systems, the infra-red
tracking system also has considerable potential to meet a variety of diagram
43
research needs because the Wiimote project is open-source and extendible. We
found that the system was more responsive than an expensive touch screen sys-
tem.3 When used with a data projector it has the advantage of allowing the
usable interactive screen area to be scaled to any size required.
The student-system interactions are logged and use in conjunction with video
recordings of the computer screen and of the learner. Using multiple sources
of data gives a rich picture of students’ learning trajectories and exemplifies
technology-enhanced research [2].
The rest of the paper is organised as follows. We begin with a detailed descrip-
tion of the infra-red finger tracking device. Next two examples of how this device
was used are presented and finally we discuss how motion-sensitive technologies
such as the Wiimote Whiteboard, Kinect, and iPhone can be used to innovative
methodologically and to design embodied interactive learning systems.
2 Infra-Red Finger Tracking Device
As mentioned earlier, the system presented in this document is a specialized
application of Lee’s “Wiimote Whiteboard” [7]. The Wiimote uses the infra-red
camera of the Nintendo Wii hand-held controller (Wiimote) to track infra-red
light. Its 1024x768 infra-red camera has built-in hardware for infra-red tracking
of up to 4 infra-red sources at 100Hz. Communication between the Wii controller
and the computer is achieved through Bluetooth and the tracked infra-red light
is translated to the computer as mouse input.
An infra-red finger tracking (IRFT) system was constructed comprised of a
tiny high energy infra-red LED4 and a small micro-switch arranged in series in
a simple DC circuit with power from a single “AAA” battery. The LED was
positioned on the nail of the index finger of the student’s dominant hand by a
small piece of surgical tape and the micro-switch was operated by the student
using his forefinger and thumb producing events that were interpreted by the
system as mouse clicks (Figure 1). The battery was enclosed in a battery holder
sewn into an armband worn by the student. The armband was a modified version
of one designed for joggers to hold keys and money. The tiny infra-red light and
switch provided the inputs tracked by the Wii controller, which in turn, send
data via Bluetooth to the Wiimote Whiteboard PC software which interpreted
it as mouse movements and clicks.
The IRFT set-up was used with a vertically or horizontally mounted data
projector or a vertically or horizontally mounted regular computer LCD display
(Figure 2). Setting up the system involved defining and calibrating the interactive
area via mouse clicks in the four corners of the projected image or on-screen
display. Note that the participant was not required to learn particular gestures
to control the system (except for activating the tiny “mouse click” button).
Next we present two different examples of how this technology was used for
research and learning. Each of the following two sections contains a brief descrip-
3
40 NEC Touch Screen MultiSync LCD4020
4
e.g. www.digikey.com.au/product-detail/en/TSSS2600/751-1233-ND/1681368
44
Fig. 1. A tiny infra-red LED on the top of the index finger (left), the micro-switch
which is used to produce “mouse clicks” (middle), and an example of how the infra-red
finger tracking device looks on a user’s hand (right).
tion of the task and describes how the IRFT device was integrated with other
research methodology tools such as data logging, video recording and screen
recording in order to richly characterise and record the student’s behaviour.
Fig. 2. Two different configurations for the infra-red finger tracking system.
3 Diagrammatic Knowledge Assessment
We designed a graphical decision task to identify students’ “graph-as-picture”
misconceptions (GAPM) and general diagrammatic knowledge (graphicacy) [4].
The aims were (1) to identify if the GAPM occurs in representational forms
other than the line graphs originally studied by Janvier et al. and (2) to improve
upon paper-and-pencil methods of assessing the GAPM. A modified version of a
card-sort task [3] was used to assess participants’ graphical knowledge. Cox and
Grawemeyer [3] asked participants to classify various representational forms and
to name them. They found that subjects who were better at using diagrams for
problem solving and reasoning tasks gave more accurate names to their classifi-
cations and organized them according to their semantics rather than superficial
similarities (p.95, [3]). The six diagrammatic categories that were explored were:
tables, bar charts, pie charts, line graphs, hierarchies/network diagrams, and set
diagrams. The original paper and pencil task was computerised in a way that
preserved the look and feel of the original desk-based card sort activity. Images
45
of “cards” were therefore presented on the display and a participant could move
them with his finger across the interactive surface in a sorting activity. The IRFT
system was particularly effective in this application. Other touch-screen systems
we tried could not be positioned horizontally over a desk or their screen-size were
too small that some items’ details could not be clearly seen. The IRFT system
addressed those two issues. It allowed the use of a large horizontal table sized
display with a large number of “cards” visible to the student.
3.1 Brief Description of the Task and the Procedure
Discrimination Task. Students were presented with 72 unique pairs of items
(one pair per trial). Each pair of items consisted of a picture and a diagram, a
diagram and a diagram or a picture and a picture. On each trial a pair of items
appeared in the centre of the screen and two green areas were displayed on the
left and right sides of the display. At the top of the left side area was written
“more like diagrams, charts, graphs” and on the top of the right side “more like
pictures”. The participant sorted the pairs presented on the screen by sliding
the cards on the interactive display. The student then clicked a “done” button
and the next stimulus pair was presented. The activity continued until either 5
minutes had elapsed or the student had completed the task.
Equipment. The IRFT system was used as described in the previous section.
The Wii controller was mounted about one meter above the student and it
tracked the pupil’s finger movement as he interacted with the stimuli. A hori-
zontally mounted LED 24” computer monitor was positioned at desk height.
Participants and Procedure. Participants were primary (elementary) school
students between third, fourth, fifth and sixth school years (age 7 to 11 years).
They were interviewed individually in a room next to the classroom. They were
first shown how to use the IRFT device and were given a few minutes practice
using a painting application. Then the student was asked to sort each card by
dragging it using his finger on the touch screen (that is, using the IRFT system)
into one of the two green areas. The experimenter demonstrated the task and
the student had some practice-trials before starting.
3.2 Example of Technology-Enhanced Research
Figure 3 presents an example of how one participant sorted a pair of cards in
one of the trials. The technology was used to observe the student’s decisions at
different levels of granularity.
The bottom part of figure 3 shows some of the student’s utterances and
the top part shows an image obtained from the video-recording. The utterances
represent the student’s explanation of why the two items (a network diagram
and a picture) were positioned within the “more like pictures” area. The video
recording works together with the on-screen recording. Student actions which
46
Fig. 3. A participant sorts a pair of cards during one of the trials. From bottom to
top: participant’s utterances obtained through video recording; the on-screen recording
shows what the computer is presenting to the student; a time-line (in milliseconds)
marking the different actions that the student performed; and a picture of the video-
recording that was used to observe the student’s gestures.
are unclear on video-recording were triangulated with the screen recording and
the IRTS data-logs. Response latencies were longer when the video-recording
showed the student pointing to a particular card but then changing his mind.
3.3 Lessons Learned
The triangulation of data allowed using technology for research innovation. In
particular, the IRFT device worked well in this task. The large interactive display
area allowed large numbers of items to be presented and provided the student
with a wider area for “grabbing” cards. The system tracking speeds were very
good and kept pace with students’ speed of card moving. In addition, young
students were really keen and seemed motivated to use it. However the IRFT
system had a couple of weaknesses. Some students noted that the IRFT device
could be used without touching the screen (i.e. with finger hovered above screen
surface level) and they had to be reminded to keep their finger close to the
display. Secondly the Wiimote software occasionally failed during the display
of multiple frames. However, this did not happen frequently and it was quickly
resolved by a “minimize and maximize screen” action. The amount of data loss
caused by this was small though compared with that endured using other touch-
screen systems such as a 23” Hewlett Packard TouchSmart interactive touch-
screen PC which crashed and had to be rebooted in the middle of an experimental
47
session causing the student to withdraw and in comparison to the very slow
response latencies of an expensive NEC MultiSync 40” LCD touch-screen which
rendered it unusable for research.
4 Dynalinked Interactive Diagrams - the Racing Car
activity
Students who were identified in the previous task as having a graph-as-picture
misconception were invited to try an interactive racing car activity designed
as a remedial intervention[5]. In the racing activity the student “drove” a car
around variously shaped racing tracks while a speed/distance graph was plotted
concurrently alongside the track. The activity was based on previous research
on interactive environments designed to help students understand abstractions
(e.g. [11], [9]) and it aimed to improve Janvier’s paper-and-pencil tasks [6].
Students were given a few trial tracks to race before starting the task proper.
Following the practice sessions, students were allowed to play with six different
racing tracks. The tracks varied in complexity and form and each student was
permitted to freely experiment and race the car at his will. The participant was
questioned about his plans and observations during the activities and he was
encourage to “think aloud” during his performance. Additionally, at the end
of each trial, a replay of the student’s experiments was presented. During the
replay, the student was asked to recall his experiences (retrospective debriefing).
The track and graph were “dynamically linked” [11] or dynalinked. The dy-
nalinking took several forms. First the track was coloured according to speed
behind the car as it progressed around the track and corresponding colours on
function graph axis reflected this. In addition a “speedometer” was displayed on
car’s roof in order to make more salient the relationship between the car’s speed
and features on the plotted graph. Each race was logged in order to allow for
immediate replay and the student was encouraged to reflect on his actions and
their effect on the graph during the replay.
4.1 Example of Technology-Enhanced Learning
Figure 4 shows an example of how data was collected and how data from various
sources was triangulated. The data analysis software allowed a detailed view of
what the student was visualizing at different points of time and video-recording
enabled us to observe student’s gestures and utterances.
During the race the student had to divide his attention between the race track
and the graph, but during the replay the learner could devote full attention to
the effect of his car racing actions upon the graph. Thus our research set-up
allowed us to combine embodied actions (student movements) and reinforced
connections between physical actions and visually perceived graph forms [5].
48
Fig. 4. A “race” during the racing car activity. From bottom to top: a time-line
that describes the two phases: the activity and the replay; screen-shots of the video-
recording; some of the student’s utterances; and images of the on-screen recording.
4.2 Lessons Learned
Compared to the assessment task, the IRFT device had a more critical function
in the racing activity. It had to be highly accurate in order to allow the student
“grabbing” the car and it had to respond with minimal latency to allow the
student to race the car at quite fast speeds. For the youngest students (7-8
years old) “grabbing” the car proved slightly challenging, but older students
had less difficulties in this aspect. Additionally, the response time of the IRFT
for dragging the car was acceptable, but some limitations were clear when fast
gestures did not produced the expected output. One of the factors that caused
some difficulties was the student covering the infra-red LED with part of his
hand, in those cases, a reminder on keeping the finger extended had to be done.
In spite of some initial difficulties, the students’ experience using the IRFT
device was much smoother (in terms of response latency) than was for a group
of students who performed the same task on a NEC MultiSync 40” LCD touch-
screen display. Thus a system that cost around $50 to build for our purposes
outperformed a piece of equipment costing several thousand dollars.
5 Conclusions
Two examples of interactive diagrammatic systems that were administrated to
young students using an IRFT device showed how technology could be used
to innovate methodologically and enhance learning and research. In particular,
the IRFT device proved cost-effective and it was adaptable to the needs of the
research. In spite of some difficulties for manipulating very small items, the
49
fast-response experience observed in the system outperformed some of the exis-
ting touch-screen displays. Table 1 presents a summary of some IRFT device’s
advantages and limitations.
The IRFT device provided a useful low-cost basis for the design of an embodied-
interactive diagram learning system and also provided rich research data streams.
Students appeared to enjoy using the system and their comments on the device
were very positive.
Advantages Limitations
⋆ Acceptable “grabbing” of large × Some difficulties to “grab” small
items. items.
⋆ Acceptable responsivity to tracking × Difficulties to track very fast ges-
of large items. tures.
⋆ Adaptable to different surfaces and × The Wiimote software could interfere
sizes. when using multiple frames.
⋆ Affordable. × Covering the infra-red light stops the
⋆ Extensible for multi-touch purposes. tracking.
⋆ Young students (9 year-olds) could
use use it satisfactorily.
⋆ Students were keen to use it.
Table 1. Advantages and limitations of the infra-red tracking system.
Acknowledgements
We would like to thank the two anonymous reviewers for their valuable comments
and suggestions.
References
1. Beichner, R.J.: The effect of simultaneous motion presentation and graph generation
in a kinematics lab. Journal of Research in Science Teaching, 27(8), 803–815 (1990)
2. Cox, R: Technology-enhanced research: Educational ICT systems as research instru-
ments. Technology, Pedagogy & Education, 16(3), 337–356 (2007)
3. Cox R., Grawemeyer B.: The mental organisation of external representations. In:
Schmalhofer, F., Young, R., Katz, G. (eds.), European Cognitive Science Confer-
ence (EuroCogSci - joint Cognitive Science Society and German Cognitive Science
Society conference). Proceedings of EuroCogSci03 (1st European Cognitive Science
Conference), 91–96. Lawrence Erlbaum Associates (2003)
50
4. Garcia Garcia, G., Cox, R.: “Graph-as-Picture” Misconceptions in Young Students.
In: Goel, A.K., Jamnik, M., Narayanan, N.H. (eds.) Diagrams 2010. LNCS, vol. 6170,
310–312. Springer (2010)
5. Garcia Garcia, G., Cox, R.: An Interactive Educational Diagrammatic System for
Assessing and Remediating the Graph-as-Picture Misconception. In: Aleven, V., Kay,
J., Mostow, J. (eds.) Intelligent Tutoring Systems 2010. LNCS, vol. 6095, 224–226.
Springer (2010)
6. Janvier C.: The Interpretation of Complex Cartesian Graphs Representing Situa-
tions - Studies and Teaching Experiments. PhD thesis, University of Nottingham
(1978)
7. Lee, J.C.: Projects – Wii. http://johnnylee.net/projects/wii/
8. Lowe, R.: Animation and learning: Selective processing of information in dynamic
graphics. Learning and Instruction, 13, 157–176 (2003)
9. Mokros, J.R., Tinker R.F.: The impact of microcomputer-based labs on children’s
ability to interpret graphs. Journal of Research in Science Teaching, 24(4), 369–383
(1987)
10. Nogueira de Lima, R., Tall,D.: Procedural embodiment and magic in linear equa-
tions. Educational Studies in Mathematics, 67 (1), 3–18 (2008)
11. Rogers Y., Scaife M., Aldrich F., Price S.: Improving children’s understanding
of formalisms through interacting with multimedia. Technical report, University of
Sussex. CSRP Technical Report No. 559 (2003)
12. Trninic, D., Abrahamson, D.: Embodied artifacts and conceptual performances.
In: J. v. Aalst, K. Thompson, M. J. Jacobson, P. Reimann (eds.) Proceedings of the
International Conference of the Learning Sciences: Future of Learning (ICLS 2012),
vol. 1, 283–290. Sydney: University of Sydney / ISLS (2012)
51