Design of a Virtual Reality System for the Study of Diagram Use in Organic Chemistry Andrew T. Stull, Trevor J. Barrett, & Mary Hegarty Department of Psychological & Brain Sciences, University of California Santa Barbara, CA 93101, USA {andrew.stull, trevor.barrett, mary.hegarty}@psych.ucsb.edu Abstract. Organic chemists must be adept at relating different 2D diagrammat- ic representations of molecules while also understanding their 3D structure. Concrete (3D) models can aid students in developing these aspects of represen- tational competence but a growing trend is to incorporate virtual 3D models in- to instruction. In this paper, we describe the design of a virtual reality system to investigate how students use virtual models, for learning about different struc- tural diagrams common in organic chemistry. We follow with preliminary re- sults of a study comparing the relative effectiveness of virtual versus concrete models. Participants performed tasks using either virtual or concrete models to match or to complete three different types of molecular diagrams. The prelimi- nary results suggest a benefit of using virtual models over concrete models. Keywords. Diagrammatic reasoning, representational competence, concrete manipulatives, virtual reality, organic chemistry 1 Introduction Molecular diagrams and models are essential tools of chemistry [1]. Arguably, such representations might even be a defining characteristic of chemistry, because its do- main is built on reasoned logic using diagrams and models as primary research tools [2]. Diagrams and models are also an important tool for chemistry instruction. Devel- oping skills to draw, interpret, and translate between these representations is essential to a student’s growth as a chemist. In previous research, we demonstrated that use of concrete models benefited performance in a diagram translation task [3]. The question addressed here is whether computer-based, or virtual, models are as helpful for such tasks. We describe the design and initial testing of a virtual reality system to investi- gate how students translate between models and diagrams in organic chemistry. 33 2 Molecular Representations Chemists use two general types of spatial representations of molecules; 3D models, which might be concrete (i.e., physical) or virtual, and 2D diagrams, which use dia- grammatic conventions to represent 3D relations in the two dimensions of the printed page (see Fig. 1). Although chemists routinely employ these diverse representations in practice, novices often have difficulty mastering their use [4-6]. For students to be successful at integrating multiple representations, they must develop the skills of con- structing, interpreting, and transforming those representations. Collectively, these skills are referred to as representational competence [7, 8]. Chemistry is an ideal do- main in which to study representational competence because chemists rely heavily on multiple representations [9-11]. Fig. 1. Four structural representations of an organic molecule. (a) A concrete (ball-and-stick) model where color is used to denote different atoms. Black is carbon and white is hydrogen. (b) Dash-Wedge diagram or side-view, (c) Newman diagram or end-view, and (d) Fischer diagram or upright-view of the same organic molecule depicted in the ball-and-stick model. Spatial thinking is also important in chemistry. Many of the representations used in the chemistry curriculum support thinking and reasoning about spatial relationships within and between molecules [12]. Spatial thinking is important because the reactivi- ty of molecules is predicted, not just by the number and type of atoms that make up a molecule, but by their spatial configuration. Spatial cognition is also central to under- standing different diagrammatic representations. The four representations in Fig. 1 represent the same molecule with the three diagrams depicting the molecule from different perspectives. Rotating the groups of atoms around the central carbon-carbon bond produce different “conformations” of the same molecule that are depicted in different diagram formats. Such transformations do not change the identity of the molecule. In contrast, breaking the bonds and rearranging the subgroups of atoms (CH3, NH2, etc.) produces a different molecule that has different reactive properties. 3 Virtual Reality Display System In previous research we have demonstrated that concrete models are helpful tools for students who are first learning to translate between different diagrams such as those in Figure 1 [3]. Here we raise the question of whether virtual models offer this same support. Although virtual models offer advantages of flexibility, and availability (for 34 example they are provided with many textbooks), they may also have disadvantages. For example, virtual models lack specific haptic and proprioceptive cues present in concrete models, virtual models typically lack stereoscopic depth cues, and interac- tion with these models is mediated by a computer interface, rather than being direct. As a first step toward investigating the relative value of virtual and concrete mod- els, we developed a virtual reality system that was designed to be as similar to a real model as possible (in terms of the cues that a viewer might have when using a con- crete model). The provided cues include stereo-depth cues, co-location of the hand- held object (i.e., interface) with the viewed image, and a direct manipulation interface that enables the user to perform the most task relevant interactions with the virtual model that they would perform with the concrete model. This paper describes the first step in developing and testing a system for studying a full virtual model. With this system, we plan to conduct controlled experiments to test if virtual models can be as effective as concrete models in promoting diagrammat- ic reasoning. Once we establish a base-line for the relative value of the two model types, we will be able to pursue several lines of research. To understand the im- portance of various perceptual cues, we plan to systematically alter or remove specific cues (such as stereoscopic viewing and co-location of the interface and model). To understand characteristics of efficient model use that promote academic achievement, we can track the actual manipulations of objects of varying visual or physical com- plexities and affordances. To understand if models of various types support or inhibit knowledge transfer, we can alter training conditions and evaluate later performance without models. To understand the role of the physical interface in model use and performance, we can systematically alter the affordances of the hand-held interface. Collectively, the knowledge gained will help us establish principles for effective model design that supports students as they develop representational competence. 3.1 System Design The virtual reality system (see Fig. 2) was modeled after an integrated graphic and haptic system developed by Ernst and Banks [17], which was configured to portray the illusion that a displayed virtual model was directly manipulated with the partici- pants’ co-located hands (see Fig. 3), via a hand-held interface (see Fig. 4). The hand- held interface was designed to allow both global rotation of the whole virtual object and local, or internal, rotation of key parts within the object, as both of these types of rotation are necessary for the task of relating models to diagrams. Local rotation al- lowed participants to make modifications to the virtual model in the course of match- ing or completing a given diagram. In addition, stereo glasses were used to provide stereoscopic depth cues. Virtual reality display. This system consisted of a 120Hz Samsung SyncMaster 2233 LCD computer display, and an Nvidia Quadro FX580 graphics board. A 32bit Windows7 machine with an Intel i5 650 3.2GHz processor and 4Gb of RAM ran Viz- ard 3.0 virtual reality software by Worldviz© (Santa Barbara, CA) to display the vir- tual models and to collect participant response data. The participant’s computer dis- 35 play was mounted horizontally on a metal frame facing downwards above the desk surface. Fig. 2. The participant’s computer display was mounted horizontally on a metal frame 53cm (A) above the desk surface and facing downward. The hand-held interface was positioned on an adjustable stand 33cm (B) from the edge of the desk. The experimenter viewed the participant’s progress by way of a separate computer display. Participants viewed the displayed image from a mirror attached to the metal frame and positioned 45° from the monitor’s screen surface. This configuration allowed participants to manipulate the hand-held interface in an area that was about 20cm beyond the surface of the mirror and about 45 cm from the viewer’s eyes. As a result, the interface was co-located with the image of the virtual model. Fig. 3. The hand-held interface was co-located with the virtual image. Hand-held interface. The interface was composed of an acrylic cylinder (length 11.8 cm; diameter: 5.1 cm) consisting of two halves of equal length. These two halves rotated independently about the interface’s long axis. One half contained a three de- gree-of-freedom (3DOF) orientation tracker and an optical shaft encoder. The orienta- tion tracker was an InertiaCube2 developed by InterSense, Inc. (Bedford, MA), which controlled and recorded changes in the global orientation of the virtual object. The optical shaft encoder was an AD4 encoder developed by US Digital (Vancou- ver, WA), which tracked local rotational movement of the second half of the inter- face. The second half of the interface was attached to the first by the rotational shaft of the encoder and it was balanced to match the weight of the first half. The overall length and width of the interface matched the general length and width of the dis- played virtual models, although differing in shape. 36 The height of the stand for the hand-held interface was adjusted for each partici- pant then the monitor was positioned to display the virtual model in the same location as the physical interface. Global (from the orientation tracker) and local (from the rotations tracker) movement and timing of the interface were used to control move- ments and timing of the virtual model. Tracking data was collected by the Vizard 3.0 software, which allowed for later playback and detailed analysis of participants’ per- formance. Fig. 4. The hand-held interface contained two tracking units. The orientation tracker is depicted in blue, and the optical encoder is depicted in red. The cords for the two devices emerged at the junction between the two halves. 3D glasses. Active liquid crystal (LC) shutter glasses (Nvidia 3D Vision Wireless Glasses) were used in conjunction with the 120Hz monitor to provide stereoscopic viewing of the virtual models. The glasses work by alternatively darkening or making transparent one or the other lens in synchrony with the refresh rate of the computer display. The computer display alternates between two images of the same object that are displaced by a horizontal distance. When synchronized with the shutter glasses, this creates the illusion of depth when viewing a graphic on the flat surface of the computer display. The high refresh-rate of the 120Hz monitor helped to reduce flicker that would have resulted from a standard monitor. Software drivers for the shutter glasses were included with the Vizard 3.0 software. 3.2 Design Challenges Image Reversal. When using a mirror to view actual graphics from a computer dis- play, the viewed apparent image is flipped across the object’s vertical axis. This was addressed by mirror-reversing the actual graphic so that the apparent image was in the correct orientation. This adjustment meant that a clockwise rotation of the interface corresponded to an apparent clockwise rotation of the viewed images, although it would appear to be counterclockwise if viewed without the mirror. In addition, be- cause we were recording the global and local rotations of the object by the user, it was necessary to adjust the program code to record mirror-reversed orientation and rota- tion data. This allowed the participant to interact with the virtual models naturalisti- cally as if the mirror was a standard monitor. Polarization. Using active LC shuttle glasses to view a graphic reflected from a mir- ror altered polarization of the reflected light. This resulted in the apparent image be- 37 ing invisible in normal head orientation due to the incompatibility between the polari- ty of the LC shutter glasses and the polarity of the mirror-reflected image from the LCD display. By rotating the LCD display 90°, the polarities were synchronized and the apparent image could be easily viewed. In addition, it was necessary to rotate the graphic by 90° (display it in portrait mode) so that it was properly oriented for the viewer. Vergence-accommodation. Viewing fatigue often results when there is a large dis- parity between a viewer’s vergence (intersection of line of sight) and accommodation (focal point). With our equipment, the focal point, the distance to the surface of the mirror, was different from the vergence point, the distance to the virtual image (see Fig. 3). In order to mitigate 3D fatigue caused by this conflict [18], the difference between vergence distance and focal distance was minimized (see Fig. 5). This al- lowed the participants’ eyes to converge and accommodate with minimizes eye strain but still providing depth information and also to have enough space to comfortably manipulate the hand-held interface in the space behind the mirror. Fig. 5. In normal object viewing, the focal point and the vergence point of the eyes are the same, but focal point (display screen) and vergence point (virtual image) may not be the same. 4 Preliminary Study We conducted an empirical study with 41 college students (23 women) (age: M=18.9, SD = 1.46) to compare the usability of concrete and virtual models as aids for stu- dents when performing diagram relation tasks in organic chemistry. None of the par- ticipants had taken organic chemistry. All participants had normal, or corrected to normal vision. A within-subject design was used to control for individual differences among the participants and to test for transfer of learning from one model type to the other. Be- fore beginning the trials, students watched a 10-minute instructional video explaining the conventions of the models, how to find and understand important features of the models, how to draw each type of diagram, the correspondence between the colors on the models and the parts of the diagrams, and how to align the model to each of the three diagrams. Each participant completed two tasks, a diagram matching task and 38 diagram completion task using both the concrete and virtual models. In the first task, students had to perform global and local rotations of the model to make it match one of the three diagram types shown in Figure 1. In the other, they were given a partially completed diagram template and had to fill in the missing atomic subgroups. Half of the participants performed the tasks first with the virtual model then switched to the concrete model and half performed the tasks first with the concrete models then switched to the virtual models. All participants completed problems with both model types in counter-balanced order,. For each model type, participants completed 3 prac- tice and 12 orientation matching trials followed by 3 practice and 12 diagram comple- tion trials. Eight organic molecules having 3-, 4-, or 5-carbon backbone were repre- sented by the models and diagrams. Dependent measures included accuracy and re- sponse time on the matching and completion tasks. Separate ANOVAs were conducted to analyze the data (Fig. 6). We observed no significant difference in accuracy between the two model types on either the match- ing, F(1, 40) = .138, p = .71, or the completion task, F(1, 40) = .739, p = .40. Howev- er, participants were significantly faster at performing the diagram matching task, F(1, 40) = 5.12, p = .03, when using the virtual models. Participants did not differ on speed on the diagram completion task, F(1, 40) = 2.01, p = .16. Fig. 5. Results of the diagram matching (top) and diagram completion (bottom) for accuracy (left) and response time (right). Error bars use standard error. 39 5 Discussion We speculate that the superior efficiency in matching the diagrams with the virtual model is because it constrains interactivity to the most task-relevant manipulations of the model. For each task, identifying the backbone of the molecule is an important step because this feature serves as a visual reference when orienting the model before determining the order of the molecular subunits. Rotations of groups of atoms around this backbone are also necessary to produce the different conformations of the model that are represented by the different diagrams, In the virtual model, the long axis of the interface was congruent with the backbone of the model making this bond more salient once the interface was moved. Furthermore, a local rotation of the two halves of the interface rotated the bond forming the carbon backbone in the virtual model, and rotations around the other bonds (which are not relevant to the task) were not possible. In contrast, the bond representing the backbone was not particularly salient in the physical models, and rotations around all of the bonds were possible. The constraints of the interface therefore facilitated efficient visual interrogation and task- relevant manipulation of the virtual model, which lead to faster use times. Although these results are preliminary, they are just the first step in a program of research to investigate the utility of virtual and concrete models as learning aids for diagrammatic reasoning in chemistry. Overall, they suggest a benefit for using virtual models when teaching students about the relation between 3D and 2D representations in chemistry. In addition, the equipment provides a flexible and scalable system to systematically study the perceptual cues and cognitive conditions important for the design of effective virtual learning aids. Acknowledgments. We would like to acknowledge the research assistance of Niklas E. Erricson, Ted Hsu, David Sanosa, Misty L. Schubert, Emily R. Steiner, and Teresa van Osdol, as well as the technical support of Jerry Tietz. This research was supported by the National Academy of Education/Spencer Postdoctoral Fellowship Program and grants 0722333 and 1008650 from the U.S. National Science Foundation. 6 References 1. Francoeur, E., & Segal, J. (2004). From model kits to interactive computer graphics. In S. de Chadarevian & N. Hopwood (Eds.), Models: Third Dimension of Science. Stanford: Stanford University Press. 2. de Chadarevian, S. (2004). Models and the making of molecular biology. In S. de Chadar- evian & N. Hopwood (Eds.), Models: Third Dimension of Science. Stanford: Stanford University Press. 3. Stull, A. T., Hegarty, M. Stieff, M., & Dixon, B. (2010). Does manipulating molecular models promote representation translation of diagrams in chemistry? In A. K. Goel, M. Jamnik, & N. H. Narayanan (Eds.), Diagrams 2010, LNAI 6170. (pp. 338-344). Heidel- berg, Germany: Springer. 40 4. Ishikawa, T., & Kastens, K. A. (2004). Envisioning large geologic structures from field observations; an experimental study; geological society of america, 2004 annual meeting. Abstracts with Programs - Geological Society of America, 36(5), 156-157. 5. Kozma, R. B. (2003). The material features of multiple representations and their cognitive and social affordances for science understanding. Learning and Instruction, 13, 205-226. 6. Novick L. R., & Catley, K. M. (2007). Understanding phylogenies in biology: The influ- ence of a Gestalt perceptual principle. Journal of Experimental Psychology: Applied, 13, 197-223. 7. Kozma, R. B., & Russell, J. (1997). Multimedia and understanding: Expert and novice re- sponses to different representations of chemical phenomena. Journal of Research in Sci- ence Teaching, 34, 949-968. 8. Kozma, R. B., Chin, E., Russell, J., & Marx, N. (2000). The role of representations and tools in the chemistry laboratory and their implications for chemistry learning. Journal of the Learning Sciences, 9, 105-144. 9. Cheng, M., & Gilbert, J. K. (2009). Towards a better utilization of diagrams in research in- to the use of representative levels of chemical education. In J. K. Gilbert, & D. Treagust (Eds.), Multiple representations in chemical education: Models and modeling in science education. Dordrecht: Springer. 10. Goodwin, W. M. (2008). Structural formulas and explanation in organic chemistry. Foun- dations of Chemistry, 10, 117-127. 11. Hoffmann, R., & Laszlo, P. (1991). Representation in chemistry. Angewandte Chemie In- ternational Edition, 30, 1-16. 12. Wu, H.-K., & Shah, P. (2004). Exploring visuospatial thinking in chemistry learning. Sci- ence Education, 88, 465-492. 13. Stieff, M. (2011). When is a molecule three dimensional? A task-specific role for imagistic reasoning in organic chemistry. Science Education, 92, 310-336. 14. Stieff, M., & Raje, S. (2010). Expertise algorithmic and imagistic problem solving strate- gies in advanced chemistry. Spatial Cognition & Computation. 10, 53-81. 15. Bethell-Fox, C. E., & Shepard, R. N. (1988). Mental rotation: Effects of stimulus complex- ity and familiarity. Journal of Experimental Psychology, Human Perception and Perfor- mance, 14, 12-23. 16. Kirsh, D. (1995a). Complementary Strategies: Why we use our hands when we think. In J. D. Moore & J. F. Lehman (Eds.), Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society, Mahway, NJ: Lawrence Erlbaum, (pp. 212-217). 17. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429-433. 18. Hoffman, D. M., Girshick, A. R., Akeley, K., & Banks, M. S. (2008). Vergence- accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision, 8, 1-30. 41