EFFECTIVENESS OF NOTE-TAKING SKILLS AND STUDENT'S CHARACTERISTICS ON LEARNING PERFORMANCE IN ONLINE COURSES Nakayama, Minoru, Tokyo Institute of Technology, Ookayama 2-12-1 W9-107, Meguro, 152-8552 Tokyo, Japan, nakayama@cradle.titech.ac.jp Mutsuura, Kouichi, Shinshu University, Asahi 3-1-1, 390-8621 Matsumoto, Japan Yamamoto, Hiroh, Tokyo Institute of Technology, Ookayama 2-12-1, Meguro, 152-8552, Tokyo, Japan Abstract Note-taking activity was introduced into two types of university courses, a blended learning course and a fully online course, and causal relationships between student’s characteristics such as note- taking behaviour and learning performance were analysed. The objective was to improve learning activities in the online learning environment. Metrics such as personality, information literacy and note-taking skills have been commonly surveyed and analysed in courses such as the above. The differences between causal paths in the two courses were measured using the structural equation modelling technique. The contributions of all metrics to test scores were surveyed and analysed, and the factors which were significant were extracted. Keywords: Online course, Note-taking, Student characteristics, Structural Equation Modelling. 1 Introduction Recently many universities have employed the online learning environment to promote learning flexibility and effectiveness. This study evaluates the note-taking behaviour of participants in online courses to measure the effectiveness of the courses and to develop methods to assist participants (Nakayama et al. 2010, 2011b, 2012a, 2012b). Note-taking is a key activity for various types of learning, including online courses. The contents of notes taken indicate the learning progress of participants (Kiewra, 1985, 1989, Kiewra et al. 1995). The causal relationships between participants’ characteristics, note-taking behaviour and learning performance have been analyzed in prior research (Nakayama et al. 2011b, 2012a, 2012c). Surveys for blended learning course and fully online course have been conducted, and the impact of the two learning environments on learning behaviour has also already been analysed (Nakayama et al. 2011b, 2012a). These results may suggest that the improvement of courses and the development of a support system for participants is necessary. To extract factors which have a common level of effectiveness between two courses, or are effective in one of the two, the relationships between note-taking, participants characteristics and learning performance were examined. Factors for note-taking skills were identified using the responses to two sets of questionnaire, and the relationships between these characteristics, based on the factors and performance in tests, were examined. The following topics are addressed in this paper. 13 - The structure of factors of note taking skills are examined using two sets of responses, and factors common to blended and fully online courses are extracted. - The note-taking behaviour and characteristics of participants are compared between the two courses. - The causal relationships between participant’s characteristics and learning achievement are measured and compared between the two courses. 2 Method 2.1 Courses 2.1.1 Blended learning course A blended learning course was conducted using a distance education system with ordinary face-to-face classroom sessions (Nakayama et al. 2010). All participants were able to use online tests used as part of the learning management system (LMS) of the course. Participants were encouraged to take online tests, and they could take tests repeatedly until they were satisfied with their scores. The LMS recorded the scores of the final test, and these test scores were used in calculating overall course grades. The total number of valid participants in the course was 40. 2.1.2 Fully online learning course Students studied using online material which consisted of slides and oral explanations of the content. The presentation slides presented the content together with audio files automatically, as this simulated face-to-face sessions. Participants were asked to study one module per week, and weekly proctored confirmation tests were conducted. The lecturer could set participants' pace of study and monitor their progress. Also, participants were encouraged to take regular online tests to benchmark their progress (Nakayama et al. 2011b, 2012a). The total numbers of valid participants in the fully online course was 53. 2.2 Note-taking assessment All participants in both courses were required to present their notebooks during most modules. The lecturer assessed these individual notes every week. The references for the evaluations were the notes of the lecturer, which contained fundamental information used in the lecturer's presentations and slides. The notes were evaluated using a set scale (Nakayama et al. 2010, 2011a), and assessed as Good, Fair or Poor and recorded every week. 14 2.3 Characteristics of participants The indices related to participant’s characteristic have been previously surveyed using existing constructs (Nakayama et al, 2007, 2008). These constructs are Personality (Goldberg 1999; IPIP 2004), Information Literacy (Fujii 2007) and a degree of Learning Experience (Nakayama et al. 2007). These metrics were calculated as factor scores from participants’ responses to questionnaires, using the factor loading metrices. Personality: The personalities of participants were measured using a public domain item pool, the International Personality Item Pool (IPIP) inventory (IPIP 2004). This inventory is based on a five factor model which was proposed by Goldberg (Goldberg 1999), consisting of ``Extroversion'' (IPIP-1), ``Agreeableness'' (IPIP-2), ``Conscientiousness'' (IPIP-3), ``Neuroticism'' (IPIP-4) and ``Openness to Experience'' (IPIP-5). Information Literacy: The survey inventories were originally developed by Fujii (2007). The survey construct consisted of 32 question items, and 8 factors were extracted, as follows: interest and motivation, fundamental operational ability, information collecting ability, mathematical thinking (reasoning) ability, information control ability, applied operational ability, attitude, and knowledge and understanding. These 8 factors can be summarized as two secondary factors: operational skills (IL-1) and attitudes toward information literacy (IL-2) (Nakayama et al. 2008). Learning experience: A construct consists of a 10-item Likert-type questionnaire was used to measure learning experience. Three factors were extracted, as follows: Factor 1 (LE-F1) -overall evaluation of the e-learning experience, Factor 2 (LE-F2) -learning habits, and Factor 3 (LE-F3) -learning strategies (Nakayama et al. 2007). 2.4 Survey of note-taking skills Note-taking skills may affect learning achievement (Nye et al. 1984). To evaluate the note-taking skills of participants, a set of survey questionnaires was developed using a Likert scale (Nakayama et al. 2011). Three factors were extracted, as follows: NT-F1 -Recognizing note taking functions, NT-F2 -Methodology of utilizing notes, NT-F3 -Presentation of notes. This construct is often used to survey note-taking skills with minor revisions (Nakayama et al. 2011, 2012a, 2012c). The details of the question items will be explained in the results. 2.5 Procedure for Causal analysis Causal analysis was applied to a set of metrics extracted from the fully online course. The causal relationship model was designed as a structure where learning performance is affected by participants' characteristics and note-taking behaviour (Nakayama et al. 2011b, 2012a). The model was developed step by step. Two subset models were created using the two sets of data from the courses. The first subset model was of characteristics and note-taking behaviour, and the second subset model was of the relationship between note-taking skills, learning experience and test scores. Finally, a unified model was created using the two subset models, and the causal relationships of metrics were examined. The calculations were conducted using AMOS structural equation modelling software (Toyoda, 2007; Kline, 2005). 15 3 Results 3.1 Note-taking assessment The results of note-taking assessment surveys are summarized in Figure 1(a) for the blended learning course and 1(b) for the fully online course. The percentages of note assessment levels are illustrated across the weeks of the course. Though the frequencies of the assessment levels are almost the same for the two courses, the balance between the "Good" and "Fair" percentages varies slightly. The percentage of "Good" is the highest in the blended learning course, while the percentage of "Fair" is the highest in the fully online course, as most participants have reproduced the presented content in their notes. When mathematical equations were explained, the difference in frequencies was relatively small. These tendencies were confirmed in a previous report about surveys of both types of courses (Nakayama et al. 2011a, 2012b). Figure 1: Grade percentages of note-taking assessments in (a) a Blended course (Nakayama et al, 2010), (b) a Fully online course (Nakayama et al. 2011b) Overall scores of weekly ratings of notes was calculated for each student and course. The overall scores were used to divide participants into two groups consisting of high (High) or low (Low) levels of note-taking assessment. 3.2 Factor structure of Note-taking skills The factor structure was extracted from the survey data for the fully online course (Nakayama et al. 2011b, 2012a). The survey data for the blended course was merged with it and an exploratory factor analysis was conducted again using Promax rotation. The factor loading matrix for 17 question items is summarized in Table 1 (Nakayama et al. 2012c). Three question items from the two courses were excluded, when being compared to the previous results for the fully online course. 16 The results again show the three factor structure, such as NT-F1 -Recognizing note taking functions, NT-F2 -Methodology of utilizing notes, NT-F3 -Presentation of notes. There is some correlation among factors since the structure was derived using Promax rotation. Table 1: Questionnaire for note-taking skills and factor loading (Nakayama et al. 2012c) Figure 2: Factor scores of note-taking skills between two note-taking assessment groups in both Blended and Fully online courses. (Nakayama et al. 2012c) 3.3 Effectiveness of learning styles and note-taking assessments Three factor scores were calculated as a mean of the major contributing responses for each factor using the factor loading matrix as mentioned above. The factor scores are compared between high and low groups of note-taking assessment in the two courses. The results are summarized in Figure 2; the 17 left side shows the blended course and the right side shows the fully online course. The means of first factor score (NT-F1) are higher than the median in both note-taking groups. According to Figure 2, there are some differences in factor scores. To determine the effectiveness of the courses and note- taking assessment groups, two-way ANOVA (analysis of variance) was conducted. The first factor is the course and the second factor is the group of note-taking assessments. In the results of the F-test for the first factor (NT-F1), the factor of the group is significant (p<0.01) while the factor of the course is not significant. For all variables of constructs, the same analysis was conducted. As a result, similar statistical effects such as the significance of the factor of the group (whether High or Low) was confirmed for the following variables: IPIP-3 (Conscientiousness), Information Literacy (IL-2: attitude), Note-taking skills (NT-F2: Methodology of utilizing notes), Learning Experience (LE-F2: Learning Habits), Online test scores, and Final exam scores (Nakayama et al. 2012a). These results suggest that most constructs were not influenced by course factors, i.e. whether blended learning or fully online, but the factor of the group was affected by the differences in the scores of the constructs. Table 2: ANOVA of note-taking factor 1 (NT-F1: Recognising note taking functions) Figure 3: Results of causal path analysis using both Blended and Fully online courses. 18 3.4 Causal analysis of note-taking activity between blended and fully Online courses The correlation analysis was conducted on the metrics for the above mentioned metrics, and some causal relationships were created step by step. Causal paths for all metrics, and two subsets of causal relationships were created. The first one indicates the causal paths from participant's characteristics and note-taking skill factors and assessments (Nakayama et al. 2012c). The second one was created from factors of note-taking skills (NT-F) to note-taking assessment (NT-A) and test scores (OT: Online test, FE: Final exams) (Nakayama et al. 2012c). According to the results of causal relationships such as subset analyses, the overall relationship was created using all metrics. Figure 3 shows the results, which are illustrated using major indices. Though the contributions of factors for learning experiences are relatively small, their effects are displayed in the figure in order to display their contribution clearly. According to the calculation, some significant paths were created with the causal coefficients. In Figure 3, coefficients which are not significant are indicated using (). This figure suggests that some factors of personality and information literacy scores affect learning experience, note assessment and test scores via factors of note-taking skills. In comparing path coefficients between two the courses, there are significant differences in the paths from NT-F3 (Presentation of notes) to NT-A (note-taking assessment), which is mentioned above, and also significant differences in one path from LE-F2 (Learning habits) to FE (Final exams) (p<0.05). For the fully online course, these path coefficients are significant, and relationships are clearly recognisable, such as the relationship between learning habits and final exam scores, and the negative relationship between note presentation skills and note assessments. These results can provide limited information, thus some ideas for improving learning are possible. Inspiring participant's consciousness of note-taking skills by taking into account the question items in Table 1 is one example. If these instructions improved the factor scores of learning habits, the scores of the final exams may be affected positively. In particular, the effectiveness of this may be expected in a fully online course, since some path coefficients from note-taking skills to both note-taking assessment and test scores are significant. As various factors such as work collaborations between students affect learning performance in a blended learning, most variables do not contribute significantly to other variables. However the participants have to encourage themselves to learn in a fully online course, as some factor scores of note-taking skills and the learning experience contribute significantly. The differences in factors contribution are confirmed between the two courses. Table 3: Results of ANOVA for scores of both online tests and final exams. 19 The above mentioned results suggest that scores of the final exams are not only affected by note- taking assessments but by various factors such as learning habits. To extract these factors, analysis of variance (ANOVA) for the scores of online tests and final exams was conducted using factors such as the two groups of the metrics and the two courses. Two groups (high and low) were created for all metrics using the means of scores as well as the means of the note-taking assessment groups. The analytical procedure is similar to the analysis used in Table 3. The results of the F values for scores of online tests and final exams are summarized in Table 5. The factors are two groups of metrics, the two courses and the interactions between the two factors. In evaluating the significance of the main effect, two of the three factors for note-taking skills and note- taking assessment are above the level of significance for online tests. For final exams, only extroversion (IPIP-1) and note-taking assessment are significant. The factors for the two courses are not significant for any of the metrics, and the interaction factor is significant for some metrics regarding information literacy and learning experience. Therefore, as these metrics may also affect learning performance in some way, they have to be considered when developing and introducing an online learning course. The statistical analysis shows that all factor scores affect test scores, online test scores and final exams. Once again, note-taking assessment groups significantly affect test scores of both types of exams. Therefore, note-taking directly provides learning benefits in the both tests. These points should be taught to all participants, in order to improve their education. 4 Conclusion To determine the causal relationships regarding note-taking behaviour between student's characteristics and learning performance for both blended and fully online courses, some common metrics in the two courses were surveyed and analysed. All participants were required to submit their notes in order to evaluate note-taking activity. According to the results of the analysis, a common factor structure in note-taking skills between the two online learning environments was confirmed. The causal relationships between various metrics of student's characteristics and their performance were examined. These results suggest that most metrics affect each other, and affect the test scores. There are some differences in path coefficients between the two courses. The contributions of all metrics to test scores were evaluated through analysis, and significant factors were extracted. The improvement or enhancement of these factors may contribute to learning performance in both leaning environments. This will be a subject of our further study. Acknowledgments This research was partially supported by the Japan Society for the Promotion of Science (JSPS), Grant-in-Aid for Scientific Research (B-22300281: 2010-2012). References Fujii, Y. (2007) “Development of a Scale to Evaluate the Information Literacy Level of Young People -Comparison of Junior High School Students in Japan and Northern Europe”, Japan Journal of Educational Technology, 30(4), pp.387-395 20 Goldberg L. R., “A Broad-Bandwidth, Public Domain, Personality Inventory Measuring the Lower- Level Facets of Several Five-Factor Models". In I. Mervielde, I. Deary, F. De Fruyt, & F. Ostendorf (Eds.), Personality Psychology in Europe, Vol. 7, pp. 7-28, Tilburg, The Netherlands: (Tilburg University Press, 1999). International Personality Item Pool, "A Scientific Collaboratory for the Development of Advanced Measures of Personality Traits and Other Individual Differences" (2001) Retrieved 27 October, 2004 from http://ipip.ori.org/. Kiewra, K. A. (1985) “Students' Note-Taking Behaviors and the Efficacy of Providing the Instructor's Notes for Review”, Contemporary Educational Psychology, 10, pp.378-386 Kiewra, K. A. (1989) “A Review of Note-Taking: The Encoding-Storage Paradigm and Beyond”, Educational Psychology Review, 1(2), pp.147-172 Kiewra, K.A., Benton, S.L., Kim, S., Risch, N., and Christensen, M. (1995) “Effects of Note-Taking Format and Study Technique on Recall and Relational Performance”, Contemporary Educational Psychology, 20, pp.172-187 Kline, R.B. (2005) Principles and practice of structural equation modelling, Second Edition, The Guilford Press, New York Kobayashi, K. (2005) “What limits the encoding effect of note-taking? A meta-analytic examination”, Contemporary Educational Psychology, 30, pp.242-262 Nakayama, M., Yamamoto, H., and Santiago, R. (2007) “The Impact of Learner Characteristics on Learning Performance in Hybrid Courses among Japanese Students”, The Electronic Journal of e- Learning, 5(3), pp. 195-206 Nakayama, M., Yamamoto, H., and Santiago, R. (2008) “Impact of Information Literacy and Learner Characteristics on Learning Behavior of Japanese Students in On line Courses”, International Journal of Case Method Research & Application, XX(4), pp. 403-415 Nakayama, M., Kanazawa, H., and Yamamoto, H. (2009) “Detecting Incomplete Learners in a Blended Learning Environment among Japanese University Students”, International Journal of Emerging Technology in Learning, 4(1), pp47-51 Nakayama, M., Mutsuura, K., and Yamamoto, H. (2010) “Effectiveness of Note Taking Activity in a Blended Learning Environment”, Proceedings of the 9th European Conference on E-Learning, pp.387-393. Nakayama, M., Mutsuura, K., and Yamamoto, H. (2011a) “Evaluation of student’s notes in a blended learning course”, International Journal of New Computer Architectures and their Applications, 1(4) pp.1080-1089. Nakayama, M., Mutsuura, K., and Yamamoto, H. (2011b) “Student’s Characteristics for Note Taking Activity in a Fully Online Course”, Proceedings of the 10th European Conference on E-Learning, pp.550-557. Nakayama, M., Mutsuura, K., and Yamamoto, H. (2012a) “Causal Analysis of Student’s Characteristics of Note-taking Activities and Learning Performance during a Fully Online Course”, The third International Workshop on Interactive Environments and Emergent Technologies for eLearning (IEETel), Proceedings of the IEEE 11th International Conference on Trust, Security and Privacy in Computing and Communication, pp.1924-1927. Nakayama, M., Mutsuura, K., and Yamamoto, H. (2012b) “Visualization analysis of student’s notes taken in a fully online learning environment”, Proceedings of the 16th International Conference of Information Visualisation, pp.434-439. Nakayama, M., Mutsuura, K., and Yamamoto, H. (2012c) “Note-taking skills and Student's characteristics in Online Courses”, Proceedings of the 11th European Conference on E-Learning, pp.388-396. Nye, P.A., Crooks, T.J., Powley, M. and Tripp, G. (1984) “Student note-taking related to university examination performance”, Higher Education, 13, pp.85-97 Toyoda, H. (2007) Kyobunsan kouzou bunseki (Structural equation Modelling) [AMOS Hen], Tokyo- shoseki, Tokyo Kline, R.B. (2005) Principles and Practice of Structural Equation Modeling, The Guilford Press, New York, USA 21