Carsten Ullrich, Martin Wessner (Eds.): Proceedings of DeLFI and GMW Workshops 2017 Chemnitz, Germany, September 5, 2017 Do Audience Response Systems Influence Learning Style? An Empirical Study with First Term Students Based on Tweedback Clemens H. Cap1, Edith Braun2 Abstract: We study the long-term effects of an audience response system (ARS) on the approach to learning of computer science students. The ARS Tweedback is used continuously throughout a one-term freshman lecture series on communications. ARS use is accompanied by three series of questionnaires for determining learning styles. The results are evaluated and interpreted. Conclusions are drawn for long-term effects of using these particular digital learning systems. Keywords: Audience Response Systems; Approach to Learning. 1 Introduction Audience response systems (ARS) are small devices (“clickers”') or Internet-based services for, preferably, mobile devices, which are supposed to improve teacher and student interaction in large lectures [FM06]. The most common features of an ARS consist of multiple-choice or estimation tests, of a chatwall for questions and discussions, and of feedback possibilities with regard to lecturing speed and quality [Fe12]. Earlier ARS consisted of small devices which were distributed to students and restricted to multiple-choice answers. With the ubiquitous availability of mobile phones, more elaborate solutions make use of the wide range of interaction possibilities of Internet portals. There is plenty of research regarding the advantages and disadvantages of using mobile phones in class and on the benefits and didactical aspects of ARS in general [PGB12]. However, not much is known on how the use of an ARS influences the approach to learning of a student in the long run. Question 1: Does the use of an ARS noticeably change the approach of students to their learning? 1 Universität Rostock, Lehrstuhl für Informations- und Kommunikationsdienste, Albert Einstein Strasse 22, 18059 Rostock, Deutschland, clemens.cap@uni-rostock.de 2 Universität Kassel, INCHER – International Centre for Higher Education Research, Mönchebergstrasse 17, 34109 Kassel, Deutschland, edith.braun@incher.uni-kassel.de Clemens H. Cap und Edith Braun Question 2: Do students adapt their level of cooperation in response to attending a lecture where regular and heavy use is made of an ARS to promote feedback and open debate in class? Both questions would be particularly of interest in the case of freshman students who, coming from high school to university, still have to define their new way of learning in the novel environment they are facing. It is the intention of this note to provide some initial and exploratory answers to this question based on an empirical survey and to draw some general conclusions on the use of ARS systems. Our contribution is structured as follows: Section 2 describes the employed ARS Tweedback, its technical features and its didactical use throughout the lecture. Section 3 describes the questionnaire and the data gathering. Section 4 reports on the results and finally we draw some conclusions. 2 Audience Response System and Lecture Didactics Our empirical study was conducted in the freshman lecture on communications and data security. The lecture is held for three hours per week for first-term students of computer science and related areas. An accompanying exercise session ensures that the students stay in constant touch with the subject material. Attendance was 145 students (formal module registration), active lecture and exercise participation was 121, and 76 students were taking the exam. The lecturer is one of the co-developers of the ARS Tweedback and in this role is an active user of this system in class. Tweedback is an Internet-portal based ARS which offers a chatwall, multiple choice quizzes and a feedback channel when issues with understanding the lecturer arise [VGC13]. In a 90 minute lecture unit, some 3-7 quizzes were used. Two didactical motivations accompany quiz use. The first variant comprises a possibility for the students to revise earlier material, to give the lecturer some impression about student progress and to provide a quick overview on student knowledge. The second variant consists of a provocative choice test, where in response to a single question either all provided answers were wrong or a single choice would have to be selected out of several options, all of which were correct. This variant was used to provide an entry into in-class debate, prompting students to give their reasons for or against certain answer options. The chatwall received some 10-20 contributions on a wide range of topics, from topical and organisational questions to remarks, more banal comments and irrelevant spam. (For a comprehensive and systematic analysis of more than 12.000 chat contributions with this system, a separate publication is in preparation). The system currently is available for registration-free inspection and use at http://twbk.de and http://twbk.io. Do Audience Response Systems Influence Learning Style 3 Method Throughout the term a survey was conducted at two times t1 and t2, using the same questionnaire. t1 was 6 weeks after the beginning of the term, a moment when most students had accustomed themselves with university, with the Tweedback system and its use. t2 was at the final exam, three weeks after the end of the lecture period, where the students had experienced learning for that exam. The questionnaire (see appendix) is based on three aspects: 1. A scale of deep learning from [En97]. Deep learning characterizes a learning style which is predominated by the motivation to deeply understand new topics. Higher values indicate student willing to connect and apply the learned topics. Students invest more time in learning the higher they are characterized by deep learning style. 2. A scale of cooperation skills from the BEvaKomp inventory [Br08]. Students rate their own ability to work with others and to be able to contribute to a shared duty. 3. A self-rated 3-stage scale in which the students report how often they used the ARS. Both scales have a good Cronbach alpha parameter (0.78 for the deep learning scale and 0.84 for the cooperation scale). This parameter can be interpreted as the correlation of tests which measure the same psychometric construct, with a value in [0.8, 0.9] generally considered as good and in [0.7, 0.8] as acceptable [GM03]. The sample size is small. Therefore, we report effect size to judge the mean differences; calculating Cohens d as effect size. In general, d< 0.2 are small, d< 0.5 are median size, and d< 0.8 are bigger effects. The answers to the individual questions were designed as identical 5-stage Likert scales. We treat the ordinal Likert scales as metric scales, as it is the usual custom in educational research [Bü11]. Questionnaires from t1 and t2 were linked by a pseudo-hash function preserving the students privacy. Due to data protection restrictions the marks obtained in the exam are not known and the questionnaires were sampled from the students which were present at the times t1 and t2. The students were informed about the goals of the data gathering. 4 Results In a first step the arithmetic means of the subscales at the two measurements have been checked. Students report slightly higher values in cooperation skills (mean for t1 is 3.94, mean for t2 is 4.05) as well as in deep learning (mean for t1 is 3.39, mean for t2 is 3.64), Clemens H. Cap und Edith Braun even if only the increase in deep learning is statistically significant (p< 0.05). Furthermore, the effect size show a small effect in increase in cooperation (d= .17), and a median effect size for deep learning (d= .47).Therefore, students have improved their skills, especially in deep learning, while they attended the course, see Fig. 1. Fig. 1: Overview on cooperation and deep learning scores at t1 and t2 with error bars. In a second step, we looked at the frequency of tool use. How much did the tool support the increase of the deep learning style of the students? Therefore, we grouped the students by three categories: 1. Students who said they never used the tool. 2. Students, who said they occasionally used the tool. 3. Students, who said they often used the tool. All three groups of students report an increase at t2 in deep learning. Especially the students, who used the tool very often, report the highest increase in deep learning, see Fig. 2. Finally, in a third step, we looked at the impact of tool use in the reported cooperation skills. Again, the three groups of students were compared. In the area of cooperation skills, converse progresses can be observed. Students, who often used the tools, improve their cooperation skills. In comparison, students, who never used the tool, report a loss of cooperation skills between the beginning and the end of the term. Students, who used the tool occasionally, are somewhere between and almost no change can be observed, see Fig. 3. We observed that the cooperation skills of the non-users declined, but we currently have no explanation for this. Do Audience Response Systems Influence Learning Style Fig. 2: Deep Learning Scores. deep t1 deep t2 coop t1 coop t2 never 3,59 3,71 4,10 3,88 occasionally 3,35 3,45 3,85 3,90 often 3,38 3,68 4,01 4,18 Tab. 1: Score values Fig. 3: Cooperation Scores. Clemens H. Cap und Edith Braun Of course, the observed changes may also have been caused by other effects, such as the lecturer. The ARS is used only as one element as part of a larger teaching context. Still, there are clear differences depending on and correlating with tool use. However, our study contained no interventions or randomizations, which would enable us to distinguish correlation from causation. 5 Discussion This contribution took a preliminary look at the connection between the use of an ARS and the approach of students to their deep learning techniques as well as the self-reported cooperation skills. The empirical data were gathered at three and used at two different time slots, so real changes can be observed. Our questionnaires indicate an increase of deep learning style and a smaller increase in cooperation skills. Furthermore, a closer look was taken by comparing students, who used the tool never, to students who used it more often. The observed increase in cooperation skills and in deep learning style can be explained statistically by the amount of using Tweedback. Students, who used the tool often, report the highest increase in their skills. This supports the overall thesis of our contribution that ARS use supports and affects learning styles. 6 Appendix: Questionnaire The appendix contains the phrases of the questionnaire which were used to identify deep learning and cooperation skill scores in their original, German form. 6.1 Deep Learning 1. In der Regel versuche ich selbst die Bedeutung des Gelernten zu verstehen. 2. Während ist lese, lege ich kurze Pausen ein, in denen ich reflektiere, was ich gelernt habe. 3. Bevor ich anfange eine Aufgabe zu bearbeiten, versuche ich zuerst den Zusammenhang zu verstehen. 4. Ich versuche das Gelernte von verschiedenen Lehrveranstaltungen miteinander in Verbindung zu setzen. 5. Ideen aus Fachbüchern oder wissenschaftlichen Artikel lösen manchmal lange Gedankengänge bei mir aus. Do Audience Response Systems Influence Learning Style 6. Ich mag es meine Gedanken zu wälzen, auch wenn ich manchmal nicht wirklich weit komme. 7. Wenn ich ein Fachbuch oder einen wissenschaftlichen Artikel lese, versuche ich selbstständig herauszubekommen, was genau die Autoren gemeint haben. 8. Wenn ich ein neues Thema anfange zu bearbeiten, versuche mir vorzustellen, wie das Thema zu anderen in Verbindung steht. 6.2 Cooperation Skills 1. Es gelingt mir leicht, mich an der Aufgabenverteilung innerhalb einer Arbeitsgruppe zu beteiligen. 2. Es fällt mir leicht, meine eigenen Vorschläge in einer Arbeitsgruppe auch mal zurückzunehmen. 3. Ich kann mich gut für eine konstruktive Arbeitsatmosphäre innerhalb von Teams einsetzen. 4. Ich kann mich gut an die Absprachen innerhalb einer Arbeitsgruppe halten. 5. Ich identifiziere mich mit dem Ergebnis einer Arbeitsgruppe. References [Br08] Braun, E. et al.: Das Berliner Evaluationsinstrument für selbsteingeschätzte, studentische Kompetenzen (BEvaKomp). Diagnostica, 54(1):30–42, 2008. [Bü11] Bühner, M.: Einführung in die Test- und Fragebogenkonstruktion. Pearson Deutschland GmbH, 2011. [En97] Entwistle, N. J.: The approaches and study skills inventory for students (ASSIST). 1997. Edinburgh: Centre for Research on Learning and Instruction, University of Edinburgh. [Fe12] Feiten, Linus et al.: SMILE – SMARTPHONES IN LECTURES - Initiating a Smartphone-based Audience Response System as a Student Project. In: 4th International Conference on Computer Supported Education. pp. 288–293, 2012. [FM06] Fies, Carmen; Marshall, Jill: Classroom response systems: A review of the literature. Journal of Science Education and Technology, 15(1):101–109, 2006. [GM03] George, D.; Mallery, P.: SPSS for windows step by step: A sample guide & reference. Boston: Allyn & Bacon, 2003. [PGB12] Pohl, Alexander; Gehlen-Baum, Vera; Bry, François: Enhancing the Digital Backchannel Backstage on the Basis of a Formative User Study. International Journal of Emerging Technologies in Learning, 7(1):33–41, 2012. Clemens H. Cap und Edith Braun [VGC13] Vetterick, Jonas; Garbe, Martin; Cap, Clemens H: Tweedback: A Live Feedback System for Large Audiences. In: 5th International Conference on Computer Supported Education (CSEDU 2013). SciTePress - Science and and Technology Publications, pp. 194–198, 2013.