=Paper= {{Paper |id=Vol-2092/paper12 |storemode=property |title=Do Audience Response Systems Influence Learning Style? |pdfUrl=https://ceur-ws.org/Vol-2092/paper12.pdf |volume=Vol-2092 |authors=Clemens H. Cap,Edith Braun |dblpUrl=https://dblp.org/rec/conf/delfi/CapB17 }} ==Do Audience Response Systems Influence Learning Style?== https://ceur-ws.org/Vol-2092/paper12.pdf

Carsten Ullrich, Martin Wessner (Eds.): Proceedings of DeLFI and GMW Workshops 2017
Chemnitz, Germany, September 5, 2017

Do Audience Response Systems Influence Learning Style?

An Empirical Study with First Term Students Based on Tweedback

Clemens H. Cap1, Edith Braun2

Abstract: We study the long-term effects of an audience response system (ARS) on the approach
to learning of computer science students. The ARS Tweedback is used continuously throughout a
one-term freshman lecture series on communications. ARS use is accompanied by three series of
questionnaires for determining learning styles. The results are evaluated and interpreted.
Conclusions are drawn for long-term effects of using these particular digital learning systems.
Keywords: Audience Response Systems; Approach to Learning.

1 Introduction
Audience response systems (ARS) are small devices (“clickers”') or Internet-based
services for, preferably, mobile devices, which are supposed to improve teacher and
student interaction in large lectures [FM06]. The most common features of an ARS
consist of multiple-choice or estimation tests, of a chatwall for questions and
discussions, and of feedback possibilities with regard to lecturing speed and quality
[Fe12]. Earlier ARS consisted of small devices which were distributed to students and
restricted to multiple-choice answers. With the ubiquitous availability of mobile phones,
more elaborate solutions make use of the wide range of interaction possibilities of
Internet portals.
There is plenty of research regarding the advantages and disadvantages of using mobile
phones in class and on the benefits and didactical aspects of ARS in general [PGB12].
However, not much is known on how the use of an ARS influences the approach to
learning of a student in the long run.
Question 1: Does the use of an ARS noticeably change the approach of students to their
learning?

1 Universität Rostock, Lehrstuhl für Informations- und Kommunikationsdienste, Albert Einstein Strasse 22,

18059 Rostock, Deutschland, clemens.cap@uni-rostock.de
2 Universität Kassel, INCHER – International Centre for Higher Education Research, Mönchebergstrasse 17,

34109 Kassel, Deutschland, edith.braun@incher.uni-kassel.de
Clemens H. Cap und Edith Braun

Question 2: Do students adapt their level of cooperation in response to attending a
lecture where regular and heavy use is made of an ARS to promote feedback and open
debate in class?
Both questions would be particularly of interest in the case of freshman students who,
coming from high school to university, still have to define their new way of learning in
the novel environment they are facing.
It is the intention of this note to provide some initial and exploratory answers to this
question based on an empirical survey and to draw some general conclusions on the use
of ARS systems.
Our contribution is structured as follows: Section 2 describes the employed ARS
Tweedback, its technical features and its didactical use throughout the lecture. Section 3
describes the questionnaire and the data gathering. Section 4 reports on the results and
finally we draw some conclusions.

2 Audience Response System and Lecture Didactics
Our empirical study was conducted in the freshman lecture on communications and data
security. The lecture is held for three hours per week for first-term students of computer
science and related areas. An accompanying exercise session ensures that the students
stay in constant touch with the subject material. Attendance was 145 students (formal
module registration), active lecture and exercise participation was 121, and 76 students
were taking the exam. The lecturer is one of the co-developers of the ARS Tweedback
and in this role is an active user of this system in class.
Tweedback is an Internet-portal based ARS which offers a chatwall, multiple choice
quizzes and a feedback channel when issues with understanding the lecturer arise
[VGC13]. In a 90 minute lecture unit, some 3-7 quizzes were used. Two didactical
motivations accompany quiz use. The first variant comprises a possibility for the
students to revise earlier material, to give the lecturer some impression about student
progress and to provide a quick overview on student knowledge. The second variant
consists of a provocative choice test, where in response to a single question either all
provided answers were wrong or a single choice would have to be selected out of several
options, all of which were correct. This variant was used to provide an entry into in-class
debate, prompting students to give their reasons for or against certain answer options.
The chatwall received some 10-20 contributions on a wide range of topics, from topical
and organisational questions to remarks, more banal comments and irrelevant spam. (For
a comprehensive and systematic analysis of more than 12.000 chat contributions with
this system, a separate publication is in preparation). The system currently is available
for registration-free inspection and use at http://twbk.de and http://twbk.io.
Do Audience Response Systems Influence Learning Style

3 Method
Throughout the term a survey was conducted at two times t1 and t2, using the same
questionnaire. t1 was 6 weeks after the beginning of the term, a moment when most
students had accustomed themselves with university, with the Tweedback system and its
use. t2 was at the final exam, three weeks after the end of the lecture period, where the
students had experienced learning for that exam.
The questionnaire (see appendix) is based on three aspects:
1. A scale of deep learning from [En97]. Deep learning characterizes a learning
style which is predominated by the motivation to deeply understand new topics.
Higher values indicate student willing to connect and apply the learned topics.
Students invest more time in learning the higher they are characterized by deep
learning style.
2. A scale of cooperation skills from the BEvaKomp inventory [Br08]. Students
rate their own ability to work with others and to be able to contribute to a
shared duty.
3. A self-rated 3-stage scale in which the students report how often they used the
ARS.
Both scales have a good Cronbach alpha parameter (0.78 for the deep learning scale and
0.84 for the cooperation scale). This parameter can be interpreted as the correlation of
tests which measure the same psychometric construct, with a value in [0.8, 0.9] generally
considered as good and in [0.7, 0.8] as acceptable [GM03]. The sample size is small.
Therefore, we report effect size to judge the mean differences; calculating Cohens d as
effect size. In general, d< 0.2 are small, d< 0.5 are median size, and d< 0.8 are bigger
effects.
The answers to the individual questions were designed as identical 5-stage Likert scales.
We treat the ordinal Likert scales as metric scales, as it is the usual custom in
educational research [Bü11].
Questionnaires from t1 and t2 were linked by a pseudo-hash function preserving the
students privacy. Due to data protection restrictions the marks obtained in the exam are
not known and the questionnaires were sampled from the students which were present at
the times t1 and t2. The students were informed about the goals of the data gathering.

4 Results
In a first step the arithmetic means of the subscales at the two measurements have been
checked. Students report slightly higher values in cooperation skills (mean for t1 is 3.94,
mean for t2 is 4.05) as well as in deep learning (mean for t1 is 3.39, mean for t2 is 3.64),
Clemens H. Cap und Edith Braun

even if only the increase in deep learning is statistically significant (p< 0.05).
Furthermore, the effect size show a small effect in increase in cooperation (d= .17), and a
median effect size for deep learning (d= .47).Therefore, students have improved their
skills, especially in deep learning, while they attended the course, see Fig. 1.

Fig. 1: Overview on cooperation and deep learning scores at t1 and t2 with error bars.

In a second step, we looked at the frequency of tool use. How much did the tool support
the increase of the deep learning style of the students? Therefore, we grouped the
students by three categories:
1. Students who said they never used the tool.
2. Students, who said they occasionally used the tool.
3. Students, who said they often used the tool.
All three groups of students report an increase at t2 in deep learning. Especially the
students, who used the tool very often, report the highest increase in deep learning, see
Fig. 2.
Finally, in a third step, we looked at the impact of tool use in the reported cooperation
skills. Again, the three groups of students were compared. In the area of cooperation
skills, converse progresses can be observed. Students, who often used the tools, improve
their cooperation skills. In comparison, students, who never used the tool, report a loss of
cooperation skills between the beginning and the end of the term. Students, who used the
tool occasionally, are somewhere between and almost no change can be observed, see
Fig. 3.
We observed that the cooperation skills of the non-users declined, but we currently have
no explanation for this.
Do Audience Response Systems Influence Learning Style

Fig. 2: Deep Learning Scores.

deep t1 deep t2 coop t1 coop t2

never 3,59 3,71 4,10 3,88

occasionally 3,35 3,45 3,85 3,90

often 3,38 3,68 4,01 4,18

Tab. 1: Score values

Fig. 3: Cooperation Scores.
Clemens H. Cap und Edith Braun

Of course, the observed changes may also have been caused by other effects, such as the
lecturer. The ARS is used only as one element as part of a larger teaching context. Still,
there are clear differences depending on and correlating with tool use. However, our
study contained no interventions or randomizations, which would enable us to
distinguish correlation from causation.

5 Discussion
This contribution took a preliminary look at the connection between the use of an ARS
and the approach of students to their deep learning techniques as well as the self-reported
cooperation skills. The empirical data were gathered at three and used at two different
time slots, so real changes can be observed.
Our questionnaires indicate an increase of deep learning style and a smaller increase in
cooperation skills. Furthermore, a closer look was taken by comparing students, who
used the tool never, to students who used it more often. The observed increase in
cooperation skills and in deep learning style can be explained statistically by the amount
of using Tweedback. Students, who used the tool often, report the highest increase in
their skills. This supports the overall thesis of our contribution that ARS use supports
and affects learning styles.

6 Appendix: Questionnaire
The appendix contains the phrases of the questionnaire which were used to identify deep
learning and cooperation skill scores in their original, German form.

6.1 Deep Learning

1. In der Regel versuche ich selbst die Bedeutung des Gelernten zu verstehen.
2. Während ist lese, lege ich kurze Pausen ein, in denen ich reflektiere, was ich
gelernt habe.
3. Bevor ich anfange eine Aufgabe zu bearbeiten, versuche ich zuerst den
Zusammenhang zu verstehen.
4. Ich versuche das Gelernte von verschiedenen Lehrveranstaltungen miteinander
in Verbindung zu setzen.
5. Ideen aus Fachbüchern oder wissenschaftlichen Artikel lösen manchmal lange
Gedankengänge bei mir aus.
Do Audience Response Systems Influence Learning Style

6. Ich mag es meine Gedanken zu wälzen, auch wenn ich manchmal nicht wirklich
weit komme.
7. Wenn ich ein Fachbuch oder einen wissenschaftlichen Artikel lese, versuche ich
selbstständig herauszubekommen, was genau die Autoren gemeint haben.
8. Wenn ich ein neues Thema anfange zu bearbeiten, versuche mir vorzustellen,
wie das Thema zu anderen in Verbindung steht.

6.2 Cooperation Skills

1. Es gelingt mir leicht, mich an der Aufgabenverteilung innerhalb einer
Arbeitsgruppe zu beteiligen.
2. Es fällt mir leicht, meine eigenen Vorschläge in einer Arbeitsgruppe auch mal
zurückzunehmen.
3. Ich kann mich gut für eine konstruktive Arbeitsatmosphäre innerhalb von
Teams einsetzen.
4. Ich kann mich gut an die Absprachen innerhalb einer Arbeitsgruppe halten.
5. Ich identifiziere mich mit dem Ergebnis einer Arbeitsgruppe.

References
[Br08] Braun, E. et al.: Das Berliner Evaluationsinstrument für selbsteingeschätzte,
studentische Kompetenzen (BEvaKomp). Diagnostica, 54(1):30–42, 2008.
[Bü11] Bühner, M.: Einführung in die Test- und Fragebogenkonstruktion. Pearson
Deutschland GmbH, 2011.
[En97] Entwistle, N. J.: The approaches and study skills inventory for students (ASSIST).
1997. Edinburgh: Centre for Research on Learning and Instruction, University of
Edinburgh.
[Fe12] Feiten, Linus et al.: SMILE – SMARTPHONES IN LECTURES - Initiating a
Smartphone-based Audience Response System as a Student Project. In: 4th
International Conference on Computer Supported Education. pp. 288–293, 2012.
[FM06] Fies, Carmen; Marshall, Jill: Classroom response systems: A review of the literature.
Journal of Science Education and Technology, 15(1):101–109, 2006.
[GM03] George, D.; Mallery, P.: SPSS for windows step by step: A sample guide & reference.
Boston: Allyn & Bacon, 2003.
[PGB12] Pohl, Alexander; Gehlen-Baum, Vera; Bry, François: Enhancing the Digital
Backchannel Backstage on the Basis of a Formative User Study. International Journal
of Emerging Technologies in Learning, 7(1):33–41, 2012.
Clemens H. Cap und Edith Braun

[VGC13] Vetterick, Jonas; Garbe, Martin; Cap, Clemens H: Tweedback: A Live Feedback
System for Large Audiences. In: 5th International Conference on Computer Supported
Education (CSEDU 2013). SciTePress - Science and and Technology Publications, pp.
194–198, 2013.