COMPS computer-mediated group discussion exercises are being added to a second-semester computer programming class. The class is a gateway for computer science and computer engineering students, where many students have difficulty succeeding well enough to proceed in their major. This paper reports on first results of surveys on student experience with the exercises. It also reports on the affective states observed in the discussions that are candidates for analysis of group functioning. As a step toward computer monitoring of the discussions, an experiment in using dialogue features to identify the gender of the participants is described.
The second Java programming class, GEEN 165, at North
Carolina A&T State University is a bottleneck for many
Computer Science and Computer Engineering students. As
an experiment in improving student learning and interest,
COMPS computer-mediated discussion exercises
GEEN 165 corresponds to the CS2 (second semester)
class in the ACM/IEEE curriculum
Lab-based computer programming classes traditionally permit unstructured group interaction. Students can talk to each other even as they require the students to write their own software. Therefore problemCopyright retained by the authors. solving discussions, where students respond to each other in normal dialogue fashion, are a natural addition to the lab component of a computer programming class.
NC A&T has migrated to an objects-later curriculum, meaning that CS2 contains more object concepts than the first semester CS1 class. The student exercises in this intervention are thus oriented toward object concepts.
Expressions of affect have three potential uses for this project. One is they are indications of emotional states that may effect student enthusiasm, self-efficacy and satisfaction. Another is they will be used in studies of group interaction. Finally, they may detectable by machine, contributing to an instructor's dashboard or other assessment of how well the group discussions are working.
COMPS is a web-delivered computer-mediated chat
environment
The exercises in this project involve students solving
multiple-choice questions. When implementing these as
group collaborations, we pay attention to three principles
that promote successful collaborative learning: a structure
or activity script for the students to follow, creative
interdependence, and individual accountability
Our collaborative inquiry learning exercises are in line
with current practices in Computer Supported
Collaborative Learning. A key concept is group cognition, where
different participants in a conversation contribute different
parts of the epistemic knowledge construction task. The
Virtual Math Teams project, where students solve math
problems through computer-mediated chat, has
documented this phenomenon
The implication for COMPS technology is that monitoring the health of student conversations could be informed by a) whether students are talking to each other, b) whether they are engaging in reasoning activities.
Group exercises address many of the components of
student interest. Interest refers to an individual’s
psychological inclination to participate in particular content over time
Recently
Viewed in this light, group exercises should address student motivation issues through social interaction at the same time as they address learning of concepts through group cognition. The exercises are constructed so that students engage with other students, providing the smallgroup interpersonal contact that best transmits enthusiasm. The students know the teacher is watching the conversations and is taking an active interest in the students' progress, sometimes by intervening and sometimes by providing answers and hints.
The implication for COMPS technology is that monitoring the health of student conversations could be informed by expressions of student affect. Affect, the observable manifestation of emotion, mediates social interaction and is related to student interest.
Self-efficacy, an individual’s belief to be capable of
performing a particular task (Bandura, 1977), has been
widely studied because of its relationship to performance
including academic achievement
We have collected data from one semester of the GEEN 165 class. There were 55 students at the start of the semes ter and 47 at the end. We administered COMPS exercises four times during the semester, with 53 group discussions in total. Most groups had 3 or 4 participants. The bulk of students were assigned to sessions quasi-randomly as students arrived in lab. Cliques of friends, who tended to arrive together, were split into different random groups. We deviated from this protocol by creating a few all female groups, for comparison with the all-male groups. Altogether there were about 8000 dialogue turns. Students were surveyed near the beginning and end of the semester regarding their enthusiasm for the class, their self-efficacy in programming, and their desire to continue. Every COMPS exercise was also accompanied by a survey of the student experiences.
The presence of discourse marker words, e.g. “now” or “therefore” near the beginning of a dialogue turn. These are linguistically associated with reasoning, and are therefore possibly indicative of productive discussion.
The presence of pronouns that include another participant in the dialogue: “you,” “we,” “us.” These are possibly indicative of transactive discussion.
The presence of question marks.
The presence of emoticons. It is possible that emoticons are associated with students attending to each others affect.
The length of a turn in words.
Whether typing this turn overlapped with other people typing.
Of particular interest are six affective states that we have chosen as initial targets. These are annotated by hand. They were chosen because they may be salient for monitoring both the learning aspects (whether the students are reasoning together) and the social health of the conversation. We show here some of the definitions that the coders have applied for consistency in recognizing and coding. • Excited. • Apologetic. Refers to a user expressing regret for previous action. This type of message is usually aimed towards another user or towards the group as a whole. • Humor. • Frustrated. • Confused. User explicitly expressing confusion, or exhibiting confusion e.g. through questions. • Sad. A negative emotion determined by keywords and sad emoticons that are usually directed at self.
Some of these affective states have been tagged and illustrated in the Table 1 dialogue. Table 2 indicates some of the textual indications for the various states. These are being used by the coders at present, but will become machine-derived features for the purpose of machineannotating the affective states.
The survey administered to all students at the beginning
and end of the semester has an interest part and a
self-efficacy part. The end-of-semester survey also inquires about
student plans for continuing in the major and registering
for the next programming class. All items use a 6-point
scale. The interest survey items are derived from a survey
from
The after-COMPS-lab survey had items covering student perceptions in three areas: student interest, whether the student learned from the lab, and how well the group exercise functioned. An example item is “I contributed to the understanding of other students in my group.”
dents’ interest toward the course in the beginning of the semester and end of the semester were 4.33 and 4.32 respectively. • Self-efficacy with respect to the course content indicated significant improvement between the beginning and the end of the semester, rising from 2.83 to 3.81.
The increase in self efficacy was significant, p < 0.01.
Table 4 shows students' perception of the COMPS labs, surveyed immediately after each lab. There seemed to be a clear improvement between the first part of the semester (Labs 1 and 2) and the later part (Labs 3 and 4). Students perceived: • more effective group work in the second part (means rose from about 3.1 to about 3.4) • better understanding of concepts in the second half (means rose from about 3.4 to about 3.9).
Multiple one-way ANOVA supports the hypothesis that mean scores are indeed different, p = 0.03 for both effectiveness and understanding. Post hoc analyses using the Tukey test for significance indicated that the mean scores of Lab 3 were significantly higher than Lab 2 for both effectiveness and understanding.
However, students’ interest in each exercise in the lab sessions seemed to fluctuate throughout the semester. Lab 3 had the highest interest, which corresponded with the highest effectiveness and understanding. But interest in Lab 4 was the approximately the same as Labs 1 and 2.
We annotated the 14 group discussions of one COMPS exercise, comprising 2147 dialogue turns, for the six affective features. In total 199 turns showed evidence of one or more feature, or 9.3%.
As a first test of the utility of these features along with the machine-generated ones, we tried to use them to predict the gender of the participant. Among 49 students we had 16 women and 33 men. First we aggregated all the turns from each student, and looked at statistical differences between the two populations. Two-tailed t-tests revealed that none of the features were significantly different between the genders at the p < 0.05 level. However expressions of apology were different at the p = 0.06 level. The most common affective feature was confusion, with 62 instances of utterances expressing confusion. Women expressed confusion in 4.6% of turns, and men in 2.2%. It suggests the two genders behave differently, but the p < 0.22 level does not show significance. The two genders also showed differences in the amount of participation. Men each uttered an average of 46 turns per dialogue and women 36 turns.
We then trained a J48 decision tree classifier and a
multiple-regression linear classifier using the Weka data
mining tool
The students experienced improvement in their experience of the COMPS exercises during the semester. They reported that the groups worked better in the last two exercises and that they learned more. It is not clear why student interest was lower in the last lab. Anecdotally there are two reasons that have been suggested by the instructor and lab TAs who supervised this session. One is that the last lab was optional, presented during Thanksgiving week. That fewer students attended could indicate that the general level of engagement was lower than usual. The other is that perhaps the novelty was wearing off. Some students expressed as much during the session. We will need to find some way to survey the reasons for student interest.
The pre- and post-semester survey is hard to interpret because of low participation rate and dropouts. In the next semester we are enforcing better participation. The increase in self-efficacy was striking, but we do not have yet any comparison with other classes. Future work includes comparing interest and self-efficacy with learning gains on the pre- and post-tests. Future work also includes comparing pre- and post-semester survey results with individual lab surveys, to see whether there are correlations between overall student interest and the situational interest in individual COMPS exercises.
Another analysis in the future will be between the participants of the same group: do they agree about learning and group functioning, do they have similar learning gains.
Hand-annotating the remaining 6000 turns of dialogue may result in more reliable statistical correlations. We are also at work toward machine-annotation of these features.
Annotation of the affective states so far has relied
entirely on the text of the dialogues. Future work will
include extra-linguistic features. In COMPS group
exercises in other classes evidence of student engagement
sometimes presents through Comic Sans typeface, big or
bold fonts, and wild colors. We are also exploring using
timing features from the overlapped typing. Students can
all type simultaneously while seeing each other's
developing chat text
Partial support for this work was provided by the National Science Foundation's Improving Undergraduate STEM Education (IUSE) program under Award No. 1504917. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
4.32 / 5 2.83 / 5 3.81 / 5 Effectiveness of group work Understanding of concept Interest in lab Mean / SD Mean / SD Mean /SD