The adoption of Large Language Models (LLMs) in education has prompted questions about their impact on programming projects. This research will explore how the use of LLMs affects learning and socio-affective outcomes on individual and group level in first software engineering projects. Existing literature explores both potential benefits and pitfalls of LLMs in educational contexts. LLMs enhancing readability, explaining others' code and providing quick answers to less experienced students could improve group work. However, there are concerns such as students' judgment of competency, effort and contributions created with LLM support affecting group collaboration dynamics. To address the gap in empirical research on LLMs' impact on perceptions of teammate competency, connectedness, self-efficacy, learning gain, and professional identification we will analyze not only self-reported measures but also work with process data from collaborative coding platforms to extract meaningful measures of collaboration behavior and issues in group code prominent when LLMs are being used.
After the vast adoption of Large Language Models
(LLMs) universities have raised questions about the
adequacy of curriculum and assessment in response to
student use of computer-generated output during their
studies
The focus on optimization of productivity enabled
by machines has thus far been central to research on
how to integrate LLMs into human cognitive practices
Another reason why a focus on solely optimizing student use of LLMs is insufficient in educational settings is that technologies continue to evolve. The models change to support humans better, and this process cannot be expected to stabilize on a certain pattern (Joksimovic et al., 2023).
Students will need to continue working together, solving problems and communicating effectively, regardless of the specific cognitive tool they may use. Hence, it is also important to understand the relationship between the use of cognitive tools and educational outcomes that reach beyond domain knowledge. Frequent use of tools like LLMs may shape students in a way that affects them profoundly, so stakeholders need a clearer understanding of how broader educational outcomes are affected by those tools so that instructional practices can be adapted to preserve the focus on developing skills essential for humans.
To address this pressing need, my project will
investigate how LLMs affect educational outcomes in a
collaborative setting where future programming
professionals practice a broad set of skills. Collaborative
work is commonly part of software engineering projects
in computer science curricula. Collaborative work is an
essential part of professional software engineering and
interpersonal skills are among the most significant for
the effectiveness of software engineers (Boyatzis et al.,
2017). Groups have been shown to innovate faster,
identify mistakes more quickly, and find better solutions
to problems; all while reporting a higher job satisfaction
Moreover, project-based assignments where
students practice collaborative work are a catalyst for
bonding and social learning, facilitating social capital
among future professionals and affecting professional
identity. Collaboration experiences can also create
precedents for exclusion and negatively affect belonging
and diversity in STEM
Current literature is limited in explaining the effects that LLMs can have on broader educational outcomes in programming group work. Existing research suggests that such effects could be both positive and negative. LLMs have the potential to support participation of students with less programming skills in code production and help to understand others’ code, important for positive collaborative work. However, it could also amplify issues of unequal effort distribution through the option to auto-generate code, which is known to create negative experiences (Nguyen et al., 2023). LLMs can, for instance, facilitate shared understanding by improving code readability and documentation as well as reduce the need for any group members to spend large parts of the time on lower-level tasks such as generating test cases, which might change previously common role distributions. With LLMs affecting most parts of the programming projects, an influence on the social aspects of group work is likely and deserves attention.
This research gap calls for empirical examination of the effects of LLMs in programming assignments on domain- specific knowledge and the effect on group processes, as well as longer-term imprint on students, such as the formation of professional identity. My thesis will focus on addressing this gap. I will employ mixed methods research design. First, I will analyze the effect of LLM integration into programing group projects, in relation to student perceptions of learning, their socioaffective attitudes towards teammates and their evolving identification with the domain. Second, I will investigate the relationship between these perceptions and process data from code progression, as student perceptions are largely mediated by the code-based communication on GitHub. The thesis is in its planning stage.
When students use LLM-based tools for their tasks in a course, this affects how and what they learn. To advance the goal of understanding the effect of LLMs in collaborative programming assignments, this section explains how LLM use in collaborative programming tasks may affect educational outcomes. This includes individual outcomes, such as learning, self- efficacy, and professional identity, as well as group-related socioaffective outcomes, such as trust in teammates competency and connectedness. I also explain why the process of how learners collaborate when they individually use LLMs must be considered.
LLMs offer a diverse range of applications for
enhancement of learning experiences, personalized to
the student
To circumvent challenges associated with LLM
use, chatbots have been developed to offer hints to
mimic human tutoring, instead of giving students full
solutions
Literature so far has shown that integrating LLMs
into practices around learning and studying can affect
individual learning gains. A major concern here is that
when students regularly offload to technology, they may
not actually learn how to perform the task on their own.
In addition to learning gains, integrating LLMs
into learning practices can impact self-perceptions, such
as self- efficacy and professional identity. Studies on
LLMs’ effects on students’ self-efficacy found that
LLMsupported review of course topics improved students’
self- efficacy and motivation
Self-efficacy further plays an important role in
securing diversity, equity, and inclusion in STEM.
Minorities and women feel less included in the
engineering groups in general, but female students’ who
plan to persist in this male-dominated domain also show
high self-efficacy
When it comes to collaborative learning settings
as in group programming assignments, LLM use reaches
beyond the effects on the individual, such as learning,
self-perceptions, and future identification. Both
sociocultural
Research on the use of LLMs in collaborative
learning has been limited to the development of tools
that target collaborative processes at the group-level.
For example, Kasneci et al. (2023) speculate that these
tools can facilitate group discussions by providing
feedback and personalized guidance to students to
improve group participation or give editing
recommendations to support collaborative writing.
LLMs could help avoid common faults in the group
processes by integrating information or promoting
knowledge convergence and decision-making – all
group-level processes essential for effective
collaboration
Previous research suggests that AI can affect
collaboration in unintended ways
The difficulty in judging contributions might also
affect the connectedness of the group. Previously, social
connectedness, defined via measurements of frequency
of social contact, task assistance and compassion, as well
as sense of belonging, has been shown to be related to
well-being
I have argued that LLMs used by individuals in collaborative programming may affect learning and socio-affective outcomes. A sole focus on outcomes in a collaborative learning scenario is insufficient. Dillenbourg et al. (1996) argued that process variables must also be considered when studying collaboration. This is because interaction effects between the many process-related mediators of collaboration outcomes would prevent reliable causal inference. Given the dearth of research on process variables related to LLMmediated contributions in a collaborative process, a relationship between the indicators of the process data with learners’ perceptions of the members and the group need to be established.
For this, individual and group-level team code submissions need to be transformed to appropriate interaction process indicators. Log data from programming projects is different from conversation data often applied in collaboration research, though conceptual similarities exist. For example, students who work on programming projects regularly merge their modified versions of the software into a common version. How the students amend the versions and who does this gives insight into success of previous coordination as well as (perceived) value of the individual members’ contributions and who maintains overview of the group’s code. In some groups, major conflicts result from not being able to amend different versions to a working product (Tushev et al., 201 8). In sum, group-level patterns of logs can make an impression on student perceptions of others and the group itself.
Moreover, depending on the type of contributions,
individual roles in relation to the group may also be
visible in GitHub traces. For example, previous work has
talked about the phenomenon of “cowboy
programming”, where a group member took over the
management of the relevant parts of the software
development without including others
Existing research on process indicators in
programming creates a foundation for analyzing both
individual and group processes. According to code
collaboration project research, best- performing teams
show equal contributions, not necessarily the highest
total number of commits, but parallel main work times
and work on separate branches of the code
To investigate the effects of individual use of LLM-based tools on group-based learning in programming projects in higher education, this project poses the following research questions: 1. How does the use of LLM in collaborative programming affect individual outcomes, such as learning gain, self-efficacy, and professional identification, and group-related socio- affective attitudes, such as perceptions of teammate competency and connectedness? 2. What is the relationship between process indicators describing individual and group- level team code with the perceptions of competency among team members and group -related socioaffective attitudes?
To address these research questions, a series of authentic studies in a university scenario are planned. I plan to collect the data from group-based programming projects from the courses targeting novice programmers. This choice is due to the focus on the first experiences with programming group projects within higher education. The courses are expected to give us a participant number of between 600 and 1800 students in 120 to 450 groups.
We will collect data via interviews, self- reports with established tools three times throughout the course and log data of the evolving code and all GitHub activities. The interviews are employed to potentially show causalities between our researched variables. Including a self- and team assessment part in group projects is a common strategy for their grading, and we will extend the usual questions with additional questions. A final presentation of the artifacts is part of the project’s examination and questions about the team’s code show each students’ understanding of the software.
The first study will address the question of how the use of LLM in collaborative programming affects individual outcomes, such as learning gain, self-efficacy, and professional identification, and group-related socioaffective attitudes, such as perceptions of teammate competency and connectedness, also considering perceived equality of contributions. We will use selfreports for established measurement tools and interviews to capture these.
In the second study, we will analyze process data to evaluate relations to perceived effort distribution, accuracy of judgment in team members’ competency, trust, differences in patterns in code contributions, mutual support, prevalent issues in engineering the software, perceived gain of skill, perceived appreciation from others and identification with the field of study and future professions.
Since I am at an early stage of this project, I am not set on the methods of analysis yet. We will tentatively analyze the first data and develop process measures appropriate to the theory of group dynamics in programming projects and relate them to self-report measures. The process measures will be sourced from log data of AI interactions, submitted milestone planning documents for coordination traces and features of the frequently logged contributions (using analytics of and building on previous code cont ribution evaluation metrics). The intra-group interactions will include information of who built on top of or modified whose code, who fixed code cohesion or formatted the others’ code, who contributed comments or documentation, and where were errors introduced and resolved.
I would like to thank my supervisor, Professor Sasha Poquet, for her extraordinary support in my research work and learning, also in the advice given for this project.