Effect of Large Language Model Use on Programming Project Groups Laura Graf Technische Universität München, Arcisstraße 21, 80333 Munich, Germany Abstract The adoption of Large Language Models (LLMs) in education has prompted questions about their impact on programming projects. This research will explore how the use of LLMs affects learning and socio-affective outcomes on individual and group level in first software engineering projects. Existing literature explores both potential benefits and pitfalls of LLMs in educational contexts. LLMs enhancing readability, explaining others’ code and providing quick answers to less experienced students could improve group work. However, there are concerns such as students’ judgment of competency, effort and contributions created with LLM support affecting group collaboration dynamics. To address the gap in empirical research on LLMs' impact on perceptions of teammate competency, connectedness, self-efficacy, learning gain, and professional identification we will analyze not only self-reported measures but also work with process data from collaborative coding platforms to extract meaningful measures of collaboration behavior and issues in group code prominent when LLMs are being used. Keywords Large Language Models, Computer Science Education, Collaborative learning, Process Mining, Socio- affective measures 1 1. Introduction ensure that integration does not deprive students from developing much needed higher order thinking. After the vast adoption of Large Language Models Another reason why a focus on solely optimizing (LLMs) universities have raised questions about the student use of LLMs is insufficient in educational adequacy of curriculum and assessment in response to settings is that technologies continue to evolve. The student use of computer-generated output during their models change to support humans better, and this studies (Kasneci et al., 2023). These questions are process cannot be expected to stabilize on a certain particularly pertinent for future programming pattern (Joksimovic et al., 2023). professionals, since LLMs are effective in generating Students will need to continue working together, code chunks (Kazemitabaar et al., 2023). The use of LLM- solving problems and communicating effectively, generated code can not only speed up programming regardless of the specific cognitive tool they may use. tasks but substantially offload thinking processes to the Hence, it is also important to understand the machine. In light of potential automation of coding relationship between the use of cognitive tools and tasks, it has become unclear as to what skills should be educational outcomes that reach beyond domain taught to future programming professionals to enable knowledge. Frequent use of tools like LLMs may shape effective integration of LLMs into the human-led students in a way that affects them profoundly, so process. stakeholders need a clearer understanding of how The focus on optimization of productivity enabled broader educational outcomes are affected by those tools by machines has thus far been central to research on so that instructional practices can be adapted to preserve how to integrate LLMs into human cognitive practices the focus on developing skills essential for humans. (Wang et al., 2019). Yet such a focus is only partially To address this pressing need, my project will relevant in educational settings. Educational outcomes investigate how LLMs affect educational outcomes in a target students' cognitive development and higher-order collaborative setting where future programming thinking in relation to the domain they study. Tools professionals practice a broad set of skills. Collaborative supporting cognitive processes can benefit the learner in work is commonly part of software engineering projects offloading some parts of such a process and allowing the in computer science curricula. Collaborative work is an learner to focus on higher- order thinking (Salomon, essential part of professional software engineering and 2003). At the same time, improper use of the tool can interpersonal skills are among the most significant for lead to a reduced, shallow understanding (ibid.). the effectiveness of software engineers (Boyatzis et al., Therefore, it is important to understand the relationship 2017). Groups have been shown to innovate faster, between student use of cognitive tools such as LLMs and identify mistakes more quickly, and find better solutions learning outcomes related to their domain knowledge to Proceedings of the Doctoral Consortium of the 19th European Conference l.graf@tum.de (L. Graf) on Technology Enhanced Learning, September 16-20, 2024, Krems, Austria © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings to problems; all while reporting a higher job satisfaction tasks may affect educational outcomes. This includes (Duhigg, 2019). individual outcomes, such as learning, self- efficacy, and Moreover, project-based assignments where professional identity, as well as group-related socio- students practice collaborative work are a catalyst for affective outcomes, such as trust in teammates bonding and social learning, facilitating social capital competency and connectedness. I also explain why the among future professionals and affecting professional process of how learners collaborate when they identity. Collaboration experiences can also create individually use LLMs must be considered. precedents for exclusion and negatively affect belonging and diversity in STEM (Miller-Young et al., 2023). This 2.1. Effect of LLMs on Learning richness of educational outcomes makes collaborative Individuals programming assignments a suitable context to examine LLMs offer a diverse range of applications for the effect of cognitive tools, such as LLMs. enhancement of learning experiences, personalized to Current literature is limited in explaining the effects the student (Kasneci et al., 2023). In introductory that LLMs can have on broader educational outcomes in programming assignments, LLM-based coding tools programming group work. Existing research suggests currently already perform at the level that outscores the that such effects could be both positive and negative. average student (Finnie- Ansley et al., 2022). Most LLMs have the potential to support participation of students prefer using a LLM, especially to get a starting students with less programming skills in code point, even when they often face difficulties in production and help to understand others’ code, understanding, editing, and debugging generated code important for positive collaborative work. However, it (Vaithilingam et al., 2022). Studies have also shown that could also amplify issues of unequal effort distribution programmers tend to defer tasks related to through the option to auto-generate code, which is comprehension to the LLM, even though this can steer known to create negative experiences (Nguyen et al., them in the wrong direction (Nam et al., 2024). Some 2023). LLMs can, for instance, facilitate shared scholars also suggests that students use LLMs for understanding by improving code readability and requesting explanations of code and general questions documentation as well as reduce the need for any group more often than for code generation (Kazemitabaar et members to spend large parts of the time on lower-level al., 2024). tasks such as generating test cases, which might change To circumvent challenges associated with LLM previously common role distributions. With LLMs use, chatbots have been developed to offer hints to affecting most parts of the programming projects, an mimic human tutoring, instead of giving students full influence on the social aspects of group work is likely solutions (Bassner et al., 2024). and deserves attention. Literature so far has shown that integrating LLMs This research gap calls for empirical examination of into practices around learning and studying can affect the effects of LLMs in programming assignments on individual learning gains. A major concern here is that domain- specific knowledge and the effect on group when students regularly offload to technology, they may processes, as well as longer-term imprint on students, not actually learn how to perform the task on their own. such as the formation of professional identity. My thesis (Darvishi et al. 2024) found that when using LLM, will focus on addressing this gap. I will employ mixed students seem to be finishing tasks well, but once the methods research design. First, I will analyze the effect LLM was removed, they did not replicate the new of LLM integration into programing group projects, in strategies used by the LLM that were helpful with the relation to student perceptions of learning, their socio- tasks. Another study showed that learning gains from affective attitudes towards teammates and their using an LLM in learning programming languages vary evolving identification with the domain. Second, I will with context and task complexity (Aviv et al., 2024). investigate the relationship between these perceptions Researchers observed that LLMs did not reduce and process data from code progression, as student metacognitive difficulties for students with limited perceptions are largely mediated by the code-based programming abilities and even introduced new ones communication on GitHub. The thesis is in its planning (Prather et al., 2024). stage. In addition to learning gains, integrating LLMs into learning practices can impact self-perceptions, such 2. Related Work as self- efficacy and professional identity. Studies on When students use LLM-based tools for their tasks LLMs’ effects on students’ self-efficacy found that LLM- in a course, this affects how and what they learn. To supported review of course topics improved students’ advance the goal of understanding the effect of LLMs in self- efficacy and motivation (Lee et al., 2022). This effect collaborative programming assignments, this section on self-efficacy appeared because LLM helped students explains how LLM use in collaborative programming become active during learning, as it provided a safe way to explore questions (Y.-F. Lee et al., 2022). A study processes. The effects of such individual use within a where interactions with an LLM supported student collaborative task have not yet been explored. thinking deeply about a topic showed improved self- Previous research suggests that AI can affect efficacy and learning achievements (Chang et al., 2022). collaboration in unintended ways (Wang et al., 2022), (Wang et al., 2023) found that AI based on good and this can also be expected when students integrate technology combined with technological skills in a LLMs to support individual programming needs within higher education program improve students’ self- a collaborative task. For instance, group-related socio- efficacy, mediating performance. Perception of self- affective outcomes, such as trust in teammates’ efficacy can also benefit from having a starting point in competency and feeling of connectedness with the coding (Vaithilingam et al., 2022). group, may be affected. Students’ perceptions of Self-efficacy further plays an important role in teammates’ contributions may change when individuals securing diversity, equity, and inclusion in STEM. submit auto-generated code without transparency of Minorities and women feel less included in the how it was created. Engagement in collaborative work is engineering groups in general, but female students’ who strongly connected to trust in team members, and plan to persist in this male-dominated domain also show motivation to perform collaborative tasks may diminish high self-efficacy (Marra et al., 2009). It therefore also when this is compromised (Dirks, 1999). Studies in may be important to ensure that integrating LLMs into software engineering emphasize the role of perceived collaborative work, where many of the exclusionary transparency for trust (P. T. Y. Lee et al., 2024) as well as practices occur (e.g. William M. Hall, Toni Schmader, the role of perceived task-related competency of another Elizabeth Croft, 2015), maintains positive impact on team member (Mayer et al., 1995). Presumably, when long- term professional orientation, mediated by group team members use LLMs to generate code that in its experiences. form resembles more advanced programmers’ code, their competency is much harder to judge, especially in 2.2. Effect of LLMs on Learning Groups the earlier stages of a project and by novices. As team members progress in collaborative tasks, building on When it comes to collaborative learning settings others’ code is necessary and requires judgment of the as in group programming assignments, LLM use reaches quality of that code. Studies about the relationship of beyond the effects on the individual, such as learning, LLM to the judgment of competency show that LLM use self-perceptions, and future identification. Both socio- can lead novice programmers to misaligned confidence cultural (Vygotsky, 1978) and socio-cognitive theories of regarding their skills and understanding (Prather et al., learning (Dillenbourg, 1990) highlight the influence of 2024). Research has not yet addressed if the difficulty the environment on learning, often enacted through associated with the judgement of competency also peer interactions. LLM use by individual learners can applies to group-related judgement. potentially influence peer interactions, mediated by The difficulty in judging contributions might also technology, and further impact group- related socio- affect the connectedness of the group. Previously, social affective outcomes, such as trust in group members and connectedness, defined via measurements of frequency connectedness. of social contact, task assistance and compassion, as well Research on the use of LLMs in collaborative as sense of belonging, has been shown to be related to learning has been limited to the development of tools well-being (Frieling, M., Peach, E. K., & Cording, J., that target collaborative processes at the group-level. 2018). Connectedness can be defined as an affective For example, Kasneci et al. (2023) speculate that these outcome of group processes developed directly from tools can facilitate group discussions by providing interactions, such as mutual support, but also feedback and personalized guidance to students to impressions of others, from their contributions against improve group participation or give editing the context of own work on the common project. recommendations to support collaborative writing. LLMs could help avoid common faults in the group processes by integrating information or promoting 2.3. The Role of Process in Collaborative knowledge convergence and decision-making – all Programming group-level processes essential for effective I have argued that LLMs used by individuals in collaboration (Westby & Riedl, 2023; Järvelä & Hadwin, collaborative programming may affect learning and 2013; Khakurel & Blomqvist, 2022). It is noteworthy that socio-affective outcomes. A sole focus on outcomes in a many of these existing propositions are limited to the collaborative learning scenario is insufficient. LLM-based tools specifically designed to support group Dillenbourg et al. (1996) argued that process variables work. However, group members can also choose to use must also be considered when studying collaboration. LLMs for individual needs, rather than to support group This is because interaction effects between the many process-related mediators of collaboration outcomes would prevent reliable causal inference. Given the which contributions fix or keep problems in the code as dearth of research on process variables related to LLM- indicated by build logs (Chen et al., 2022). These process mediated contributions in a collaborative process, a measures have been analyzed in relation to student relationship between the indicators of the process data performance, but not in relation to student perceptions with learners’ perceptions of the members and the group of each other and the group, as well as with LLM- need to be established. ingestions within the contributions. For this, individual and group-level team code submissions need to be transformed to appropriate interaction process indicators. Log data from 3. Research Questions programming projects is different from conversation data often applied in collaboration research, though To investigate the effects of individual use of LLM-based conceptual similarities exist. For example, students who tools on group-based learning in programming projects work on programming projects regularly merge their in higher education, this project poses the following modified versions of the software into a common research questions: version. How the students amend the versions and who does this gives insight into success of previous 1. How does the use of LLM in collaborative coordination as well as (perceived) value of the programming affect individual outcomes, such as individual members’ contributions and who maintains learning gain, self-efficacy, and professional overview of the group’s code. In some groups, major identification, and group-related socio- affective conflicts result from not being able to amend different attitudes, such as perceptions of teammate versions to a working product (Tushev et al., 2018). In competency and connectedness? sum, group-level patterns of logs can make an impression on student perceptions of others and the 2. What is the relationship between process group itself. indicators describing individual and group- level Moreover, depending on the type of contributions, team code with the perceptions of competency individual roles in relation to the group may also be among team members and group-related socio- visible in GitHub traces. For example, previous work has affective attitudes? talked about the phenomenon of “cowboy programming”, where a group member took over the management of the relevant parts of the software 4. Research Design development without including others (Tushev et al., To address these research questions, a series of authentic 2018). Similarly, “free riders” and “social loafers” studies in a university scenario are planned. I plan to describe common patterns of individual roles students collect the data from group-based programming projects take on, bringing out negative group work dynamics from the courses targeting novice programmers. This (Nguyen et al., 2023). choice is due to the focus on the first experiences with Existing research on process indicators in programming group projects within higher education. programming creates a foundation for analyzing both The courses are expected to give us a participant number individual and group processes. According to code of between 600 and 1800 students in 120 to 450 groups. collaboration project research, best- performing teams We will collect data via interviews, self- reports show equal contributions, not necessarily the highest with established tools three times throughout the course total number of commits, but parallel main work times and log data of the evolving code and all GitHub and work on separate branches of the code (Tushev et activities. The interviews are employed to potentially al., 2018). Team roles can be visible as one person show causalities between our researched variables. contributing documentation while another contributes Including a self- and team assessment part in group the code (Tushev et al., 2018). (Gitinabard et al., 2020) projects is a common strategy for their grading, and we look at teamwork features on GitHub projects and will extend the usual questions with additional classify the student teams into three groups, questions. A final presentation of the artifacts is part of collaborative, cooperative, or solo-submit, the project’s examination and questions about the differentiating each contribution into types such as bug team’s code show each students’ understanding of the fix, documentation, test case, or implementation. They software. look at how many lines of code the members change, The first study will address the question of how across how many different files, how much they delete, the use of LLM in collaborative programming affects and informativeness of commit messages written by a individual outcomes, such as learning gain, self-efficacy, contributor. Another line of work goes deeper into and professional identification, and group-related socio- contribution quality from code analysis and considers affective attitudes, such as perceptions of teammate competency and connectedness, also considering Achievement. Educational Technology & Society, perceived equality of contributions. We will use self- 25(4), 245-256. reports for established measurement tools and [5] Chen, H.-M., Nguyen, B., & Dow, C.-R. (2022). interviews to capture these. Code-quality evaluation scheme for assessment of In the second study, we will analyze process data student contributions to programming projects. to evaluate relations to perceived effort distribution, Journal of Systems and Software, 188, 111273. accuracy of judgment in team members’ competency, https://doi.org/10.1016/j.jss.2022.111273 trust, differences in patterns in code contributions, [6] Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & mutual support, prevalent issues in engineering the Siemens, G. (2024). Impact of AI Assistance on software, perceived gain of skill, perceived appreciation Student Agency. Computers & Education, 210. from others and identification with the field of study and 104967. future professions. https://doi.org/10.1016/j.compedu.2023.104967 Since I am at an early stage of this project, I am not [7] Dillenbourg, P. (1990). What do you mean by set on the methods of analysis yet. We will tentatively “collaborative learning”? In Collaborative analyze the first data and develop process measures Learning: Cognitive and Computational appropriate to the theory of group dynamics in Approaches (pp. 1-19). Pergamon. programming projects and relate them to self-report [8] Dillenbourg, P., Baker, M., Blaye, A., & O'Malley, measures. The process measures will be sourced from C. (1996). The Evolution of Research on log data of AI interactions, submitted milestone Collaborative Learning. In Learning in Humans planning documents for coordination traces and features and Machine: Towards an Interdisciplinary of the frequently logged contributions (using analytics Learning Science (pp. 189– 211). Elsevier. of and building on previous code contribution [9] Dirks, K. T. (1999). The effects of interpersonal evaluation metrics). The intra-group interactions will trust on work group performance. Journal of include information of who built on top of or modified Applied Psychology, 84(3), 445–455. whose code, who fixed code cohesion or formatted the https://doi.org/10.1037/0021-9010.84.3.445 others’ code, who contributed comments or [10] Duhigg, C. (2019). What Google Learned From Its documentation, and where were errors introduced and Quest to Build the Perfect Team. The New York resolved. Times. [11] Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton- Acknowledgements Reilly, A., & Prather, J. (2022). The Robots Are Coming: Exploring the Implications of OpenAI I would like to thank my supervisor, Professor Sasha Codex on Introductory Programming. Poquet, for her extraordinary support in my research Australasian Computing Education Conference, work and learning, also in the advice given for this 10–19. https://doi.org/10.1145/3511861.3511863 project. [12] Frieling, M., Peach, E. K., & Cording, J. (2018). The Measurement of Social Connectedness and Its References Relationship to Wellbeing. Ministry of Social [1] Bassner, P., Frankford, E., & Krusche, S. (2024). Development New Zealand. Iris: An AI-Driven Virtual Tutor For Computer [13] Gitinabard, N., Estrada, T., & Bowers, A. (2020). Science Education. Teamwork Features and Their Impact on Student https://doi.org/10.1145/3649217.3653543 Collaboration in GitHub Projects. Journal of [2] Begel, A., & Nagappan, N. (2009). Coordination in Learning Analytics, 7(3), 25–45. Large-Scale Software Development: Helpful and https://doi.org/10.18608/jla.2020.73.3 Unhelpful Behaviors. [14] Hall, W. M., Schmader, T., & Croft, E. (2015). https://www.microsoft.com/en- Engineering Exchanges: Daily Social Identity us/research/publication/coordination-in-large- Threat Predicts Burnout Among Female scale-software-development-helpful-and- Engineers. unhelpful-behaviors/ https://journals.sagepub.com/doi/abs/10.1177/ [3] Boyatzis, R., Rochford, K., & Cavanagh, K. V. 1948550615572637 (2017). Emotional Intelligence Competencies in [15] Järvelä, S., & Hadwin, A. F. (2013). New Frontiers: Engineer’s Effectiveness and Engagement. Career Regulating Learning in CSCL. Educational Development International, 22(1), 70–86. Psychologist, 48(1), 25–39. https://doi.org/10.1108/CDI-08-2016-0136 https://doi.org/10.1080/00461520.2012.748006 [4] Chang, R., & Fang, Z. (2022). Effect of AI- Assisted [16] Kasneci, E., Seßler, K., Kuechemann, S., Bannert, Tutoring on Student Self-Efficacy and M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023). https://doi.org/10.1145/3597503.3639187 ChatGPT for Good? On Opportunities and [25] Prather, J., Reeves, B., Leinonen, J., Macneil, S., Challenges of Large Language Models for Randrianasolo, A., Becker, B., Kimmel, B., Wright, Education. Learning and Individual Differences, J., & Briggs, B. (2024). The Widening Gap: The 103, 102274. Benefits and Harms of Generative AI for Novice https://doi.org/10.1016/j.lindif.2023.102274 Programmers. [17] Kazemitabaar, M., Hou, X., Henley, A., Ericson, B. [26] Salomon, G. (2003). Distributed Cognitions: J., Weintrop, D., & Grossman, T. (2023). How Psychological and Educational Considerations. Novices Use LLM-Based Code Generators to Solve Cambridge University Press. CS1 Coding Tasks in a Self-Paced Learning [27] Tushev, A., Gross, S., & Schaarschmidt, M. (2018). Environment. Collaborative Programming Dynamics in Software arXiv. http://arxiv.org/abs/2309.14049 Engineering Education: Insights from Student [18] Kazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Project Logs. ACM Transactions on Computing Denny, P., Craig, M., & Grossman, T. (2024). Education, 18(3), Article 14. CodeAid: Evaluating a Classroom Deployment of https://doi.org/10.1145/3134822 an LLM-based Programming Assistant that [28] Vaithilingam, P., Zhang, T., & Glassman, E. L. Balances Student and Educator Needs. (2022). Expectation vs. Experience: Evaluating the Proceedings of the CHI Conference on Human Usability of Code Generation Tools Powered by Factors in Computing Systems, 1–20. Large Language Models. CHI Conference on https://doi.org/10.1145/3613904.3642773 Human Factors in Computing Systems Extended [19] Khakurel, J., & Blomqvist, K. (2022). Artificial Abstracts, 1–7. Intelligence Augmenting Human Teams. A https://doi.org/10.1145/3491101.3519665 Systematic Literature Review on the [29] Vygotsky, L. S. (1978). Mind in Society: Opportunities and Concerns. In H. Degen & S. Development of Higher Psychological Processes. Ntoa (Eds.), Artificial Intelligence in HCI (pp. 51– Harvard University Press. 68). Springer International Publishing. [30] Wang, L., Gao, R., Váncza, J., Krüger, J., Wang, X. https://doi.org/10.1007/978-3-031-05643-7_4 V., Makris, S., & Chryssolouris, G. (2019). [20] Lee, Y.-F., Hwang, G.-J., & Chen, P.-Y. (2022). Symbiotic human-robot collaborative assembly. Impacts of an AI-based chatbot on college CIRP Annals, 68(2), 701–726. students’ after-class review, academic https://doi.org/10.1016/j.cirp.2019.05.002 performance, self-efficacy, learning attitude, and [31] Wang, Q., Camacho, I., Jing, S., & Goel, A. K. motivation. Educational Technology Research and (2022). Understanding the Design Space of AI- Development, 70(5), 1843–1865. Mediated Social Interaction in Online Learning: https://doi.org/10.1007/s11423-022-10142-8 Challenges and Opportunities. Proceedings of the [21] Mayer, R. C., Davis, J. H., & Schoorman, F. D. ACM on Human-Computer Interaction, (1995). An Integrative Model of Organizational 6(CSCW1), 1–26. Trust. The Academy of Management Review, https://doi.org/10.1145/3512977 20(3), 709–734. [32] Wang, Shaofeng, Zhuo Sun, & Ying Chen. (2023). https://doi.org/10.2307/258792 Effects of Higher Education Institutes’ Artificial [22] Marra, R. M., Rodgers, K. A., Shen, D., & Bogue, B. Intelligence Capability on Students' Self-Efficacy, (2009). Women Engineering Students and Self- Creativity, and Learning Performance. Education Efficacy: A Multi-Year, Multi-Institution Study of and Information Technologies, 28(5), 4919– Women Engineering Student Self-Efficacy. Journal 4939. https://doi.org/10.1007/s10639-022-11246-8 of Engineering Education, 98(1), 27–38. https://doi.org/10.1002/j.2168- 9830.2009.tb01003.x [23] Miller-Young, J., Jamieson, M., & Beck, S. (2023). Diverse Experiences and Belonging in an Online, First-Year, Team-Based Engineering Design Course. Teaching in Higher Education, 1–17. https://doi.org/10.1080/13562517.2022.2162816 [24] Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024). Using an LLM to Help With Code Understanding. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 1–13.