This workshop paper critically examines the implications of generative artificial intelligence (GEN-AI) for collaborative learning by exploring the intersection of two essential 21st-century skills: collaborative problem solving (CPS) and artificial intelligence (AI) literacy. GEN-AI, such as advanced language models, has revolutionized human-computer interaction by simulating naturalistic dialogues, leading to its integration into educational settings as a tool for collaboration and learning. However, CPS encompasses social and emotional dimensions-such as empathy, shared understanding, and intentionality-that GENAI cannot replicate. While GEN-AI can simulate certain aspects of CPS and offer scalable opportunities for skill development, it may lack the ability to foster genuine collaboration. This paper argues that interacting with GEN-AI requires a distinct skill set, including prompt engineering, critical evaluation of AI output, and ethical awareness, which collectively define AI literacy. We raise critical questions about the overlap and divergence between CPS and AI interaction considering cultural and task-specific differences, discuss the potential of GEN-AI to approximate CPS processes and explore the implications for designing learning environments. This paper aims to provoke a discussion on how to balance the strengths of AI with the irreplaceable human dimensions of collaboration in education and beyond.
[
This collaboration can be realized in a human-to-human(H-H) setting or in a human-to-agent
(HA) setting like in the PISA2015 assessment of collaborative problem solving. While H-H collaboration
tasks entail a more authentic representations of natural collaboration they lack controllability [
As social and economic challenges become more complex, the ability to work effectively in diverse
groups to solve problems has become a critical ability. Moreover, CPS is increasingly important in a
world that demands innovative solutions to constantly evolving problems. Across different
professional sectors, employees are expected to work with diverse teams, often in a digital
environment, highlighting the importance of CPS in today's workforce [
The OECD framework for CPS integrates individual problem-solving and social collaboration [
The PISA study of 2015 used a human-agent approach to measure students' CPS skills [
The introduction of GEN-AI tools like advanced language models seems to address these limitations by creating more naturalistic interactions. Unlike the rigid, pre-programmed behaviors of traditional human-agent approaches, GEN-AI can simulate human-like dialogue, adapt to a wide range of prompts, and produce contextually relevant responses. This has led to a growing perception that interacting with GEN-AI approximates the experience of working with a human collaborator. However, this raises important questions about the nature of these interactions: Are they truly collaborative? How do they compare to the social and emotional dimensions of human-human collaboration? To explore this, it is essential to examine the skills required to interact effectively with GEN-AI —commonly referred to as AI literacy—and how these skills relate to CPS.
AI literacy requires a fundamental understanding of AI principles, data interpretation skills, and an awareness of the ethical implications of AI use [12]. Furthermore, it emphasizes the need for effective communication with AI, which differs from human-to-human communication paradigms due to the lack of shared understanding or emotional engagement [13]. Long and Magerko define AI literacy as “a set of competencies that enable individuals to critically evaluate AI technologies, communicate and collaborate effectively with AI, and use AI as a tool online, at home, and in the workplace” [14]. This definition encapsulates the need for individuals to understand the underlying principles of AI, evaluate its implications, and use it effectively.
In interacting with GEN-AI models, a major competency lies in writing effective prompts [15]. Prompting refers to the technique of phrasing user input to elicit the desired output from the AI. Understanding how different prompts can lead to different responses is a key skill for effective AI communication. Such a task presents its own set of challenges because, unlike humans, AI lacks the ability to understand the context of a problem beyond the given prompt. The response of an AI model can vary dramatically depending on how a question or statement is framed. A study by ZamfirescuPereira and colleagues [16] examined the ability of non-AI experts to maneuver 'end-user prompt engineering' in a language learning model (LLM)-based Chabot design. The study revealed that participants interacted with the tool in an opportunistic rather than systematic manner, encountering hurdles similar to those found in end-user programming systems and interactive machine learning systems. Furthermore, the study found that participants' inherent expectations, derived from human interactions, and a tendency to overgeneralize were significant barriers to effective prompt design.
In addition, an integral part of AI literacy is understanding the limitations of AI in communication. Despite its ability to generate human-like text, AI lacks the ability to comprehend meaning or express genuine emotion. It merely simulates these facets based on its training data, which can sometimes lead to contextually inappropriate or emotionally tone-deaf output. This is particularly evident in the generation of factually incorrect responses, which LLMs produce with a high degree of confidence. According to Zheng and colleagues [17], these missteps are a result of the LLM's inability to evaluate the veracity of the information it generates. It merely mimics the style and format of accurate responses based on the data on which it has been trained.
This lack of factual validity checking is a significant challenge in AI communication and can
potentially lead to misinformation if the generated content is taken at face value. Therefore, the
ability to critically evaluate AI-generated content becomes another essential skill for users. This
involves checking not only the emotional and contextual appropriateness of AI responses, but also
their factual accuracy. As AI technologies continue to evolve and permeate various aspects of daily
life, the development of this facet of AI literacy will become increasingly important [18].
2. Is interacting with gen-ai collaborative problem solving?
However, it is important to clarify that while GEN-AI systems are becoming increasingly
sophisticated, they do not possess understanding, consciousness or agency [19]. They generate
outputs based on patterns extracted from vast amounts of data, but they do not understand or
experience the content they produce. This distinction is critical when considering the nature of
interaction with GEN-AI in the context of CPS [20]. CPS is characterized by the aim to achieve a
common goal, especially in situations where the solution is not immediately obvious [
A key limitation in human-AI interactions is the absence of shared goals and intentionality. AI models do not possess their own objectives; they simply produce outputs in response to human instructions. This one-sided dynamic prevents genuine CPS and means cognition is not truly distributed between agents [22]. Moreover, most current AI models rely primarily on text inputs and ignore all other social cues such as tone, gestures, or mimics, which are all essential in human collaboration [23]. For effective use, humans must recognize this asymmetrical relationship and understand how the AI generates its outputs based on training data and algorithms.
We therefore assume that while CPS skills and the skills required to interact with GEN-AI do overlap to a certain extent, they are not inherently similar. Following this assumption, we assume that GEN-AI has the potential to simulate certain aspects of CPS skills, while it is currently not capable to account for all of them at the same time. In addition, we further assume that there are aspects of CPS skills that are easier to simulate with GEN-AI while others can currently not adequately be represented through using GEN-AI. As a basis for discussion, we suggest a rating for different subskills of CPS how complex it is to simulate them using current GEN-AI (see Table 1). Thus, GEN-AI can be seen as a form of approximation of practice [24] which is not able to copy the real complexity of CPS processes but simulate specific aspects of CPS processes by reducing complexity. They allow to provide “opportunities to engage in practices that are more or less proximal to the practices of a profession” [24]. For example, GEN-AI tools can be seen as an approximation of CPS focusing on certain, but not all, aspects of CPS skills.
Simulation Complexity with GEN-AI CPS skills from the PISA Framework that are • simulated by GEN-AI with low complexity • CPS skills from the PISA Framework that are • simulated by GEN-AI with high complexity • • • • • • •
Identifying and describing tasks to be completed Describing roles and team organization (communication protocol/rules of engagement) Communicating with team members about the actions to be/being performed Enacting plans Following rules of engagement, (e.g. prompting other team members to perform their tasks) Monitoring results of actions and evaluating success in solving the problem Monitoring, providing feedback and adapting the team organization and roles
team members Building a shared representation and negotiating the meaning of the problem (common ground) CPS skills from the PISA Framework that can • currently not be simulated by GEN AI •
interaction to solve the problem, along with goals Monitoring and repairing the shared understanding
Although AI language models can simulate human-like communication, they lack consciousness
and the ability to understand or derive meaning from interactions [22]. This means they cannot truly
collaborate or share understanding with humans. Instead, they function as advanced tools or
resources that generate responses solely based on user prompts, without any conceptual grasp of the
content. This does not exclude them from being used as tools in training aspects of CPS in human
learners, though.
3. Can interacting with gen-ai be transferred to human collaboration
settings?
There is evidence that collaborating with a computerized agent did not significantly diverge from
human-human interaction [
While GEN-AI can simulate certain aspects of human interaction, true CPS involves a broader range of social and emotional skills that AI fundamentally lacks. GEN-AI offers opportunities to approximate CPS processes by providing structured interactions and feedback, but these interactions lack mutual understanding, shared intentionality, and the emotional dynamics that are critical and influential in human-human collaboration. This distinction underscores the need to carefully delineate AI literacy from those needed for genuine CPS. Developing AI literacy entails mastering the principles of AI, understanding its limitations, and critically evaluating its outcomes, including the ethical implications of its use. As GEN-AI becomes increasingly embedded in education and the workplace, recognizing its potential and limitations will be crucial for leveraging its capabilities without conflating them with human collaborative skills. While GEN-AI can serve as a tool to train certain CPS-related skills, transferring skills learned in human-agent interactions to human-human settings requires additional emphasis on social cognition, empathy, and shared understanding.
Looking forward, integrating GEN-AI into educational and professional settings will require balancing its strengths as a simulation tool with the need to foster the uniquely human dimensions of collaboration. As we strive to equip individuals with 21st-century skills, cultivating both AI literacy and genuine CPS abilities will be critical to preparing learners for a future where human-AI collaboration complements, rather than replaces, human-human teamwork [25, 26]. 4. What are the implications for collaborative learning? While both CPS and interacting with GEN-AI involve elements of problem solving, the competencies required for each differ fundamentally. Human-human collaboration relies on social cognition, shared understanding, and emotional engagement—dimensions that are currently beyond the scope of AI capabilities. In contrast, effective interaction with GEN-AI as part of AI literacy demands a different skill set, including an understanding of AI principles, ethical awareness, critical evaluation of AI outputs, and familiarity with AI's strengths and limitations.
These distinctions are critical to consider when designing learning environments that aim to foster both collaboration and interaction with AI. It is essential to intentionally integrate these considerations into the design of tasks and activities to ensure learners develop the appropriate skills for interacting with GEN-AI and for engaging in meaningful collaborative learning. By addressing these differences, educators can better equip learners to leverage GEN-AI effectively while also cultivating human collaboration skills. Moreover, the design of CPS tasks should take into account factors such as cultural differences among users of GEN-AI (i.e., users who more or less approximate their communication with AI to communication with a human) and task-specific characteristics (i.e., collaborative tasks that differ in the degree to which they require emotional engagement or social cognition) as factors that may affect the transferability of skills acquired in human-AI collaboration to human-to-human collaboration. Recent studies (e.g. [27]) highlight that cultural backgrounds can influence how learners perceive and use GEN-AI as a collaboration partner, as well as how they communicate with it. Understanding and accommodating these variations can enhance the inclusivity and effectiveness of AI-driven collaborative learning environments. Finally, since not all aspects of human collaboration can be accurately simulated using GEN-AI, careful task design is essential to balance AI's strengths as a tool with the unique, irreplaceable aspects of human-tohuman collaboration.
Declaration on Generative AI During the preparation of this work, the authors used GPT-4o for grammar and spelling checks. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [10] P. Griffin, E. Care, B. McGaw, The changing role of education and schools, in: P. Griffin, B.
McGaw, E. Care (Eds.), Assessment and Teaching of 21st Century Skills, Springer, Dordrecht, 2012, pp. 1–15. doi:10.1007/978-94-007-2324-5_1. [11] E. Care, C. Scoular, P. Griffin, Assessment of collaborative problem solving in education environments, Appl. Meas. Educ. 29 (2016) 250–264. doi:10.1080/08957347.2016.1209204. [12] D. T. K. Ng, J. K. L. Leung, S. K. W. Chu, M. S. Qiao, Conceptualizing AI literacy: An exploratory review, Comput. Educ. Artif. Intell. 2 (2021) 100041. doi:10.1016/j.caeai.2021.100041. [13] L. Pessoa, A network model of the emotional brain, Trends Cogn. Sci. 21 (2017) 357–371.
doi:10.1016/j.tics.2017.03.002. [14] D. Long, B. Magerko, What is AI literacy? Competencies and design considerations, in: Proc. 2020 CHI Conf. Hum. Factors Comput. Syst., ACM, New York, NY, USA, 2020, pp. 1–16. doi:10.1145/3313831.3376727. [15] N. Knoth, A. Tolzin, A. Janson, J. M. Leimeister, AI literacy and its implications for prompt engineering strategies, Comput. Educ. Artif. Intell. 6 (2024) 100225. doi:10.1016/j.caeai.2024.100225. [16] J. D. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, Q. Yang, Why Johnny can’t prompt: How non-AI experts try (and fail) to design LLM prompts, in: Proc. 2023 CHI Conf. Hum. Factors Comput. Syst., ACM, New York, NY, USA, 2023, pp. 1–21. doi:10.1145/3544548.3581388. [17] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, I. Stoica, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. doi:10.48550/arXiv.2306.05685. [18] E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, G. Kasneci, ChatGPT for good? On opportunities and challenges of large language models for education, Learn. Individ.
Differ. 103 (2023) 102274. doi:10.1016/j.lindif.2023.102274. [19] G. Marcus, E. Davis, Rebooting AI: Building artificial intelligence we can trust, Vintage Books,
New York, NY, 2019. [20] J. Bareis, C. Katzenbach, Talking AI into being: The narratives and imaginaries of national AI strategies and their performative politics, Sci. Technol. Hum. Values 47 (2022) 855–881. doi:10.1177/01622439211030007. [21] J. Funke, A. Fischer, D. V. Holt, Competencies for complexity: Problem solving in the twentyfirst century, in: E. Care, P. Griffin, M. Wilson (Eds.), Assessment and Teaching of 21st Century Skills, Springer, Cham, 2018, pp. 41–53. doi:10.1007/978-3-319-65368-6_3. [22] U. Cress, J. Kimmerle, Co-constructing knowledge with generative AI tools: Reflections from a
CSCL perspective, Comput. Support. Learn. 18 (2023) 607–614. doi:10.1007/s11412-023-09409-w. [23] M. Li, H. Liu, M. Cai, J. Yuan, Estimation of individuals’ collaborative problem solving ability in computer-based assessment, Educ. Inf. Technol. 29 (2024) 483–515. doi:10.1007/s10639-02312271-w. [24] P. Grossman, C. Compton, D. Igra, M. Ronfeldt, E. Shahan, P. W. Williamson, Teaching practice: A cross-professional perspective, Teach. Coll. Rec. 111 (2009) 2055–2100. doi:10.1177/016146810911100905. [25] D. T. K. Ng, J. K. L. Leung, M. J. Su, I. H. Y. Yim, M. S. Qiao, S. K. W. Chu, AI literacy for all, in: D. T. K. Ng, J. K. L. Leung, M. J. Su, I. H. Y. Yim, M. S. Qiao, S. K. W. Chu (Eds.), AI Literacy in K-16 Classrooms, Springer, Cham, 2022, pp. 21–29. doi:10.1007/978-3-031-18880-0_3. [26] D. Ferrucci, A. Levas, S. Bagchi, D. Gondek, E. T. Mueller, Watson: Beyond jeopardy!, Artif.
Intell. 199-200 (2013) 93–105. doi:10.1016/j.artint.2012.06.009. [27] J. Kharchenko, T. Roosta, A. Chadha, C. Shah, How well do LLMs represent values across cultures? Empirical analysis of LLM responses based on Hofstede cultural dimensions, 2024. doi:10.48550/arXiv.2406.14805.