This is a work-in-progress paper, in which we outline the challenge of supporting context-specific AI literacy and present the Competence Imitation Game aimed at addressing this challenge. The Competence Imitation Game is a co-designed educational approach and practical tool inspired by the Turing Test. The way in which AI technology is adopted, and its impact, varies depending on the domain, meaning that AI literacy may comprise diferent competences depending on the context. Domain experts possess the tacit knowledge and insider perspectives required to recognize and evaluate these context-specific competences. The work addresses the question on how to harness the experience of practitioners to promote context-specific AI literacy by centering the learning process on collegial exploration of the boundary between human and machine competence. Through collective eforts to identify peers from Large Language Models imitating them, the Competence Imitation Game ofers a new model for promoting AI literacy rooted in situated judgment and domain-specific expertise.
The following excerpt is taken from an imitation game experiment between professional programmers and a large language model (LLM). The format is inspired by the Turing Test: a programmer (the Judge) attempts to distinguish between a human programmer (Respondent 2) and a LLM prompted to imitate one (Respondent 1) through typed questions and answers. to collegially reflect upon, and discover the knowledge, skill and experience that remain uniquely human in the age of AI.
We begin by discussing the concept of AI literacy, outline key challenges in its development, and show how the Competence Imitation Game can help address these challenges. We also present the co-design of an accessible browser-based application initiated to support this goal.
Artificial intelligence (AI) is one of the fastest growing sub-fields of computer science (CS) with direct
impact on society and our daily lives. AI in education is a nascent field, with no clear guidelines on
what to teach or how to pedagogically benefit from AI in classrooms [
An influential framework posited by Long and Magerko [
Promoting AI literacy as a citizen competence across society presents several challenges. These include: (i) the complexity and unfamiliarity of AI technologies for many people, (ii) the lack of hands-on activities beyond direct content teaching resources, and (iii) the variation in the type of AI literacy required across diferent domains of practice where AI systems are used.
The first challenge, the complexity and unfamiliarity of AI, echoes the dificulties faced when
programming was added to K-12 curricula. At that time, many teachers lacked programming experience
[
Third, the use and impact of AI technologies and, therefore, the type of AI literacy required varies
depending on the context. In sociological terms, modern society is divided into a multitude of
increasingly specialized social worlds, sometimes referred to as forms of life, cultures, speech communities,
communities of practice, and so on (see [
This project addresses these needs by developing the Competence Imitation Game (henceforth, COMIG), inspired by the now classical Turing Test [19]. In the COMIG, members of a social world explore the limits and capabilities of AI within their own unique domain of practice. The purpose of the COMIG is to establish a platform for peer learning, which harnesses the shared context-specific knowledge and experience of participants to promote exploration and reflection around the role and impact of AI technologies on their lives. Our goal is to develop the COMIG into an accessible educational resource that supports the exploration and advancement of AI literacy across K–12 education, higher education, and professional contexts. In what follows, we describe the COMIG in more detail, its underlying theoretical framework, and outline its potential to explore and enhance AI literacy. We then turn to the ongoing co-design process through which the game is being collaboratively developed into a publicly accessible web-based application.
The Competence Imitation Game is a method of researching and developing AI literacy, taking human
diversity and societal specialization as a starting point. In this activity, members of a chosen social
group are tasked with distinguishing and identifying a group member from a Large Language Model
(LLM), prompted to imitate a fellow member, through typed question-answer-assessment dialogues.
The game itself, combined with a facilitated review and discussion of the results, creates a space for
peer-to-peer exchange and context-specific reflection on the key themes raised by Long and Magerko
[
Originally, the Imitation Game was a Victorian parlor game in which a judge, through written dialogue, had to determine which of two players was genuine and which was the imitator; for example, which player was a woman and which was a man pretending to be a woman. The game became famous through Alan Turing [19], who used it as a thought experiment, later known as the Turing test, to examine the ability of machines to mimic human intelligence. Turing proposed that if a human interrogator, after a five-minute interview, could not distinguish a machine from a human, the machine has achieved a human level of intelligence.
During the past two decades, sociologists have adapted the Imitation Game as a method to investigate the distribution of knowledge in society and promote mutual understanding between diferent social groups [20, 21, 22, 23]. The core idea of the game is simple: instead of a human trying to identify a machine, a member of a social group is tasked with identifying a genuine group member from an imitating nonmember; for example, Scots identifying fellow Scots from English people pretending to be Scottish. Educational applications show that the method promotes insight into shared knowledge and experience, identity construction, and mutual understanding within a group by encouraging participants to express and evaluate competences that the other party does not have access to [24, 25]. A key strength of the method is that, by positioning participants as proxy researchers who conceive the questions and evaluate the answers from their perspective, it harnesses the experience of domain experts to define what constitutes relevant knowledge.
The COMIG combines the legacy of Alan Turing’s test on machines’ ability to mimic human behavior with the sociological imitation game paradigm, which views social groups as loci of knowledge and expertise. It is essentially a role-playing game involving three roles: the Judge, the Non-Pretender, and the Pretender. Both the Judge and the Non-Pretender are experienced in a selected domain, while the Pretender is played by an LLM. The Judge’s task is to ask questions they believe will help reveal the true identity of the respondents. The Non-Pretender answers sincerely, based on their real experience, while the LLM is prompted to respond as if it were a domain expert. For each question, the Judge receives both answers, without knowing which is which, and must decide which response comes from the human expert, indicate their level of confidence, and explain their reasoning.
In a pilot study where rock climbers were tasked with identifying whether a response came from a human rock climber, or ChatGPT imitating a climber, results showed that while the model could convincingly reproduce climbing jargon and domain-specific knowledge, it lacked competencies related to embodied and emotional experiences, such as fear and training discipline [26]. The method not only showed how climbers perceived the diference between human- and machine-generated responses, but also revealed the actual diferences in capabilities. The research thus explored the boundaries between human and machine competence of the social world of rock climbing.
While the study on rock climbing showed the value of the COMIG for research on context-specific human-machine diference, the potential of the method for promoting AI literacy remains largely unexplored. This is the aim of the current project: we develop and test the Competence Imitation Game app as a pedagogical tool that engages participants, ranging from pupils, students, educators and professionals, in context-specific exploration and reflection on the nature and diference between human and machine competences. By embedding the game into specific domains of practice and spaces of learning, the project seeks to generate an understanding of what AI can and cannot do in relation to specific tasks and to invite learners to consider why those distinctions matter for their own practices and everyday lives.
Competence Imitation Games are arranged based on specific topics, selected to reflect domains of insider knowledge and experience shared by the participants. These could include high school mathematics, school life, friendship, parenting, or professional fields such as teaching, journalism, therapy, or social work — any context characterized by a specific competence domain. While the game typically focuses on such domain-specific expertise, it can also be played as a classical Turing Test, addressing what it means to be human in general. The organizer can select the topic and develop the prompt according to their own context and needs, making the game adaptable to a wide range of learning environments. The goal is to support participants in developing AI literacy through a collaborative exploration of the limits and capabilities of AI in relation to the competences shared among peers both during the game, and post-game group discussions.
While the core structure of the game remains relatively fixed, its practical implementation can take many forms. Variations include playing individually or in groups, experimenting with diferent game topics, incorporating specific educational themes in post-game discussions, and integrating the game into classroom settings or professional contexts. These elements are explored and developed through a co-design process. In what follows, we present the rationale, design philosophy, and initial plans for the co-design process of the COMIG app and educational approach aimed at developing AI literacy through role-based, human-machine interaction.
Unlike existing educational interactive tools for AI literacy, such as GenAI Teachable Machine [27] and Somekone [28], which focus on learning outcomes related to specific and well-defined AI topics (e.g., training and deploying classifiers, recommendation algorithms), the COMIG does not seek to convey a predetermined body of knowledge. Instead, by inviting participants to distinguish between human and AI-generated responses based on their own domain-specific understanding, each gamedialogue represents the unique features of the group at play, and the learning outcome is emergent and open-ended. In fact, each game is an opportunity for the participants to act as proxy researchers and collectively discover the limits of AI capabilities in their own particular social world. This shift from propositional to procedural learning has important implications for both design and evaluation. In contrast to tools where success is measured by task completion or correct application of predetermined concepts, our app is designed to support learning that is emergent, collective, and grounded in dialogue among peers. Success is not determined by whether a participant “learned the principles of machine learning” but by whether they are able to articulate, reflect on, and question the nature of intelligence, competence, and machine integration in their specific domain.
The design of the COMIG is guided by a set of assumptions: 1) there are a multitude of social worlds, characterized by distinct shared activities and competences, including tacit knowledge, which is hard to explicate and transmit through text; 2) those fluent in these practices are considered experts within their domains; 3) AI literacy may involve a varied set of competences from one social world to another; 4) experts within these contexts possess insider perspectives that external AI literacy educators may lack; 5) a central component of AI literacy is exploring the boundaries of human and machine competence in relation to specific practices; 6) peer interaction, group membership, and shared identity are essential for surfacing context-specific AI literacy.
Thus, the app is envisioned not merely as a game, but as a learning environment that supports participants to collectively articulate the contours of their own expertise, explore what is uniquely human, and reflect on the role of AI in their lives.
The COMIG is being developed through a co-design process which began in March, 2025. So far, the process has included a seven-day design sprint with a team of researchers from sociology, education, and computer science, as well as educators, programmers, designers, and domain-experts (e.g., educators and learners across K-12 and higher education) who act as both testers and informants. Currently, we work through iterative cycles of feedback and prototyping. Prototypes of the app have already been tested among teacher education students, linguistics students, and in-service programmers at the University of Helsinki, Finland. Each session produced distinct game-dialogues, validating the method’s potential while also informing further development. Upcoming classroom trials will involve pupils (aged 13–16 years ) in social studies and religion.
We adopt an emergent framework that combines qualitative and quantitative dimensions for evaluating the app and its use. The analysis focuses on how, and how accurately, participants identify each other during the game, reported new insights and experiences of learning, quality and content of post-game discussions and afective engagement. Data will include surveys, game dialogues, recordings of group interactions during the game and post-game discussions, including selected interviews. The results will inform the design process for further refinement before introducing the app for larger-scale testing. The final app will have functionality for easy and flexible use in group settings, and for data collection to support continuous research.
By centering the learning process on collegial exploration of the diference between human and machine competence, emerging through collective eforts by domain experts to identify their peers from LLMs imitating them, the COMIG ofers a new model for researching and promoting context-specific AI literacy rooted in situated judgment and domain-specific knowledge. Rather than focusing on predetermining the content of AI literacy, we create a space in which the complexities of machine participation in human life can be explored contextually and collaboratively by domain experts.
The author(s) used ChatGPT-5 for proofreading, including spelling checks and suggestions to improve the writing. After using this tool, the author(s) reviewed and edited the content as needed and take full responsibility for the publication’s content. intelligence (ai) literacy in teacher education: A scoping review, Computers and Education Open 6 (2024) 100169. URL: https://www.sciencedirect.com/science/article/pii/S2666557324000107. doi:10.1016/j.caeo.2024.100169, accessed January 22, 2025. [19] A. M. Turing, Computing machinery and intelligence, Mind 59 (1950) 433–460. URL: http: //www.jstor.org/stable/2251299, accessed via JSTOR. [20] H. Collins, R. Evans, R. Ribeiro, M. Hall, Experiments with interactional expertise, Studies in History and Philosophy of Science Part A 37 (2006) 656–674. URL: https://linkinghub.elsevier. com/retrieve/pii/S003936810600080X. doi:10.1016/j.shpsa.2006.09.005, accessed February 6, 2023. [21] H. Collins, R. Evans, Quantifying the tacit: The imitation game and social fluency, Sociology 48 (2014) 3–19. URL: http://journals.sagepub.com/doi/10.1177/0038038512455735. doi:10.1177/ 0038038512455735, accessed February 6, 2023. [22] I. Arminen, O. E. A. Segersven, M. Simonen, Active and latent social groups and their interactional expertise, Acta Sociologica 62 (2019) 391–405. URL: http://journals.sagepub.com/doi/10.1177/ 0001699318786361. doi:10.1177/0001699318786361, accessed February 8, 2023. [23] O. Segersven, Identifying Social Groups, Boundaries and Cultural Competence – An Imitation
Game Inquiry, Ph.D. thesis, University of Helsinki, 2023. Doctoral dissertation. [24] R. Wehrens, The potential of the imitation game method in exploring healthcare professionals’ understanding of the lived experiences and practical challenges of chronically ill patients, Health Care Analysis 23 (2015) 253–271. doi:10.1007/s10728-014-0273-8. [25] Airaksinen, Tiina and Koponen, Heikki, IMITAATIOPELI: Opas pelin järjestämiseen, https://link.
otavia.fi/imitaatiopeliopas, 2020. Accessed January 9, 2025. [26] O. Segersven, I. Arminen, Can a machine ‘talk’ like a rock climber? a turing test on rock climbing,
Discourse, Context & Media 67 (2025). doi:10.1016/j.dcm.2025.100915. [27] N. Pope, J. Kahila, H. Vartiainen, M. Tedre, Children’s ai design platform for making and deploying ml-driven apps: Design, testing, and development, IEEE Transactions on Learning Technologies (2025) 1–16. URL: https://ieeexplore.ieee.org/abstract/document/10842355. doi:10.1109/TLT. 2025.3529994, accessed January 27, 2025. [28] N. Pope, J. Kahila, J. Laru, H. Vartiainen, T. Roos, M. Tedre, An educational tool for learning about social media tracking, profiling, and recommendation, in: 2024 IEEE International Conference on Advanced Learning Technologies (ICALT), 2024, pp. 110–112. doi:10.1109/ICALT61570.2024. 00038.