Smart Learning Environments (SLEs) typically use Learning Analytics (LA) to personalize learning experiences. Nevertheless, the adoption of such technological innovations may be a challenging process for the teachers, and their agency may be threatened, especially if they do not intervene in the design of such innovations. One alternative to face this challenge is to involve teachers in the design and implementation of learning experiences supported by SLEs. However, more insights are needed on how teachers, developers, researchers, and other stakeholders can achieve an equitable agency by means of implementing a learning scenario supported by LA in which algorithms have their own agency. This paper reports a case study in which a higher education teacher and the SLE developer/researcher were involved in the co-design process and produced the course learning scenario. The study aimed at achieving a better understanding of how the co-design process of the implementation of the SLE in the course could help increase teachers' agency. Preliminary results suggest that human-centered approaches when designing LA-based systems may contribute to addressing potential threats to teacher agency.
Learning Analytics (LA) provides insights into the students’ behavior by monitoring their
digital traces, and thus, potentially optimizing the decisions related to the learning processes
[
In this regard, achievement of agency is tightly linked with the degree of control that agents exhibit in their actions across past, present, and future dimensions [9, 10]. In the educational context, according to Priestley et al. [11], teacher agency constitutes an emergent phenomenon that can be achieved by individuals by means of their acts, and as a result of the interplay of personal capacities, resources, affordances, and constraints embedded in the environment. The model (see Figure 1) suggests that the achievement of agency is informed by teachers’ professional and personal experiences. Even though the achievement of agency is futureoriented, it is enacted in a particular situation, either constrained or supported by environmental elements.
One of the most prominent contextual resources is technology, which can shape the space of opportunities for teachers and ultimately transform teacher agency [13]. Focusing on the specific functionalities that SLEs can provide (e.g., personalized recommendations to the students based on reaction scripts), it can be argued that a system with a certain level of smartness (e.g., on the rules that trigger reactions) may contribute to mine teacher agency. Consequently, it can have potentially negative effects on human control and oversight [6], e.g., hindering teachers’ capacity to monitor the ongoing classroom development. Agency can be enhanced by involving stakeholders in the design, deployment, and assessment of LA-based technologies. This is aligned with recent human-centered approaches to the design of LA-based solutions, which advocate for co-design and participatory design processes, so that students’ and teachers’ voices and experience may allow for wider adoption and refinement of the LAbased tools [14]. A representative approach is co-design, which stands for a team-based, highly guided process in which teachers, students, developers, researchers, and other stakeholders work together to pursue the materialization of an educational innovation. Thus, co-design is motivated by an innovation challenge in which involved stakeholders have well-defined roles and work actively with a high level of involvement through the process [15]. The field of LA has already begun to explore the inherent tensions involved in co-designing LA-based tools with both teachers and learners; including these stakeholders in the entire process of design is thought to benefit the whole learning ecosystem [16]. One of the key goals of designing technologies following human-centered approaches is not only to create more effective tools (in terms of performance), but also to ensure that all relevant stakeholders are involved in the process. This helps to promote the development of ethical, lawful, and reliable tools, which are essential for building trust and ensuring the widespread adoption of the created tools [17]. However, involving other stakeholders and effectively implementing human-centered design (HCD) approaches adds an extra challenge to technology researchers and developers, and may be the reason why few LA solutions have been developed with teachers’ involvement [18]. Fully adopting human-centered design methods implies attending to the needs of the critical stakeholders and identifying their needs in the ecosystem in which the designed tool will operate [19]. Even if the LA-based environment has not been designed through a humancentered approach, at least the learning scenario in which the LA-based SLE will be deployed should be jointly co-designed with teachers and the rest of the critical stakeholders. As Jørnø et al. [13] pointed out, the implications of integrating technologies that offer adaptive support have yet to be fully disclosed or appropriately addressed by system developers.
Moreover, research is still needed to understand teachers’ beliefs and attitudes (which should ultimately allow empowerment of teacher agency) towards data use and management (which the smart component of an SLE can handle) [20]. To gain an understanding of the need to address the aforementioned challenges, we defined the following research question (RQ): How can a co-design process of a learning scenario to be supported by an SLE foster teacher agency? In this paper, we present the preliminary results of a case study aiming to shed light on the above RQ. The rest of this paper is structured as follows. Section 2 presents the methodology, followed by the study design and the data collection and analysis approaches. Section 3 shows the preliminary results. Finally, Section 4 includes a discussion on the findings and sets the future lines of research.
This study was framed under an interpretive paradigm, as the aim is to gain a deep understanding of the particular phenomena to be studied, and we assume that participants build their own subjective meaning as they interact with the environment [21]. Therefore, this study does not aim at generalizing its findings and eventual conclusions, but to understand the complexities of a complex phenomenon, and eventually inform and transfer knowledge to similar studies. The study took place in a mid-sized Spanish university course on School Organization and Planning, which involved one teacher and 71 pre-service kindergarten teachers. The course was designed by the teacher who had seven years of experience in higher education. The educator was willing to implement a learning module in which SCARLETT (Smart Context-Aware Recommendation of Learning Extensions in ubiquiTous seTtings), an SLE developed by a collaborating research group, was expected to provide support. SCARLETT recommends personalized, geolocated learning tasks in informal settings (e.g., streets, public buildings) connected with the formal contents taught in the onsite classroom or in the Learning Management System (LMS) [22]. Once the teacher defines the learning objectives and students’ contextual information, the SLE performs the following actions: (i) it collects students’ data from the learning situation (the action described above as ‘sense’), (ii) it monitors students’ progress and derives relevant LA indicators (the action described above as ‘analyze’), and ultimately (iii): it provides personalized recommendations, through a web-based app (the action described above as ‘react’). Figure 2.a illustrates the learning scenario of the course and its relation to the supporting SLE. An initial quiz was implemented via Kahoot! so that students could detect the contents that they need to reinforce, followed by field research across the playgrounds of the primary schools. The learning objective of the learning scenario was to review content related to the spatial and temporal organization of schools, and to school management bodies and school planning documents. It is noticeable that SCARLETT was developed (its first version) solely by researchers and developers who did not implement human-centered approaches when developing internal processes of the SLE, such as (i) which interactions are traced, (ii) which indicators are computed based on the data traces, (iii) how these indicators are embedded in the learner model, or (iv) what are the rules guiding the reaction scripts that generate the personalized recommendations. Teacher agency in this scenario may have been threatened already due to not following an HCD approach for designing the SLE. Figure 2.b represents the flow of activities. However, the co-design of the learning situation for the previously mentioned course, i.e., the joint development of the specific learning scenario of the SLE-supported course, provided an opportunity for both developer and teacher to gain understanding of how SCARLETT could help the teacher to achieve her agency. a a
Organization and School Planning Course Components
of the co-designed learning scenario
Initial quiz via Kahoot!
Field research Sense Learner traces
Analyze
React Learner model
Learner's context
Reaction Scripts
Notifications Related resources Personalized quiz b b SCARLETT
Developer Initially developed without following Human-Centered Design approaches
Co-design of the learning scenario
Learning scenario implementation
As this study aimed at better understanding how the co-design process might affect teacher agency, a Case Study was chosen as the approach to explore the previously described real-life, contemporary bounded system (the case) over a period of time [24]. This case study was particularly structured following the guidelines proposed by Stake [25] (e.g., delimiting clear boundaries, collecting data from multiple sources, looking for patterns and themes across data). To operationalize the aforementioned RQ, we formulated an issue (I): How can a co-design process of a learning scenario to be by SCARLETT foster the agency of a higher education teacher? Subsequently, we have defined three topics related to the posed RQ. Topics are defined by researchers to anticipate areas of the RQ where tensions are likely to arise in the study [25]. Topic 1 (T1), Learning design, aims at identifying existing limitations and potentialities encountered to create the learning scenario (e.g., SCARLETT’s affordances, communication processes among stakeholders). Topic 2 (T2), Teacher agency, refers to teacher agency in the codesign process (following the model shown in Figure 1), i.e., to understand how the teacher exerts her professional practice in accordance with her previous experiences, with available resources, and seeks the achievement of short-term goals (e.g., successfully integrating an SLE in the learning situation). Specifically, we aimed at comprehending how the teacher perceived and co-existed with the “smartness” of the SLE, eventually unveiling threats and opportunities for achieving agency. The third topic (T3), Developer role and agency, deals with understanding the developer’s role, i.e., how the actions implemented by the tool’s developer may have influenced the whole process, particularly focusing on understanding if the developer’s decisions, expectations, and actions were supporting or interfering with the teacher’s agency. Agency-oriented topics (T2 and T3) have been approached through the ecological model of agency [11], which has been proven as a helpful analytical tool to guide empirical research [26]. Each topic was illuminated by several informative questions. This way, an anticipatory data condensation schema (see figure 3) was initially developed [27]; coding was carried out by one researcher through a combination of inductive and deductive coding.
RQ: How can a co-design process of a learning scenario to be supported by and SLE foster teacher agency? I: How can a co-design process of a learning scenario to be supported by SCARLETT foster the agency of a higher education teacher?
IQ 1.1.
T1 Learning design
IQ 1.2.
IQ 1.3.
T1 Learning design IQ1.1. What has been the process followed by the teacher to create the learning scenario? IQ1.2. What limitation has the teacher encountered to create the learning scenario? IQ1.3. To what extent has the teacher been able to define the main parameters (i.e., type and number of activities) of the learning scenario? TaegaeTc2nhceyr 2I.Q1.2I.Q2. IIItTQQQh2r222eT...a123et...aeDWWcnohehheedaasrttinataahgrrteeeheentttehhclaeeeycammhrneaairiinnnpgoteshprcrcpeeeoanirvtatseurtitnoohi?tatihetesthatehgiaertnatcgheyinsocsfyctheisenateriaocher?
provides to the development of the agency of the teacher? DevTe3loper 3I.Q1. raogleenacnyd 3I.Q2.
T3 Developer role and agency IQ3.1. How is the developer supporting or interfering with the teacher and the learning scenario?
IQ3.2. What was the developers’ main purpose?
Several researchers checked the interpretations to ensure trustworthiness [28]. Regarding data sources: two interviews were conducted with the teacher; the first one took place before implementing the learning situation [INT-T-A], and the latter took place when the enactment of the learning scenario concluded [INT-T-B]. Another interview was conducted with the main developer of the system once the learning situation concluded [INT-D]. Moreover, the learning scenario itself constituted another source of data. The process of co-design was documented since the agreement of collaboration between the teacher and the developer was reached. In that process, the developer informed the teacher about the minimum requirements so that SCARLETT could provide personalized support. The teacher created an initial draft of the learning scenario, which was later shared with the developer/researcher. The learning scenario fine-tuning took place after a demo session (in which the teacher could integrate the contents and the learning objectives in SCARLETT’s environment) and a final meeting in which other researchers discussed the adequacy level of the learning scenario for SCARLETT’s purposes.
This section presents the preliminary findings in correspondence to the aforementioned topics. With regard to T1 (learning design), the preliminary results showed that the learning scenario, as the product of the co-design, was tailored to fit SCARLETT’s affordances (e.g., selected learning tools, activity goals). Thus, the learning scenario followed the minimum requirements that the SLE needed to perform its actions. The first activity of the learning scenario, the completion of an online quiz to know students’ previous knowledge on the course topic, constituted the major data source accessible by the sensing module of the SLE. Along with the second activity, students were asked to perform geolocalized tasks (in primary schools in Spanish cities) by using an ad-hoc-developed, web-based app, which sent them recommendations considering their previous performance in the test and their physical location, as well as their level of engagement with previous recommended tasks. The final phase of the learning scenario, i.e., the field research, was another geolocated activity in which SCARLETT did not directly intervene, but its previously collected data informed the teacher to design tasks implemented in this phase. The activities of the learning scenario emerged as a result of a high number of internal communications between the teacher and the developer teacher, shared drafts, and informal meetings.
Regarding T2, teacher agency, it is relevant to point out that the teacher's attitudes towards adaptive technologies were positive, possibly influenced by her prior technological knowledge, as she had experience with technological tools that aim to support ubiquitous learning situations. Before the implementation, the teacher believed that the SLE could have potential benefits for her students’ learning: “To the best of my knowledge at this point [when the learning scenario was not fully conceived], the tool is going to boost students’ learning as it will contribute to managing their learning progress [...]. Regarding myself, I still have to learn how I can access and take full advantage of data” [INT-T-A]. Since the teacher was not fully aware of SCARLETT’s affordances at early stages, she could not design the final phase: “I need to know exactly how I'm going to be able to visualize the data. So, until I have that information, I am not going to be able to have a clear understanding of SCARLETT’s functionalities” [INTT-A]. This difficulty was progressively solved by means of a demo session and informal meetings to the extent that, for the teacher, it was crucial to trust the tool's fault tolerance and corresponding reliability: “At initial stages of the co-design, I was not fully aware of SCARLETT’s affordances. Thankfully, with the help of the developer, through the meetings we had to discuss how the learning scenario could be implemented. Thus, I gained some insights on how to manage the situation” [INT-T-B].
With respect to T3, the developer role and agency, SCARLETT’s developer showed a strong will to support the learning scenario as much as possible. Adaptations of the tool were needed, as well as real-time monitoring of the scenario development since some technical failures of the tool emerged: “We had to perform special arrangements on the tool, which increased the workload. The tool had to fit the technical requirements of the course’s Learning Management System [Moodle] and resources provided by the teacher [Kahoot! for implementing the quiz]” [INT-R].
Preliminary analysis of the case study shows how a teacher can partially achieve her agency by means of the co-design process of the learning scenario. Although the initial design of the tool did not consider the teacher’s perspectives (i.e., a human-centered process had not been carried out), the teacher could understand the basis of the developed SLE functioning by means of maintaining a fluid and periodic communication with the developer when creating and finetuning the learning scenario. The level of commitment acquired by the studied stakeholders in the co-design of the learning scenario was proved to be effective, as both teacher and developer ended up obtaining a learning scenario that suited their needs and was aligned with their agency (in the case of the teacher, having a tool that could help her students to access formal learning contents in informal settings).
The teacher showed her willingness to share the control of the learning situation with the autonomous recommendations provided by the SLE. However, the teacher was not fully aware of the affordances and functioning of SCARLETT. Thus, her range of action was restricted; in other words, she could not completely control the learning scenario. This study showed why the co-design process of a learning scenario was useful to overcome initial issues. Nevertheless, having adopted an HCD approach when designing the SLE itself could have improved SCARLETT’s trustworthiness [6]. This study suffers from some limitations regarding its contextual and temporal features. SCARLETT’s implementation involved just one teacher with her specific views and experiences towards the use of technological support in ubiquitous learning contexts. Co-designing similar learning scenarios with the support of other SLEs may result in different outcomes on teacher agency. As stated previously, these preliminary results cannot be generalized or extended to others, due to the uniqueness of the learning scenario and its involved human stakeholders. Moreover, the fact that the SLE was developed and maintained by the associated research group with whom the teacher has been collaborating enabled the developer to react to technical problems on the fly but added even a higher level of uniqueness to a study of this nature, as it was also pointed out in [29]. Future work should point towards studying the impact of involving educational stakeholders at early stages of the design process of LA tools to identify particular needs to be covered, given that co-design at early stages may be a way to democratize LA solutions and comply with ethical standards [14, 30]. Then, exploring the concept of agency in LA-enhanced contexts in more depth stands out as one of the focuses of our research agenda. There are still many challenges surrounding the implementation of LA in authentic educational settings. For instance, what are the main barriers hindering the adoption and effective integration of LA-based tools, and to what extent can these challenges be mitigated?
This manuscript reports preliminary findings; more evidence from non-reported data sources is being analyzed. We expect to provide a full comprehension of the phenomena by connecting both the affordances and functioning of the SLE to dimensions of teacher agency. Even though addressing the problem around algorithmic agency and teacher agency was not the focus of this study, we acknowledge its relevance for the Human-Centered Learning Analytics research community. Future studies should try to position human-centeredness, as it stands out as a feature of intelligent systems that have been designed through the identification of the critical stakeholders, their relationships, and their opportunities to assist them to achieve agency [11, 19, 31]. In conclusion, we argue that an early involvement of teachers (and other educational and non-educational stakeholders) in the complete design process of LA tools and their implementation in authentic contexts can have potential benefits in their agency achievement. Increasing the participation of teachers in LA-enriched settings and ultimately empowering their roles has been acknowledged as one crucial challenge of the LA research community [32].
This research has been partially funded by the Spanish State Research Agency (MICIU/AEI/1013039/501100011033) together with the European Regional Development Fund, under project grants PID2020-112584RB-C32 and PID2023-146692OB-C32. Víctor Alonso-Prieto has received funding from the call for UVa 2021 pre-doctoral contracts, co-financed by Banco Santander. The authors would like to express their gratitude to Vanesa Gallego-Lema and Sergio Serrano-Iglesias for their participation in this research. The authors also extend their thanks to Eduardo Gómez-Sánchez and Miguel L. Bote-Lorenzo for their involvement in the co-design.
The authors have not employed any Generative AI tools. Proceedings of the Annual Hawaii International Conference on System Sciences, vol. 2016March, (2016) 4597–4605. doi: 10.1109/HICSS.2016.571. [5] A. Bandura, Toward a Psychology of Human Agency: Pathways and Reflections,
Perspectives on Psychological Science, 13.2 (2018) 130–136. doi: 10.1177/1745691617699280. [6] High Level Expert Group on AI, Ethics guidelines for trustworthy AI, European
Commission, Brussels, 2019. [7] B. Shneiderman, Human-Centered AI. Oxford University Press, 2022. [8] T. Valtonen, T. Paavilainen, S. López-Pernas, M. Saqr, and L. Hirsto, Elementary and Secondary School Teachers’ Perceptions of Learning Analytics: A Qualitative Approach, Technology, Knowledge and Learning (2025) 1-19. doi: 10.1007/s10758-025-09847-5. [9] G. Biesta, M. Tedder, Agency and learning in the lifecourse: Towards an ecological perspective, Studies in the Education of Adults, 39.2 (2007) 132–149. doi:10.1080/02660830.2007.11661545. [10] M. Emirbayer and A. Mische, What is agency?, American Journal of Sociology, 103. 4 (1998) 962–1023. doi: 10.1086/231294. [11] M. Priestley, G. Biesta, S. Robinson, Teacher agency: What is it and why does it matter?, in: R. Kneyber, J. Evers (Eds.), Flip the System: Changing Education from the Ground Up, Routledge, London (2015) 134–148 doi:10.4324/9781315678573 [12] M. Priestley, G. Biesta, and S. Robinson, Teachers as Agents of Change: An exploration of the concept of teacher agency, 2012. doi: 10.5040/9781472553195.ch-010. [13] R. L. Jørnø, B. L. Andersen, and P. Gundersen, The imaginary of personalization in relation to platforms and teacher agency in Denmark, Nordic Journal of Studies in Educational Policy, 8.1 (2022) 20–29. doi: 10.1080/20020317.2021.2022073. [14] J. P. Sarmiento and A. F. Wise, Participatory and Co-Design of Learning Analytics: An Initial Review of the Literature, in: Proceedings of the 12th international learning analytics and knowledge conference, ACM International Conference Proceeding Series, 2022, pp. 535–541. doi: 10.1145/3506860.3506910. [15] J. Roschelle, W. R. Penuel, N. Shechtman, Co-design of innovations with teachers: Definition and dynamics, Proceedings of the International Conference of the Learning Sciences (ICLS) 2 (2006) 606–612. [16] K. Holstein, B. M. McLaren, and V. Aleven, Co-designing a real-time classroom orchestration tool to support teacher–ai complementarity, Journal of Learning Analytics 6.2 (2019): doi: 10.18608/jla.2019.62.3 [17] C. Lang and L. Davis, Learning Analytics and Stakeholder Inclusion: What do We Mean When We Say ‘human-Centered’?, in: Proceedings of the 13th International Learning Analytics and Knowledge, ACM Conference Proceeding Series, 2023, pp. 411–417. doi: 10.1145/3576050.3576110. [18] M. J. Rodríguez-Triana, L. P. Prieto, A. Martínez-Monés, J. I. Asensio-Pérez, and Y.
Dimitriadis, The teacher in the loop: Customizing multimodal learning analytics for blended learning, in Proceedings of the 8th international conference on learning analytics and knowledge, ACM, 2018, pp. 417–426. doi: 10.1145/3170358.3170364. [19] S. B. Shum, R. Ferguson, and R. Martinez-Maldonado, Human-centred learning analytics,
Journal of Learning Analytics, 6.2 (2019) 1-9. doi: 10.18608/jla.2019.62.1. [20] A. Datnow and L. Hubbard, Teacher capacity for and beliefs about data-driven decision making: A literature review of international research, Journal of Educational Change, 17.1 (2016) 7–28. doi: 10.1007/s10833-015-9264-2. [21] W. J. Orlikowski, Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations, Organization Science, 11.4 (2000) 404–428. doi: 10.1287/orsc.11.4.404.14600. [22] S. Serrano-Iglesias, E. Gómez-Sánchez, M. L. Bote-Lorenzo, J. I. Asensio-Pérez, A. RuizCalleja, and G. Vega-Gorgojo, Demonstration of SCARLETT: A Smart Learning Environment to Support Learners Across Formal and Informal Contexts, in T. De Laet, R. Klemke, C. Alario-Hoyos, I. Hilliger, A. Ortega-Arranz (Eds.), Technology-Enhanced Learning for a Free, Safe, and Sustainable World, International Publishing, Cham, 2021, pp. 404–408. doi.org/10.1007/978-3-030-86436-1_44. [23] V. Alonso-Prieto, Studying the Impact of Orchestrating Intelligent Technologies in Hybrid Learning Spaces for Teacher Agency, in: D. Di Mitri, A. Ortega-Arranz, O. Poquet (Eds.), Proceedings of the Doctoral Consortium of the 18th European Conference on Technology Enhanced Learning (DCECTEL 2023), co-located with the 18th European Conference on Technology Enhanced Learning, Aveiro, Portugal, 2023. URL: https://ceurws.org/Vol-3539/paper13.pdf [24] J. W. Creswell, Qualitative inquiry & research design: choosing among five approaches, 3rd ed., SAGE, Los Angeles, 2013. [25] R. E. Stake, The Art of Case Study Research. Sage, Thousand Oaks, 1995. [26] C. Zhang and J. Wright, Teacher Agency: A Two-level Ecological Model, English21, 3.4, (2017) 441-459. doi: 10.35771/engdoi.2017.30.4.020. [27] M. B. Miles, A. M. Huberman, and J. Saldaña, Qualitative Data Analysis: A Methods
Sourcebook, 3rd ed., SAGE Publications, Thousand Oaks, CA, 2015 [28] E. G. Guba, Criteria for Assessing the Trustworthiness of Naturalistic Inquiries,
Educational Resources Information Center Annual Review Paper, 29.2 (1981) 75–91. [29] A. Ruiz-Calleja, M. L. Bote-Lorenzo, J. I. Asensio-Pérez, S. L. Villagrá-Sobrino, V. AlonsoPrieto, E. Gómez-Sánchez, P. García-Zarza, S. Serrano-Iglesias, and G. Vega-Gorgojo, Orchestrating ubiquitous learning situations about Cultural Heritage with Casual Learn mobile application. International Journal of Human-Computer Studies 170 (2023), International Journal of Human-Computer Studies, 170 (2023) doi: 10.1016/j.ijhcs.2022.102959. [30] M. Deschênes, Recommender systems to support learners’ Agency in a Learning Context: a systematic review, International Journal of Educational Technology in Higher Education, 17. 1 (2020): doi: 10.1186/s41239-020-00219-w. [31] A. Mouta, E. M. Torrecilla-Sánchez, and A. M. Pinto-Llorente, Comprehensive professional learning for teacher agency in addressing ethical challenges of AIED: Insights from educational design research, Education and Information Technologies, 30.3 (2024) 33433387. doi: 10.1007/s10639-024-12946-y. [32] R. Ferguson, Ethical challenges for learning analytics, Journal of Learning Analytics, 6.3 (2019) 25–30. doi: 10.18608/jla.2019.63.5.