In this paper we propose a number of approaches to using formative design interventions to enable secondary school teachers to inform future design of artificial intelligence. The aim is to provide them with increased control, responsibility, and accountability for the deployment of AI-based applications in education to ensure fairness. The motivation for this project draws on results from our prior research on adaptive digital textbooks with AI-based technology. Participatory design has been recognized as a way of exploring workers knowledge and gaining knowledge of workplaces to improve system design when building new tools. However, we argue that applying PD methods in design of AI-based applications is somewhat different. Our intention with this paper is to introduce a discussion of methods and techniques for user involvement in design of AI, as well as to propose a possible remedy i.e., meta design.
Participatory design (PD) has traditionally been a way to understand technology and how technology
could be integrated in work activities, and also to empower future users participating in design and use
of technology [
We propose that using formative design interventions could enable users (in our case secondary
school teachers) to inform future design with the aim of providing them with increased control,
responsibility, and accountability for the deployment of AI-based applications. In our case we are
particularly interested in AI systems in education (AIEd) in realtion to the concept of fairness. The
work presented here is preparatory for future project, so to this point we have not yet started data
collection, rather we intend to introduce a discussion of methods and techniques aiming for participants
in a PD project to engage in telling of stories, making of things, and enacting possible futures [
Our research project stems from the fact that intensified use of educational technologies in schools
have led to a growing proportion of digital data being integrated in teachers’ and students’ everyday
work [
There is an ongoing discussion about potential risks of harm and unfairness dependent on
algorithmic discrimination, which has been demonstrated in areas such as criminal justice [
The remainder of this paper is structured as follows. In the next section we briefly highlight aspects of fairness in AI since that is the motivation for our research project and what the participants are going to work with. The next section introduces formative interventions and meta-design in relation to our project, followed by a section where we argue for PD but also highlight challenges when involving AI technology. Finally, we present our suggested methods and techniques in the PD process.
Choices during the design process reflect designers’ and developers’ ethical values, knowingly or
not, and shape the technology and in the long-term shapes people’s life [
Further, the data set that are used to train algorithms can be endowed with societally prejudices encompassed with historical experiences and produce biases. Algorithms are based on machine learning that “find patterns within datasets that reflect our own implicit biases and, in so doing, emphasize and reinforce these biases as global truth” [23, p. 1524]. The language translation Google Translate offer one such example when training data reflected societal bias, reinforced gender roles, and maybe amplified them. Some languages gender nouns and some are gender neutral, and the translation to feminine and masculine forms reinforced gender bias and changed gender in a stereotypical way. Google translated for example presupposed doctors to be males, nurses to be females, and that he works, and she cooks.
An algorithm is designed as a statistical model of reality to predict potential outcomes and even a
complex model is inevitably a simplification. A teaching and learning activity consist of a multitude of
intertwined relations and the complexity in educational contexts cannot be underestimated [24]. It
follows that AI technology needs to capture a messy, interactive, changeable and context bound world
[25]. It is important to consider how groups are represented, which people belong to a group, and not
treat diverse groups as a single entity. Baker and Hawn [
Kitchin [26] has addressed that data largely have come to be pre-analytical and pre-factual, meaning that data representing the truth. As such, algorithms are approached as inherently neutral and formal constructs. Similarly, Birhane [25] puts forward that in these times when social activities are becoming increasingly automated through algorithmic decision making and predictions, social activities are at the same time being transformed and aimed to be understood with a mathematical solution. The problem to be solved by algorithms are formulated as a mathematical model and unfair results are treated with a rational and logical solution. In line with this, Akgun and Greenhow [27] discuss AI in K-12 education and highlight that: “The ethical challenges and risks posed by AI systems seemingly run counter to marketing efforts that present algorithms to the public as if they are objective and value-neutral tools” (p. 4). For example, during the COVID-19 pandemic, students in the United Kingdom were awarded A-level grades through application of an algorithm. However, the algorithm was inconsistent and unfair, and in favor of private/independent school students whilst those from disadvantage backgrounds were negatively affected. Smith [28] stressed that it had to become uproar and public pressure for the government to make a U-turn and abandon the algorithm and issue correct grades. In summary, aspects of fair treatment of students are a huge challenge to AI in education. It is also the case that fairness must be understood as local and situated in particular educational practices. What might be fair to one group of students, in one set of activities, might be highly biased in another group. To address this issue, it is imperative that teachers are involved in the design process, as well as to investigate possibilities of adapting already deployed systems. We argue that meta-design [30] could be a possible form for allowing local adaptation of AIEd.
To identify relevant forms for meta-design, i.e., how it would be possible and relevant for teacher to adapt their AI-based applications to their classroom, we engage with teachers in PD. To formulate suitable interventions, we draw on Activity Theory [29]. Our unit of analysis will be the work activity (teaching) conceptualized as a collective activity system that participants can re-design and transform by identifying and solving problems and contradictions. Developing formative interventions for metadesign will allow teachers to rethink their role in their interaction with AIEd and will offer them methods they can use to find solutions to address evolving circumstances and pedagogical challenges. Therefore, we will engage with teachers, researchers, and other stakeholders (e.g., students, developers, and school leaders) in formative interventions, an approach that builds on and purposefully fosters participants’ agency. As researchers, we will set up these interventions to investigate the possibilities for educational improvement by bringing in a new actor - namely, adaptive AI-based teaching materials - potentially seen as a problematic and contradictory object which can generate conflicts and resistance when used by teachers. This collective design effort is seen as part of an expansive learning process including participatory analyses and implementation phases. In fact, during these interventions, teachers and other participants will be involved in negotiations and debates to make their voices heard. Besides, the collaboration between teachers, researchers, and the other stakeholders within these formative interventions can result in the construction and implementation of a new organization of work by the collective. The result of these interventions is not known ahead of time to the researchers, as the outcome is determined by participants.
The purpose of our interventions is to develop new models for increasing the agency of teachers concerning the design and use of AI-based applications in education to ensure fairness. Based on previous work on algorithmic fairness we argue for an increased focus on situated fairness, i.e., fairness in practice. Such a perspective also addresses the difficulties and complexities of achieving fairness in algorithms and allows for teachers to compensate to ensure fairness in their practice. To reach the purpose of this project, we will develop formative interventions involving teachers and other stakeholders to aim to: 1. Identify the affordances and constraints of the AI-based application and how teachers would use it to appropriate and adapt it in their local practices. 2. Describe how and under what circumstances teachers perceive the AI-based application to benefit their local teaching practices. 3. Identify with teachers and other school stakeholders which features of the system can facilitate fairness and minimize bias for vulnerable groups.
We will focus our investigation on the potential of adaptive AI-based teaching materials. This type of application is intended to serve as a text for a course and integrates an intelligent tutoring system. We have chosen it because it is in use and is commercialized by a large established publisher in Sweden.
We argue that it is necessary to identify how teachers interpret fairness in their local situations and to ensure that their interpretations underlie concrete system functionalities. Involving teachers is crucial as the interaction between them and AIEd should be one in which teachers become able not only to understand but also challenge algorithmic decisions and predictions. 4.1.
Participatory design has been recognized as a way of exploring workers (tacit) knowledge and
gaining knowledge of workplaces to improve system design when building new tools. To achieve this
goal, researchers and designers and future users collectively participate in iterative re-design processes
[
AI uses data to recognize specific patterns or properties through statistical analyses. The data is
labeled, and the performance of algorithms is calibrated against the correctness to the data, to make
constant improvements, i.e., AI learns and changes over time. There are a wide range of AI features
that can be designed, such as training data, algorithms, user interface and explanations of
decisionmaking. However, AI-based applications is difficult to understand with its advanced technology, but
also due to its black box properties making the relationship between input and output opaque and gives
an unpredictable behavior. This means it becomes difficult to control actions and foresee effects on the
social environment in the long term, which are valuable features in PD [
An overview of our suggested research method is provided and summaries the details of data collection and data resources (Table 1).
situated fairness.
what values they think should be prioritized.
Future Workshops with The purpose here is to allow a different groups: students- group of users, that are quite teachers-administration - unaware of the practical developers. Focus will be on consequences of AI technology future workshops where to think about and envision each group will be what such use might mean to introduced to AI, envision their everyday work. future practice, and discuss how fairness and inclusion in the development could be set up, as well as for which purposes AI are seen as beneficial in relation to their practice.
Design studios. The different groups participate in design studios where they create applications on paper/computer with desirable applications. AI and fairness are discussed based on these created systems.
Semi-structured interviews with students-teachersadministrators and school leader-developers. Selected parts from videos from the workshops will be used as triggers for discussion
engage in actual design work. And through involvement in design explore the possibilities, but also limitations of such systems. In particular we want to address solutions including metadesign and local adaptation.
involvements with the AI application we now finally want to explore the experiences that teachers made in the previous activities.
The broad purpose here is to identify what would be relevant dimensions for teachers to be able to locally adapt their use of AIEdsystems – i.e., in what ways should the systems cater for meta-design? This work is funded by Marianne and Marcus Wallenberg Foundation.
[20] Blanchard, E.G.: Socio-Cultural imbalances in AIED research: investigations, implications and opportunities. International Journal of Artificial Intelligence in Education 25(2), 204–228 (2015). [21] Schiff, D.: Out of the laboratory and into the classroom: the future of artificial intelligence in education. AI & society 36(1), 331–348 (2021). [22] Nye, B.D.: Intelligent tutoring systems by and for the developing world: A review of trends and approaches for educational technology in a global context. International Journal of Artificial Intelligence in Education 25(2), 177-203 (2014). [23] Howard, A., Borenstein, J.: The ugly truth about ourselves and our robot creations: the problem of bias and social inequity. Science and engineering ethics 24(5), 1521-1536 (2018). [24] Selwyn, N.: Education and technology. Key issues and debates. Bloomsbury Academic, London (2017). [25] Birhane, A.: Algorithmic injustice: a relational ethics approach. Patterns 2(2), 1–9 (2021). [26] Kitchin, R.: The data revolution: big data, open data, data infrastructures and their consequences.
Sage, London (2014). [27] Akgun, S., Greenhow, C.: Artificial intelligence in education: Addressing ethical challenges in K12 settings. AI and Ethics (2021). https://doi.org/10.1007/s43681-021-00096-7 [28] Smith, H.: Algorithmic bias: should students pay the price? AI & society 35(4), 1077–1078 (2020). [29] Engeström, Y.: Learning by expanding. An activity-theoretical approach to developmental research. Cambridge university press (2015). [30] Fischer, G., Fogli, D., Piccinno, A.: Revisiting and broadening the meta-design framework for enduser development. In: Paterno, F. and Wulf, V. (eds.) New perspectives in end-user development, pp. 61-97. Springer (2017). [31] Bratteteig, T., Wagner, I.: What is a participatory design result? In: Proceedings of the 14th Participatory Design Conference (2016). http://dx.doi.org/10.1145/2940299.2940316