Over a decade of work in the domain of learning analytics (LA) is available to inspire higher education institutions. While several have launched smaller experiments, examples of LA implemented at scale remain scarce. In recent years, several guidelines and frameworks have been published to support institutions in the development of their LA policies. Likewise, multiple implementation project methodologies are available. However, little guidance is provided on how policy making and implementation for LA projects at scale can be matched. In this work, we start from a recently published approach to coordinate a concrete implementation timeline with policy making efforts supported by the SHEILA framework. We extend this method originally aimed at LA experts to larger audiences of educational and institutional professionals with less LA experience. This is accommodated by a reusable workshop format using a set of cards and worksheets. In this paper we share the methodology and particular approach of the workshop and describe our findings from two runs of the workshop: one in the context of LA scale-up project and one focusing on a broader scope of innovative TEL projects. Our main observation is that the workshop format fosters constructive discussion and improves awareness and buy-in, while also delivering valuable input for the project team. With this paper, we share a pragmatic approach and instrument to engage stakeholders in the realization of projects aiming at realizing educational technology innovation at institutional scale.
Learning Analytics was defined by Duval & Verbert (2012) [
In this paper, we present a workshop format called WETS (Workshop For Obtaining
Educational Technology at Scale) that can serve as a particular instrument to support
the realization of Learning Analytics at scale by the engagement of stakeholders.
Hereby, it primarily addresses the first barrier, stakeholder engagement and buy-in, that
Tsai et al. identified for Learning Analytics at scale [
Active engagement and consulting of stakeholders is considered key to ensure
acceptance of institution-wide technologic innovations. Based on the coordination model
for Learning Analytics policy making and implementation at scale, proposed by Broos
et al. [
In WETS the stakeholders jointly work towards a common goal: a timeline of
priorities that must be handled to realize the institution-wide implementation of a particular
Learning Analytics innovation. To this end WETS builds on the coordination model
[
The coordination model of Broos et al. [
Prototyping phase: When prototyping, one or more artefacts are being
produced, not with the intention of finality, but as an instrument to support design
activities, discussion and improvement through iteration. […]
Piloting phase: This phase aims at testing the solution design in a natural
setting. It involves the use of real data that will be accessed by real users in a
context that is representative for the intended end goal of the solution. It differs
from the subsequent scaling phase in that only a subset of the intended user
population is targeted. [..]
Scaling phase: This last phase starts from what was learned from the previous
phases, especially from the piloting phase, to re-implement, or at least
re-deploy, the envisioned solution at scale. Here the full population is targeted: all
intended courses, programs, and faculties. Several challenges related to the
scalability of the solution will need to be tackled. This includes, but is not
limited to, technical issues, e.g., the requirement for system resilience, and
maintenance. […]” [
The SHEILA framework is a process model [
WETS challenges the stakeholders to jointly structure priorities, consisting of a subset of the 181 SHEILA items, along a timeline, consisting of the four implementation phases. The format of the workshop consists of structured discussions around tangible representations of both the four implementation phases and the SHEILA items. Tangible objects or artefacts are believed to facilitate and enhance the discussion in the workshop. The physical representation of the four implementation phases are four large A3 papers, containing the title of the phase and a short description. The physical representation of the 181 SHEILA items are a physical set of cards: each card represents one SHEILA item (Fig. 2). Color coding is used to represent the six SHEILA dimensions and icons on the background of each card represent if the item is an action point, challenge, or policy question.
The final product of the workshop is one timeline with for each of the four implementation phases the most important SHEILA items to be handled. All workshop activities, elaborated in the next section, work towards that final goal. The basic workshop structure, assuming a duration of three hours, consists of four parts: 1. plenary introduction (±25 minutes), where the goal of the workshop is introduced, the particular educational technology project is proposed, and the workshop formats are clarified; 2. group work in jigsaw format (±1 hour 50 minutes), where the participants work in smaller groups to structure a subset of the cards along the timeline; 3. plenary discussion (±40 minutes, Fig. 6), where the results of the different groups are collected on one joint timeline and discussed; and 4. closing (±5 minutes).
The group work and plenary discussion are detailed below. The format is based on a basic number X, accommodating 2X2 participants, which are engaged om X jigsaw rounds. Examples and figures are given with X=3. As elaborated in Section 2.4, changing the basic number X is one of the ways to up-or downscale the WETS format flexibly, depending on the number of participants or the available workshop space. The jigsaw format consists of X jigsaw rounds, structured around sets of X tables (in the example: A, B, and C) (Fig. 3). Each set of X tables will iterate during X subsequent jigsaw rounds to jointly structure in each round 1/(2Xth) of the SHEILA items. Therefore, if the goal is to handle all SHEILA items at least two such sets of X tables should be organized. For each particular workshop, depending on the experience and job description of the participants and their involvement in the educational technology project, the different roles, and if needed, experience levels are defined. As the goal of the workshop is to trigger discussion between stakeholder groups, the groups are formed to be as heterogeneous as possible, not only in the first round but also in the subsequent rounds.
When arriving, all participants receive a pre-printed card with their name and a set of three letters, representing the sequence of tables they should go to in each of the X jigsaw rounds. The individual table sequences are defined to ensure that individuals are regrouped in each round. One experienced participant -we advise to use people familiar with the project and the workshop format- will stay at one table during all X jigsaw rounds and is responsible for the coordination. Moreover, we advise to start with a diverse set of roles at each table.
In the first, most elaborate round (50 minutes) each table gets the instruction to place the cards assigned to their table to one of the four phases according to during which phase most attention should be paid to it. If they strongly believe the card belongs to multiple phases or if they have additional remarks regarding this item, they can add an amendment (post-it) to the item. In the final five minutes of the first round, all participants are requested to individually add one (red) priority sticker to the five items at their table they believe to be most important. Fig. 4. presents the result of one the first round at one table. The placement of the cards on the timeline does not change in the next rounds. For the second round (30 minutes) participants move to their next table, indicated by the second letter on their name cards. The moderator who stayed at the table introduces the now fixed timeline obtained in the previous round and initiates discussion around this timeline and its items. Amendments (post-its) can be added to a card if the participants do not agree with the card’s placement or if they have other remarks concerning the item. The second round again concludes with an individual 5-minute prioritization. The next X-2 rounds are a repetition of the second round, every time at new tables. At the end of these X rounds all participants, except the moderators at each table, have attended a discussion at each of the X tables in one series.
The plenary discussion, which starts from the timeline at the tables, is organized in three steps. First, each of the participants returns to their initial table and the moderator is asked to introduce the final result, focusing on the cards that received the highest number of priority-votes. Next, one participant per table brings the 10 highest priority items to the plenary timeline (Fig. 5). The workshop moderator initiates a discussion around the results and asks the participants to clarify particular choices and to comment (Fig. 6). Finally, each of the participants is requested to add names of experts or stakeholders that could or should be consulted for particular prioritised items. The proposed workshop format can be up- or downscaled to better accommodate the actual number of participants and the available workshop space: - The number of participants (X) at one table can be adapted. We advise to aim for between 3-6 participants. Together with the number of participants at one table, the number of tables and the number of jigsaw rounds will change accordingly. One could e.g. accommodate five people at each table, divide the SHEILA items over five tables, and organize five jigsaw rounds. - When accommodating many participants in one workshop one could also organize N parallel series, hereby allowing for N*X2 participants. 3
A webpage [
Two WETS were organized, of which the context, the course of the workshop, and the results are discussed below. A planned third round of WETS involving over 80 participants was postponed to a later date due to the COVID-19 pandemic. 3.1
KU Leuven, a general university in Flanders, Belgium is currently up-scaling three
Learning Analytics Dashboards that were developed in two European projects: (i)
LISSA, a dashboard supporting the dialogue between student advisors and students [
To prioritize the activities of the KU Leuven scale-up project, they organized a workshop with the project’s multi-disciplinary core team, including policy makers (delegate of vice-rector of education, head of study advice service of KU Leuven), IT staff, data managers, student advisors (practitioners), researchers, and a student. This workshop was also the first WETS workshop organized and was therefore used to gather experience in order to improve the workshop format. 11 people attended the workshop. The workshop was organized with a setup of X=3 and half of the SHEILA items.
The 11 participants succeeded in identifying 23 high-priority SHEILA items and placing them on the implementation timeline (Table 1). They placed 10 items in the initialization phase, 4 each in the prototyping and piloting phase, and five in the scaling phase. All SHEILA dimensions were present, in order of prevalence: develop engagement strategy (8 items), analyse internal capacity to effect change (6 items), Identify key stakeholders (5 items), Establish monitoring and learning frameworks (2 items), Identify desired behaviour changes (1 items), and map political context (1 item).
C A P A
Identify key stakeholders Map political context Develop engagement strategy Identify key stakeholders
Item Define ownership and responsibilities among diverse professional groups within the university.
Identify internal and external drivers for learning analytics (e.g., problems to solve or areas to enhance).
Will learning analytics be used as a management tool to monitor students or staff? Identify primary users of learning analytics (e.g., students, teaching staff, and senior managers).
C P P P A P A A Develop engagement strategy Establish monitoring and learning frameworks Identify desired behaviour changes Identify key stakeholders
Analyse internal capacity to P effect change
Analyse internal capacity to C
effect change Prototyping Phase
Develop engagement stratP
egy A
Identify key stakeholders
Develop engagement stratP egy
Develop engagement stratA
egy Piloting Phase
Develop engagement stratC egy
Analyse internal capacity to A
effect change P Identify key stakeholders
Establish monitoring and P
learning frameworks Scaling Phase
Develop engagement stratC egy Develop engagement strategy Analyse internal capacity to effect change Analyse internal capacity to effect change Analyse internal capacity to effect change
Feedback is provided without proper support, which leaves students in anxiety or complacency, thereby demotivating them.
What defines success or failure? How will success be measured? What are success indicators? What positive changes will learning analytics bring to the current situation (e.g., learning and teaching landscapes)? Will learning analytics exclude certain groups of students? Will there be mechanisms to address inequality? Will the design of selected learning analytics tools address teaching and learning needs? Institution-wide buy-in is hard to reach.
How will students’ responsibility for learning be highlighted and considered in the design and implementation of learning analytics? Identify required expertise (e.g., learning analytics expertise, IT expertise, statistical expertise, educational expertise, psychological expertise) How will the results of analytics be communicated in a way that motivates learning? Decide forms of interventions (e.g., automatic systems, personal contacts, learning resources).
Peer comparison may demotivate students.
Evaluate technological infrastructure.
Whose data will be collected? Stakeholder engagement Are there any measures to ensure that students are equipped with sufficient knowledge to make opt-in/out decisions? Learning analytics is used as a metric to judge students and teachers rather than evidence to support learning and teaching.
Raise awareness and understanding of learning analytics among teaching staff and students through publicity and meetings/ workshops/ conferences.
What communication channels or feedback mechanisms will be in place? How will the implementation address the problem of time poor among teaching staff? Evaluate institutional culture (e.g., trust in data and openness to changes and innovation).
Establish indicators of data quality and system efficacy
The participants believed that the workshop format was well-suited to realize the goals of the workshop. They suggested that the moderator at each table should be someone who is well-informed about the project and is well-briefed about the workshop format and goals. This moderator should focus on moderating the discussion, managing time, clarifying in case of doubt, and repeating the practical arrangement to participants. The workshop participants suggested to translate the SHEILA items to the native language of the participants (Dutch) for subsequent formats and asked to simplify overly complex language whenever possible. To ease the processing of the results it was decided to add the item number to the cards. 3.2
After the successful first run of WETS we were invited to organize a second run for
project teams working on a KU Leuven SEED-project. SEED projects are part of the
KU Leuven approach to stimulate innovation around educational technology [
The workshop was adapted to a shorter 2-hour format by removing the final jigsaw round. Based on the feedback of the previous workshop all SHEILA items were translated to Dutch and wherever possible the language was simplified. 15 project collaborators participated in the workshop. It was clarified that although some of the items referred to Learning Analytics, they should interpret this broadly and think about educational technology for feedback.
The participants found the workshop format useful to get a picture of the many things educational technology projects should consider. They recognized many of the items from their experiences in the projects, including the hard experiences of realizing that something was omitted or handled too late into the project. Therefore, the attendees stated that such a workshop should be organized earlier on in the project phase and that WETS should be part of the support package developed by the universities’ support teams. They also believed that WETS could be useful to engage and consult their stakeholders. On the negative side, they indicated that the items were too strongly focused on Learning Analytics, which is not surprising as the SHEILA items were designed with this domain in mind. Therefore, in the future the workshop could be organized with items adapted to more educational technologies. 4
In this paper we introduced a reusable hands-on workshop approach, WETS, to
support higher education institutions in the realization of innovative educational
technology projects at scale. The WETS approach builds on the recently proposed coordination
model [
The output of the workshops also acts as additional supporting evidence for the
findings of Broos et al. [