One challenge faced by workplaces is having enough relevant data to support datadriven decision making related to the further education/training of their employees, to identify competence gaps within their work force, and to contribute to organisational learning. We are propose that data driven decision-making can be supported by the combination of competence mapping, collection of assessment and performance data from various sources, learning analytics of the data, and visualisation of the results in an open learner model (OLM). In the iComPAss project we work with two learning scenarios, health workers studying for a Masters in health leadership, and the on-the-job training of firefighters to study this challenge. The project builds on and extends ideas from the former EU-project Adapt-It, and on a learning analytics approach and an OLM developed in the EU NEXT-TELL project. This paper elaborates on our approach.
The iComPAss project arises from the needs of two professional education/training situations that want to raise the quality of their education and training, and improve decision-making related to their students/employees and the organisations. Scenario 1 is a Master programme for Organization and Management (Health and Welfare line) at a University College. Scenario 2 is the education and training of firefighters, organized by a Fire and Rescue Service (FRS). Both scenarios are situated in Norway.
The Master programme (Scenario 1) is a combination of formal learning and on the job competence development. The professions for which they educate their students have a demand for highly qualified professionals. During the part-time study the students attend lectures, write a master thesis and use their professional background (i.e., from where they are currently working) in order to complete their education. Then the students have competence to assume 1) a leadership position within their field of specialization, 2) administration positions within the healthcare or educational sectors, or 3) independent positions with leadership responsibilities.
During the 4-years study programme, most students combine the part-time study with family
obligations and practical health care work. This combination results in many challenges, resulting in a large
need for individual adjustment of the study programme and logistic challenges. Furthermore, in the latest
NOKUT Study Barometer
In Norway, potential firefighters are recruited from vocational schools with a focus on careers such as a carpenter or mason (Scenario 2). The Fire and Rescue Service (FRS) is then responsible for the education and training of these potential firefighters, and is further required to ensure that the firefighters maintain extreme skills and meet intensive fitness standards. The FRS is emergency response professionals and specialists in an enormously wide range of tasks, from the rescue and protection of people and property during fires, smoke, and accidents, to giving courses and informing on fire safety. For example, firefighters have to use a variety of techniques to control or suppress fires, but they may also perform initial medical treatment related to fire and accidents. The FRS has a high demand for employees with competence in several fields of rescue and fire prevention, and needs to have an overview of these competences, at an individual employee level, at a team level, and at organizational level. Another challenge within the FRS is how to combine on the job performance of skills and performance during formal training, as evidence of competence development.
Although the two cases at a first glance seem very different, they have many similarities. Both units where the cases are anchored have a need to address quality issues, to focus on student/ employee learning and work, and ensure that competence goals within their education and training are tightly tied to their professional practices. To be more specific, our first workshop with the FRS revealed that there is a need for overview of the organisation’s competence exists on three levels: 1. Organisation: The organisation’s competence needs related to current manpower. The leadership needs an overview of the entire expertise situation of their manpower. For example, does their aggregated competence profile match the National rules and regulations for a fire corps? Another example is that the leader needs to plan for competence development within the corps a few years before a veteran firefighter retires (e.g., Is there a firefighter who with a little training will acquire new competence to cover that of a retiring firefighter?). 2. Team: Competence within a team. Do the team members (each firefighter belongs to a specific team) have the competence and updated certificates required for the team as a collective (for example, there if there is a requirement for 2 smoke divers does the team have this competence?). Does the team have the competence required when a specific emergency arises (e.g., a heathland fire)? 3. Individual: Each firefigher’s competence. Leadership and the individual firefighter need an overview of their competence profile and certifications. For example, the individual firefighter could use their own profile for self-reflection over where they stand on individual competencies, and also how they fit into the overall profile of the corps. The leadership could use the individual competence profiles to determine who needs training to meet certification standards, whom might be assigned to a new team, and who to develop further with thought of replacing a firefighter who has either resigned or gone off with pension.
By combining the competence profiles of the individuals, the leadership can see the needs for each team and for the organisation as a whole. In FRS at the current time this is done manually. It is also a challenge for FRS to maintain a competence profile for each individual firefighter, as their competencies must be continually updated, both with theory, training and on-the-job practice, and certifications run out at certain dates, etc. Data from multiple sources such as emergency log files, experience reports, national and local authorities requirements and certification, etc. needs to be combined to determine competence for an individual, for teams (did the team perform well at the latest emergency situation, did one firefighter under preform (e.g., if a firefighter shows fear this needs to be recorded)), and for the organisation as a whole.
While these needs are not novel to FRS, our approach is unique in professional competence development (as far
as we have ascertained) in that it investigates how to support data-driven decision-making by individuals,
instructors, and leadership through a combination of competence mapping, data from multiple assessment and
performance sources, learning analytics, and visualization of the learning data in an Open Learner Model
(OLM), and investigates how this supports learning and managerial decision-making.
iComPAss builds on our previous work (i.e., in the Adapt-it project) using the instructional design for
professional learning method 4C/ID
To data there are no general instruments connected to 4C/ID, except for two loosely coupled portfolios called Perflect (Becker, et at, 2014), and the e-pass (EPASS, 2016), and these do not fit our purposes. Thus, existing and new assessment methods will be supported, electronic if possible, and linked to the relevant competences. For example, a tool to support team leaders in reporting the performance of their team members during an incident (e.g., a house fire) will generate performance data related to particular competences, will be developed. Similarly, such a tool can be used to report on healthcare workers performance on leadership skills.
As each evidence related to a single competence can come from several sources, we choose to build on
experience from the Next-Tell project in using extending the use of learning analytics and educational data
mining with features from the psycho-pedagogical framework Competence-based Knowledge Space Theory,
known as CbKST
The learning analytics output will feed a learner model and be visualised in an Open Learner Model
(OLM)
An OLM traditionally represents a learner’s competences and abilities while the learner interacts with
learning material in an intelligent tutoring system (ITS), however, state-of-the-art research has extended the use
of OLMs for multiple data sources feeding its learner model
In summary, in our approach there is a tight coupling between research on the pedagogical design of competence assessment and visualisation for learning, the technical development of these assessments and visualisations, extraction of data from multiple sources, learning analytics, and an evaluation of their impact on student learning and managerial decision-making.
As pointed out earlier, the two Scenarios, while different, share a number of similarities. While Scenario 1 is based on formal learning organised during the sessions and unorganised between the sessions, Scenario 2 is a mix of theoretical and practical learning at the workplace. Both, however, need support in gaining an overview of the competence development of their learners/employees.
In our approach there are several challenges we have to address. First, one challenge lies in the identification of skill/competence hierarchies for the two domains. Second is the identification of relevant theoretical and performance data (i.e., assessment) and how to collect it from multiple sources, and third how the interaction between these data from multi data sources provides evidence of competence. .A fourth challenge is how to use learning analytics to analyse and interpret this data for the learner model, and fifth, how to best visualise the complex patterns and relations in an OLM to support decision-making by professionals with high competence demands.
Though there are many challenges, conceptually and technically, both scenarios will have the potential to have valuable impact and improve on decision making concerning: • Training needs
Ability to assess and identify competence gaps (individuals, teams, workforce)
Identification of necessary training and competence development before the need becomes problematic We will be developing a number of tools to support this research and in order to determine the impact of our approach we ask questions such as: Do the tools provide a better overview of the competence and training needs than current practice? What new insights can be gleaned? Do our OLM visualisations support individual or leadership decision making? Can the visualisations provide an overview of both the individual and collective expertise?
The ability to develop practice through the inquiry of learner data should have impact and be of interest for a multitude of institutions organisations, as well as for teachers, health and school leaders and policy makers. Understanding the relationship between capturing data, visualising competences and professional development can be of major impact for these institutions.
The work is supported by the Norwegian Research Council grant number 246765/H20. The authors would thank the researchers we have been collaborating with in the previous projects upon which this project builds. The authors would also like to thank the reviewers for valuable feedback that resulted in a much improved paper.