=Paper=
{{Paper
|id=Vol-1601/CrossLAK16Paper18
|storemode=property
|title=Data Analysis of Workplace Learning with BOOST
|pdfUrl=https://ceur-ws.org/Vol-1601/CrossLAK16Paper18.pdf
|volume=Vol-1601
|authors=Miloš Kravčík,Kateryna Neulinger,Ralf Klamma
|dblpUrl=https://dblp.org/rec/conf/lak/KravcikNK16
}}
==Data Analysis of Workplace Learning with BOOST==
https://ceur-ws.org/Vol-1601/CrossLAK16Paper18.pdf
Data Analysis of Workplace Learning with BOOST
Miloš Kravčík, RWTH Aachen University, kravcik@dbis.rwth-aachen.de
Kateryna Neulinger, RWTH Aachen University, neulinger@dbis.rwth-aachen.de
Ralf Klamma, RWTH Aachen University, klamma@dbis.rwth-aachen.de
Abstract: Micro enterprises with less than 20 employees represent the majority of companies
in developed economies. To achieve a positive attitude from both sides (i.e. company and
employee) towards workplace learning in these settings, target competences of employees
need to relate to business needs and learning should be done at the workplace. In this paper,
we combine methodologies focusing on business critical needs that are linked seamlessly with
learning resources by means of a simple, ready-to-use Web-based learning environment. As a
result a cross-border and cross-cultural approach to increase participation of micro enterprises
in the workplace learning was created. We collected data about 79 users from 56 micro
enterprises in different countries with various settings. Our analysis investigates behavior of
employees and managers in this workplace-learning environment, considering our original
requirements.
Keywords: workplace learning, informal learning, personal learning environments, design
Introduction
Capable and professional employees are essential for the success of any enterprise. Therefore, employees have
to be trained and engaged into the lifelong-learning process at the workplace. However, small and micro
enterprises have a decreasing participation in vocational education and training. Thus, there is a real need to
engage them in developing a positive attitude towards training (European Commission, 2012). In our studies we
mainly focused on small enterprises (with less than 20 employees), as they represent a majority of companies in
developed economies.
Here we present a ready-to-use Web-based learning environment, which considers business critical
needs and provides tailored learning solutions for small companies. We assume that a positive attitude towards
workplace learning requires to relate target competences of employees to the business critical needs and to
integrate learning seamlessly into work processes. Our empirical evaluation studies include 79 users from 56
companies in five European countries. We track user interactions to better understand workplace learning and
the roles of manager and employee in it.
The rest of the paper is organized as follows. First, we summarize a research done in this domain.
Afterwards we describe our methodology and technical prototype. Then our analysis and its results are
presented. Finally, we conclude the paper.
Related Work
Cross (Cross, 2007) claims that formal learning receives a lot of attention and funding. On the other side,
informal learning remains often unrecognized even by the individuals that undertake it and is left to chance. He
also argues that formal learning has almost no impact on job performance, while informal learning is the major
source of knowledge transfer and innovation.
Skule (Skule, 2004) developed a framework for measuring, assessing and comparing learning
environments in the workplace, as informal learning could not be measured by means of indicators traditionally
used in the field of education and training. The framework consists of seven learning conditions, which have
significant effects on informal learning at work: a high degree of exposure to changes, a high degree of exposure
to demands, managerial responsibilities, extensive professional contacts, superior feedback, management
support for learning, and rewarding of proficiency.
Berg & Chyung (Berg & Chyung, 2008) investigated factors influencing informal learning at the
workplace, as learning and performance improvement practitioners gain new knowledge from informal learning
activities more frequently than they do from formal training. The rank-order of factors affecting informal
learning engagement can be utilized as a tool in prioritizing potential interventions to encourage informal
learning: interest in current field, computer access, personality, professional capability, relationship with
colleagues, job satisfaction, and job itself.
Eraut & Hirsh (Eraut & Hirsh, 2010) identified balancing ‘top down’ and ‘bottom up’ learning
priorities as one of key challenges. They provide various recommendations, including the following ones: Set
98
Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes.
aside some training resources to respond to individual as well as organizational needs; Try to involve the
learners and their managers in the design as well as asking for their feedback afterwards; Train managers to act
as coaches to employees; Encourage a work climate, in which sharing knowledge and helping others is seen as
positive.
Based on these outcomes, we can conclude that it is highly important to address the issues of informal
workplace learning. The demands both of companies and employees need to be considered. Managers are
responsible for specification of the business critical needs and for identification of the competence gaps in their
companies. They also have to assign learning goals and to monitor the learning progress of their employees, as
well as to provide them with appropriate feedback. Here the employee’s capability, personality, and satisfaction
must be taken into account.
BOOST Approach
We aimed to address the above mentioned and additional demands in the BOOST project (Business
perfOrmance imprOvement through individual employee Skills Training), where a sound methodology from the
BeCome project13 was integrated with widget-based technology from the ROLE project14. As a result, a ready-
to-use Web-based learning environment to support and monitor learning process at the workplace has been
created.
Our aim was to support these phases of workplace learning: planning, tutoring, learning, and reflection.
Planning includes identification of critical business goals in the company (together with related competences
and learning resources) and selection of suitable employees to address them. In Tutoring phase relevant learning
resources are assigned to target competences. In Learning phase access to relevant learning resources is
provided, together with searching facilities, in order to look for additional ones. Reflection means monitoring of
the learning progress of the company, as well as of individual employees.
The basic data model is simple and includes key Business Goals (like project administration or Web
development) at the top level. Each of them can refer to relevant Learning Indicators (competences, like
reporting or HTML5). For these suitable Learning Resources (materials, tools, and peers) can be recommended.
In BOOST, we distinguish two user roles: Manager and Employee. Manager can specify business
goals with learning indicators (e.g. relevant competences), and assign them to employees. This role can also
monitor and assess the learning progress of employees. Employee gets an overview of the assigned business
goals, learning indicators, and recommended learning resources. Then they can learn by accessing the learning
resources as well as search for new ones and assign them to learning indicators. Employee can also reflect on his
or her own learning progress.
BOOST Platform
The BOOST technical prototype has been already described in (Kravcik et al, 2014a; Kravcik et al, 2014b).
BOOST learning platform follows a widget-based web application development approach [8] and it was
developed using ROLE Software Development Kit 15 (SDK). A widget is a relatively simple software component
created for a particular task. By combining widgets we create specific predefined learning areas (Start,
Management, and Learning) that support different phases in workplace learning. Users can easily adjust the
design of individual areas to their needs by selecting the relevant widgets and arranging them appropriately. The
BOOST platform enables inter-widget communication and supports real-time chats too. The BOOST software is
open source and fully available for further development and extensions. It can be hosted by vocational education
and training (VET) providers as demonstrated in the project. Its translations into five European languages are
currently available.
In Start area one can choose the preferred language, read a brief introduction about the platform, and
Manager can also assign roles (Manager or Employee) to users. Here also the license agreement has to be
confirmed at the very beginning.
Management area (Figure 1) includes three components: Goals widget to specify relevant business
goals and assign then learning indicators (competences), Personnel widget to assign learning goals (learning
indicators with target proficiency levels and deadlines) to individual employees, and Progress widget to monitor
learning progress and recognize competence gaps. Due to privacy issues, Manager and Employee have different
access rights here. Manager can see all Employees and can edit specification of business goals and learning
indicators, as well as their assignment to employees. Moreover, Manager can also assess proficiency levels of
13
http://become.dedi.velay.greta.fr
14
http://www.role-project.eu
15
https://github.com/rwth-acis/ROLE-SDK
99
employees. On the other side, Employee can see only his or her own data, which was one of the key
requirements from our project partners.
Figure 1. Management area – Manager view
Learning area (Figure 2) consists of the following modules: Resources widget shows learning resources
assigned to learning indicators (and business goals), Viewer widget displays the selected learning resource for
learning, and Search widget allows searching for additional learning materials in several repositories:
YouTube16, Scribd17, Wikipedia18, and SlideShare19. Newly found documents can be displayed in Viewer and
added easily to Resources. Both Manager and Employee can add new learning resources, either as public or
private ones.
Figure 2. Learning area – Employee view
Data Analysis
BOOST partners have conducted evaluation studies during period of three months (13 th May - 19th August,
2015) with 79 users from 56 different companies in five countries: Great Britain, Germany, France, Czech
Republic, and Greece. Typically, each company has used its own installation of the BOOST platform, in order
to respect the privacy issues. BOOST partners helped the companies by introducing them to the platform,
assisting them in specification of their own data, and running their pilots.
Compared to community supported dashboards [9, 10], we focused primarily on individual users. Our
aim was to trace user actions and then to visualize it for each individual from various perspectives, in order to
understand his or her behavior. We wanted to know when the user was active, what kind of activities they
performed, and in which order. To collect user specific interactions a JavaScript based tracking system was
implemented. We recorded each traceable action (click) of the user, together with all relevant data related to it.
The stored data included the following information on each (traceable) user action: user identifier, installation
URL, widget, user role, element (button) name, user action, and time. Due to privacy issues, we did not store
user identities, just random user identifiers for further analysis.
To visualize the collected data sets, we have created a Web-based dashboard. Each user interaction is
assigned a specific color based on the corresponding widget and the interactions are visualized chronologically
over various time scales. We expect that in this way we can interpret the user behavior and compare it with the
16
https://www.youtube.com
17
https://www.scribd.com
18
https://en.wikipedia.org
19
http://www.slideshare.net
100
workflow we intended to support originally, considering the predefined phases of workplace learning. Not all
users have used the platform intensively over a longer time. Therefore, for further analysis we have mainly
focused on the most active users, including both managers and employees.
Two types of visualization can be considered. Long-term views illustrate user activities over several
days, weeks, or months, providing various perspectives on them – a chronological view, an assignment of
activities to different areas or phases, as well as an assignment of activities to individual widgets. Short-term
views show daily activities assigned to concrete widgets, usually from one session. It is worth mentioning that
some widgets are more interaction demanding than the others. For instance, interactions in Viewer are not
traceable at all, as the viewed learning material is selected in Resources or Search widget, and the embedded
player works independently of our platform.
Manager role
To analyze the collected data, we aimed to identify and visualize the following phases in Manager’s actions:
1. Start: in Start area
a. Welcome actions in Welcome widget
b. Role setting actions in Role widget (Access)
2. Planning & Reflection: in Management area
a. Goal setting in Goals widget (BCN)
b. Checking progress in Progress widget (Graph)
c. Assigning goals & checking personnel in Personnel widget (Employee)
3. Tutoring: in Learning area
a. Checking and setting learning resources in Resources widget
b. Viewing learning resources in Viewer widget
c. Searching learning resources in Search widget
We have counted 3957 interactions of managers (Figure 3). Just few of them (290) came from the Start
area, representing language changes in Welcome widget (17) and right changes in Role widget (273). About one
third of manager interactions (1293) happened in the Management area performing planning and reflection,
namely specification of goals and learning indicators in Goals widget (477), showing learning progress in
Progress widget (251), as well as assigning goals with priorities to employees and checking their progress in
Personnel widget (565). Nevertheless, managers mostly interacted (2374) in the Learning area doing tutoring,
especially assigning learning resources in Resources widget (1525) and searching for new learning resources in
Search widget (849).
Figure 3. Manager interactions with widgets
There are several alternatives of a long-term view. Figure 4 shows the distribution of Manager’s
activities over several months and times of the day. We can see that most of the work has been done in June and
afterwards the user came back just sporadically, at the end just checking the progress of employees and
reflecting (Graph).
101
Figure 4. Manager’s long-term view – user activities
Figure 5 shows more precisely the distribution of activities (clicks) per day, indicating that actually in
the first half of June the user was most active, busy both with Planning (Management area) and Tutoring
(Learning area).
Figure 5. Manager’s long-term view – usage of areas
A more detailed overview of the interaction with individual widgets provides Figure 6. We see that
Manager started with searching for learning resources (Search) and assigning them to individual learning
indicators (Resources). Moreover, the user also specified the goals (BCN) and later on assigned them to
employees (Employee).
Figure 6. Manager’s long-term view – usage of widgets
Short-term views display in a chronological order what happened in one session. Interesting are
especially the longer sessions, which illustrate how the two main phases of Planning and Reflection on one side,
and Tutoring on the other, complemented each other. Figure 7 (visualizing 87 interaction activities) shows that
Manager started with checking learning resources, searching for new ones, and assigning them to the existing
learning indicators (Tutoring in rows 1-2). Then the business goals were checked and newly created, as well as
assigned to employees, monitoring their progress occasionally (Planning & Reflection rows 3-5) and checking
the assigned learning resources (Tutoring in row 1).
102
Figure 7. Manager’s short-term view
Our analysis of 8 most active Managers shows that they spent 86% of their interactions in Tutoring
phase and they mostly used Resources widget. This means that they put a lot of effort to preparation of suitable
learning resources for Employees. In addition, 11% of Managers' interactions belonged to Planning &
Reflection phase, where they mainly specified business goals and assigned them to Employees, which
apparently consumed much less effort than searching and assignment of learning resources. It is not surprising,
as to each business goals, several learning indicators are assigned and to each learning indicator several learning
resources belong. Finally, 3% of their interactions happened in Preparation phase.
Employee role
From another perspective, we intended to identify and visualize activities performed by Employee in these
phases of a cycle:
1. Start: in Start area
a. Welcome actions in Welcome widget
b. Role actions in Role widget (Access)
2. Learning: in Learning area
a. Accessing learning resources in Resources widget
b. Viewing learning resources in Viewer widget
c. Searching learning resources in Search widget
3. Reflection: in Management area
a. Checking goals in Goals widget (BCN)
b. Checking progress in Progress widget (Graph)
c. Checking personnel in Personnel widget (Employee)
Surprisingly, there were only 1118 employee interactions (Figure 8), which is much fewer than the manager
ones. Less than a quarter of them (235) happened in the Start area, mainly changing the language in Welcome
widget (230), while in Role widget employees could only show help (5). As expected, employees were most
active in the Learning area (461), using learning resources (149) and intensively searching for additional ones
(312). Reflection phase in the Management area was also properly used (422). In Goals widget employees could
check their own assigned goals and deadlines (89). Then they could compare it with their learning progress in
Goals widget (160), which can be initiated also from Personnel widget (173).
Figure 8. Employee interactions with widgets
Figure 9 illustrates activities of one employee over two weeks. The first week was quite intensive
(except of the weekend in the middle), especially on two days. The second week was rather relaxed.
103
Figure 9. Employee’s long-term view – user activities
Figure 10 shows that the employee spent most of the time in Learning phase (Learning area), but
devoted enough time also to Reflection (Management area).
Figure 10. Employee’s long-term view – usage of areas
In Figure 11 (showing 252 interaction activities) we see how one session of an employee looked like.
The first row belongs to Start phase, while the next two ones represent Learning phase and the last three rows
show Reflection phase. The user starts with an introduction (Access – Start in row 1) and then (Learning in rows
2-3) checks available learning materials (Resources) as well as looks for new ones (Search). This is gradually
complemented (Reflection in rows 4-6) by checking the current progress (Employee, Graph) and the assigned
goals (BCN).
Figure 11. Employee’s short-term view
We have found out that 3 most active Employees did 51% of their interactions in Learning phase, 45%
in Reflection phase, and 4% in Start phase. The most used widgets were Search, Employees and Resources. It is
104
not surprising to see that this role put most effort into learning activities (considering also the invisible
interaction in Viewer), but the fact that neither reflection was neglected, is quite positive.
Conclusion
In this paper, we have presented a ready-to-use Web-based learning environment, which is based on an
innovative methodology for workplace learning. In order to investigate it in small companies, we have evaluated
platform with more than 50 enterprises in five European countries. It is necessary to take into account that these
companies usually cannot afford long-term planning and training, as they are typically busy with operative
tasks. Therefore, it is pretty challenging to get them engaged in evaluation, which requires certain time
investment from their side.
The involved companies belonged to various domains, not only technical ones, and in some cases
encountered usability issues. Nevertheless, the evaluation outcomes suggest that the BOOST approach can be a
viable one and has a potential for a further improvement, especially in the user interface area. Our data analysis
shows that it was often not easy for the companies and their employees to find enough time for a comprehensive
evaluation. Nevertheless, some of them worked in certain periods intensively and followed the envisioned
workflow, dividing their activities among planning, tutoring, learning, and reflection accordingly.
An important finding for us was that openness and sharing, quite typical in the academic environment,
are not so common in the corporate settings. Therefore, means for different levels of privacy and data security
have to be provided, to allow for various levels of integration. Configuration options should enable
organizations to tailor the infrastructure to their specific requirements. The BOOST approach is not necessarily
restricted on small companies, but can be useful also for larger ones.
References
European Commission. (2012). Rethinking Education: Investing in skills for better socio-economic outcomes.
COM, 669.
Cross, J. (2007). Informal learning: Rediscovering the natural pathways that inspire innovation and
performance. John Wiley & Sons.
Skule, S. (2004). Learning conditions at work: a framework to understand and assess informal learning in the
workplace. International journal of training and development, 8(1), 8-20.
Berg, S. A. & Chyung, S. Y. Y. (2008). Factors that influence informal learning in the workplace. Journal of
workplace learning, 20(4), 229-244.
Eraut, M. & Hirsh, W. (2010). The significance of workplace learning for individuals, groups and organisations.
Oxford: SKOPE.
Kravčík, M., Neulinger, K., & Klamma, R. (2014). Supporting workplace learning in small enterprises by
personal learning environments. In Posters, Demos, Late-breaking Results and Workshop Proceedings of
the 22nd Conference on User Modeling, Adaptation, and Personalization (Aalborg, Denmark, July 7-11,
2014). CEUR vol. 1181.
Kravčík, M., Neulinger, K., & Klamma, R. (2014). Boosting Informal Workplace Learning in Small Enterprises.
In Proceedings of the 4th Workshop on Awareness and Reflection in Technology Enhanced Learning
(ARTEL). In conjunction with the 9th European Conference on Technology Enhanced Learning (EC-TEL):
Open Learning and Teaching in Educational Communities (Graz, Austria, September 16, 2014). CEUR vol.
1238, 73-75.
Nussbaumer, A., Kravcik, M., Renzel, D., Klamma, R., Berthold, M., & Albert, D. (2014). A Framework for
Facilitating Self-Regulation in Responsive Open Learning Environments. arXiv preprint arXiv:1407.5891.
Derntl, M., Erdtmann, S., & Klamma, R. (2012). An embeddable dashboard for widget-based visual analytics on
scientific communities. In Proceedings of the 12th International Conference on Knowledge Management
and Knowledge Technologies (i-KNOW '12). ACM, New York, NY, USA, Article 23 , 8 pages.
DOI=http://dx.doi.org/10.1145/2362456.2362486
Hannemann, A., Liiva, K., & Klamma, R. (2014). Navigation Support in Evolving Open-Source Communities
by a Web-Based Dashboard. In Open Source Software: Mobile Open Source Technologies. Springer
Berlin Heidelberg, 11-20.
Acknowledgments
The presented research work was partially funded by the German National Agency BiBB within the Lifelong
Learning Programme Leaonardo da Vinci: "Business PerfOrmance imprOvement through individual employee
Skills Training" (project no: DE/13/LLP-LdV/TOI/147655) and by the 7th Framework Programme large-scale
integrated project "Learning Layers" (grant no: 318209).
105