The company need to align their future success and growth with the personal growth of all its employees. One of the most critical objectives is establishing yearly goals for their employees related to their career development and advancement. Specifically, companies expect their employees to enhance their skills and competencies over a business year to be eligible for a new role or promotion [1].
Companies must monitor employees' progress in three vital areas throughout the year to ensure business success. These areas include quality work and efficiency, where employees contribute to the company's profits through efficient project execution; career development, where employees improve skills related to their engineering disciplines; and employee satisfaction, where employees receive an acknowledgement from clients, team members, and management for their contributions to growth and development [2].
Each aspect of employee growth and advancement is the focus of exceptionally specialized professionals in the company. Project managers and delivery managers monitor the effectiveness of the employee on the project and can identify gaps in knowledge or experience that are the basis for defining training programs [3]. People management monitors the employee's satisfaction and develops the soft skills he needs to effectively understand the complex social and psychological relationships in the team, company, and client. The result of HR's work with employees is the construction of learning frameworks focused on understanding the framework of satisfaction and are also a valid input for the lifelong education program to improve soft skills [4]. A career coach monitors the employee's career progression and guides him by setting goals and objectives according to the target career progression.
As a starting point, we analyzed the metrics that are used today in the practice of student learning on the e-learning platform. As a result, propose a set of mandatory metrics directly related to the employee's career selling.
In addition to metrics, suggest a process that should support:
1. Improvement of existing courses 2. Cancellation of obsolete or non-updated courses
This paper proposes an effective process of structuring and monitoring learning processes in a software company focusing on the applicability of acquired knowledge in real projects, addressing the following research questions:
1. How can an engineer's skills, experience, and qualifications be assessed, measuring the gap between their knowledge and readiness for the role? 2. How do we determine engineers' specific training needs and interests, regularly updating training materials to align with industry demands and emerging technologies? 3. How do we evaluate the effectiveness of engineers' education by measuring various business metrics that reflect their knowledge acquisition, skill development, and overall growth? 4. How can we use learning system analytics to calculate KPIs that align with business performance? (How do we calculate business-result-oriented KPIs based on typical analytics supplied by learning systems and correlate them with actual business performance?)
The critical question is how to raise the readiness of engineers for an adequately configured learning process in companies focused on improving business outcomes like productivity, efficiency, customer satisfaction and adaptivity to market need changes.
To effectively monitor corporate education, continuous data collection and analysis must guide decisions on improving the process. Even at the beginning of this century, after the mass adoption of the e-learning system, proposals for the appearance of the Learning analytic cycle began. The authors propose a process consisting of four basic steps in the paper [5]: learnings-data-metrics -interventions, Figure 1: The preliminary learning analytics cycle proposed by Clow proposed by Clow. When it comes to learning analytics, the third step involves taking the data collected and turning it into clear metrics or analytical constructs that can provide deep insights into the learning process. These insights can handle many forms, such as visual representations, detailed dashboards, lists of students needing extra support, comparisons to previous benchmarks or cohorts, and overall summaries [5]. This step is crucial to learning analytics and drives innovation in many areas, including dashboards, predictive models, social network analysis, and personalized recommendations. This stage is a crucial aspect of learning analytics projects. Today, the learning analytics cycle consists of up to seven significant steps, each contributing to a deeper understanding of the learning process. There are four distinct stages: Fetch, Know, Proceed, and Enhance, each with one or more steps. For details, see Figure 2. The four phases of the cycle are get, learn, continue and improve, each with one or more steps.
The four phases of the cycle are get, learn, continue and improve, each with one or more steps.
Stage 1: Fetch (steps 1,2) 1. Data collection: In this phase, data is collected from various sources such as learning management systems, online platforms, assessments, surveys and interactions.
Data may include student demographics, interaction patterns, quiz scores, time spent on assignments, and social interactions.
2. Data processing: Once collected, the raw data is processed to clean and organize for analysis. Data can be transformed into usable formats, missing values can be addressed, and irrelevant or redundant information can be removed.
Stage 2 Know (3,4) 3. Data analysis: This is the essential stage where the data is analyzed using various techniques to discover patterns, trends, correlations and insights. Descriptive analytics summarise what happened (e.g. engagement rates, completion rates). Predictive analytics predicts future performance based on historical data (e.g., identifying students at risk of dropping out). Prescriptive analytics suggests interventions or strategies to optimize learning outcomes (e.g. recommending personalized learning paths).
4. Interpretation and creation of insights: Data analysis results are interpreted to gain insight into student behaviour, engagement, and performance. These insights can be used to understand factors contributing to success or challenges in the learning process.
Based on the insights gained, educators, trainers and instructional designers can make informed decisions to improve the learning experience. This may include adapting teaching strategies, providing additional support for struggling students, or improving course materials.
6. Evaluation and assessment: After implementing the interventions, the cycle continues with the assessment of the impact of these changes on learning outcomes.
This evaluation phase provides feedback on whether the interventions effectively addressed the identified problems.
Stage 4: Enhance (7) 7. Continuous improvement: Insights gained from each cycle refine future learning strategies and interventions. This iterative process continuously improves instructional design, content delivery, and student support.
Overall, the learning analytics cycle emphasizes data-driven decision-making and the iterative nature of improving the learning experience. It empowers educators and trainers to adapt their approaches, improve student engagement and optimize the learning journey [6].
The challenges of learning analytics are: Technical challenges include data privacy and ethics, data integration, data quality and accuracy, interpretation and contextualization, and identifying relevant metrics.
Predictive challenges: These challenges include predictive accuracy interventions and actionability [7].
Organizational challenges include resistance to change, skill gaps, scalability, cultural shifts, and resource allocation.
Learner challenges: These challenges include learner diversity and long-term sustainability. The focus of this paper is on the importance of metrics in education. As mentioned earlier, selecting appropriate and high-quality metrics is a significant technical challenge, which will be discussed in the following chapter. The effective use of learning analytics systems to select suitable metrics can help us find answers to the research question of this paper. This includes improving existing courses, discontinuing outdated or irrelevant classes, and introducing new areas of knowledge.
To accurately evaluate and track the success of educational programs, obtaining accurate and detailed data inputs is of utmost importance. This allows for the implementation of effective strategies and the identification of areas that may require improvement.
What has implied the effectiveness of the entire process described in the previous chapter is highquality data inputs, clear and precise, which speak about the phenomena we want to measure and monitor [8].
What are the main challenges in choosing the appropriate metric to base the analytical learning monitoring system?
The challenge of identifying relevant metrics in learning analytics is choosing the right metrics to analyze that reflect learning progress and outcomes. This cannot be easy because many metrics can be used to measure learning, and the right metrics will vary depending on the educational context. Some of the factors to consider when identifying relevant metrics include: 1. The educational goals of the institution or course.
2. The specific learning objectives that are being targeted.
3. The types of data that are available. 4. The technical capabilities of the learning analytics system. 5. The needs of the stakeholders who will be using the analytics results. It is also essential to consider the limitations of metrics. No single metric can perfectly measure learning; different metrics can be more or less relevant depending on the context [9]. It is essential to use a variety of metrics to get a more complete picture of learning.
The steps before selecting relevant metrics for monitoring e-learning effectiveness: 1. Start by identifying the specific learning objectives of the course.
2. Consider the types of data that are available.
3. Talk to stakeholders who will be using the analytics results to get their input on the metrics that are most important to them. 4. Be aware of the limitations of metrics and use a variety of metrics to get a more complete picture of learning progress and outcomes. Implementing learning analytics metrics provides valuable insights into the impact of knowledge and skill acquisition on the effectiveness of software engineering [7].
A company must treat its employees with the value they bring since employees' work determines what customers perceive and determines business success or failure. An employee might be able to be replaced physically, but his skill sets and knowledge cannot be [10]. This is because each person engaged brings a different skill set to the table even though the job yields the same set of skills.
Selection of relevant metrics focused on improving productivity and efficiency: the course content, design, and delivery. 6. Transfer of learning: Measures how well learners can apply the knowledge and skills they learned in the eLearning course to their work. It can be measured by tracking factors such as performance on job-related tasks, productivity, and customer satisfaction. 7. Business impact: Measures the financial or operational benefits of the eLearning course. It can be measured by tracking cost savings, increased revenue, or improved customer satisfaction. This approach allows for a comprehensive analysis of the learning process and its effect on the software engineering process. This methodology will enable businesses and academic institutions to understand better the correlation between knowledge acquisition and the software development cycle, ultimately improving productivity and efficiency.
This paper aims to explore how we can connect the metrics we obtain from e-learning analytical systems to a company's operational goals and performance. To achieve this, we analyzed the metrics discussed in the previous chapter, focusing on how they relate to the company's profits. To demonstrate the company's performance, we utilized the Balanced Scorecard method introduced by Harvard University experts Robert S. Kaplan and David P. Norton in a 1992 HBR article [10]. This approach has significantly impacted how we monitor and manage the performance of companies in the future.
The Balanced Scorecard approach to monitoring and improving a company's performance is a strategic management tool that provides a comprehensive view of the company's performance from four perspectives [10]:
• Financial perspective: Measures how well the company achieves its financial goals, such as profitability, return on investment, and cash flow. • Customer perspective: Measures how well the company is meeting the needs of its customers, such as customer satisfaction, customer loyalty, and market share. • Internal business processes perspective: Measures how well the company executes its internal business processes, such as operational efficiency, product quality, and time to market. • Learning and growth perspective: Measures the company's ability to learn and grow, such as employee satisfaction, training and development, and innovation. The balanced scorecard approach is valuable for monitoring and improving a company's performance because it provides a holistic view rather than focusing solely on financial measures. This allows companies to identify and address potential problems early on and make more informed decisions about allocating resources.
To align the metrics gathered from learning platforms' analytical systems with the Balanced Scorecard methodology's four perspectives, we followed a five-step process during workshops and established it as a regular practice.
It is identifying goals and aligning the eLearning system.
1. We identified our short-term and long-term goals and considered how the eLearning system could help us achieve them. 2. We then chose metrics for the four perspectives of the balanced scorecard (financial, customer, internal business processes, and learning and growth) and tied them to KPIs that directly relate to our company's strategic goals. These KPIs were measurable and actionable. Data collection and analysis 3. Semi-annually, we collected data from our earnings analytical systems and calculated the KPIs we monitored. 4. We then analyzed the results to identify trends and improvement areas. Performance improvement 5. Annually, we took planned and systematic action to improve our performance, which required an investment. This may have included developing new strategies, changing existing processes, or investing new resources. While working on several workshops and monitoring the impact of metrics obtained from analytical systems, we created a table that shows a direct correlation between investment in engineering education and company performance [11]. The effects of employee skills and competencies on a software company's Balanced Scorecard are shown in Table 1-The impact of employee skills and competencies on company performance: an analysis from a balanced scorecard viewpoint.
The impact of employee skills and competencies on company performance: an analysis from a balanced scorecard viewpoint.
Area Employees' skills and expertise can be better matched to tasks to maximize utilization.
The data are shown in Table 1-The impact of employee skills and competencies on company performance: an analysis from a balanced scorecard viewpoint serves as an illustrative sample, demonstrating the chosen approach. The specific metrics included may exhibit variability contingent upon the goals and objectives of the observed organization. However, it is noteworthy that this dashboard offers a comprehensive overview of critical domains that have the potential to influence employee growth and development.
By consistently monitoring these metrics, an organization can gather valuable insights into how employee learning and development impact financial and overall performance. This information can be pivotal in optimizing employee development programs and initiatives [12].
For instance, when tracking the metric "time to completion," one can assess the duration it takes for employees to acquire the skills and knowledge requisite for success in their respective roles. This data can be instrumental in identifying areas where enhancements may be warranted in employee development programs.
Similarly, tracking the metric "customer satisfaction" allows for an evaluation of the level of satisfaction among customers concerning the services provided by employees. This information can aid in pinpointing areas where the company's employee development programs can be refined to better align with customers' needs.
A company can better understand how employee learning and development impact their performance through diligent tracking of these and other metrics. This information can be used to make judicious, data-driven decisions regarding optimizing employee development programs and initiatives, aligning them with the organization's goals and objectives.
To ensure that business-related Key Performance Indicators (KPIs) are aligned with actual business performance, it is essential to derive them from the abovementioned metrics.
Two important metrics to consider are: 1. Transfer of learning: This measures how effectively learners can apply the knowledge and skills gained from the eLearning course to their work. It can be evaluated by monitoring their performance on job-related tasks, productivity, and customer satisfaction. 2. Business impact: This measures the financial or operational benefits of the eLearning course. It can be evaluated by keeping track of cost savings, increased revenue, or customer satisfaction improvements for new and existing projects. Appropriate metrics in learning analytics involve considering educational goals, learning objectives, data availability, technical capabilities, and stakeholder needs. Employing a variety of metrics is essential to obtain a comprehensive picture of learning progress and outcomes.
Identifying relevant metrics in the context of learning analytics is a challenging task that requires careful consideration and a deep understanding of educational goals and processes. This paper introduces an effective process for structuring and monitoring learning processes within software companies, emphasizing the practical application of acquired knowledge in real projects.
To initiate the process of implementing learning analytics, we analyze existing metrics employed in e-learning platforms and propose a set of mandatory metrics directly linked to employee career development. Beyond metrics, we suggest a process to support the improvement of existing courses, the elimination of obsolete content, and the introduction of new knowledge areas.
The paper discusses learning analytics challenges, including technical, predictive, organizational, and learner-related challenges, emphasizing the significance of selecting appropriate and high-quality metrics to assess educational effectiveness. The paper provides a comprehensive set of metrics focused on improving productivity and efficiency in software engineering education, addressing completion rates, pass rates, time to completion, engagement, feedback, transfer of learning, and business impact.
This paper emphasizes the critical importance of aligning a company's future success and growth with the personal development of its employees. A key objective is the establishment of annual goals for employees, explicitly targeting career development and advancement [12]. Employees are expected to enhance their skills and competencies over the business year to become eligible for new roles or promotions.
We also suggested a process to support improving existing courses, cancelling obsolete or nonupdated courses, and introducing new areas of knowledge. This process includes the following steps:
1. Collect data on the mandatory metrics. This can be done through surveys, interviews, and performance reviews. 2. Analyze the data to identify trends and areas for improvement. For example, if the course completion rate for a particular course is low, this may indicate that the system is too challenging or not relevant to the needs of the employees. 3. Make changes to existing courses or develop new procedures as needed. When changing existing systems, it is vital to consider employee feedback and ensure that the courses align with the company's strategic goals. 4. Monitor the metrics after making changes or developing new courses to ensure the desired results are achieved. This will help to identify any areas where further improvement is needed.
In conclusion, this paper underscores the pivotal role of employee skills and competencies in a software company's Balanced Scorecard and presents a comprehensive framework for aligning organizational success with employee development through the judicious use of metrics and learning analytics.