Advantages and challenges of extracting process knowledge through serious games

Abstract

Digitalization promises huge improvements in various domains, such as production, health care, or mobility, through the integration of big data and artificial intelligence (AI). However, AI often builds on labelled data but labeling data can be complex or expensive, depending on both the properties of the data and access to people with domain knowledge. In particular, an underexplored field is capturing process knowledge, i.e., knowledge about the relationships among process steps. In this work, we propose and evaluate a game-based approach for capturing process knowledge. Taking the cooking domain as an example, we developed a prototype, in which players act as chef and cook dishes following their own recipes while each action is logged. The captured data is then compared to ground-truth models of common recipes. While the quantitative evaluation shows a decrease in motivation as well as fewer logged steps, qualitative feedback from participants identifies possible improvements of the concept. In summary, games can be a suitable approach for extracting experts' process knowledge, when certain user requirements are considered.

Keywords

• 1 Process knowledge
• knowledge harvesting
• process mining
• knowledge extraction
• domain expertise
• serious games

1. Introduction

Sustainable knowledge management is a key topic in numerous domains, such as production [ 1 ], [ 2 ], health care [ 3 ], and management [ 4 ]. One particular question is how (expert) knowledge can be systematically captured digitally so that it can later be used as a knowledge base, for training, or for the creation of data-driven decision support systems [ 5 ]–[ 7 ]. While capturing specific types of knowledge is easy and can build on a vast pool of novices (for example, massive image classification via MTurk [ 8 ]), capturing expert knowledge becomes hard when access to experts is limited, expensive, or the tasks to be captured are complex [ 9 ].

In this paper, we consider the special case of capturing domain-specific process knowledge, i.e., when not individual items need to be classified or labelled but when also the 6th International GamiFIN Conference 2022 (GamiFIN 2022), April 26-29, 2022, Finland EMAIL: [e-mail] (A. 1 – A.4); ORCID: 0000-0002-8584-0126 (A. 1); 0000-0003-2837-5181 (A. 3); 0000-0002-6105-4729 (A. 4) ️© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

CEUR Workshop Proceedings (CEUR-WS.org) relationships between different entities are of interest. A question in this area is if the digitization of knowledge can be improved by amplifier concepts such as gamification or serious games in terms of the amount of data or data quality and whether this can be linked to individual user characteristics. Although our research addresses the extraction of expert knowledge from process planning in manufacturing in the long term, here we consider process knowledge that many people have: The preparation of food with the recipes as manifestations of their process knowledge. At a later stage, we will transfer our concept and findings to the production domain. 1.1.

Background

In the long run, we aim at generating a digital representation of the process knowledge from experienced process planners in textile engineering. On the one hand, this sector is characterized by domain experts that have much tacit experiential knowledge or even knowledge in motor memory. On the other hand, most companies in the sector are often reluctant to exploit the opportunities offered by digitization and digital knowledge management [ 10 ]. Consequently, the potential of capturing and then using digital knowledge for training or building automated decision support systems is untapped.

Currently, process planning is more manual than digital: Planners usually write down their executed steps for a certain production process on paper, shortly after manufacturing the product [ 10 ]. Normally, this only includes the steps taken and not the reasoning behind the decisions. To make these textual artifacts usable for building training materials, knowledge bases, or for training an AI, they must be digitalized and formalized. Yet, this is cumbersome and errorprone for the workers, as many modern tools that are used within production settings, such as Excel, are confusing due to poor user experience and complexity [ 11 ]. Also, multiple workers will note down information differently, so the resulting digitalized information must be unified.

Another approach for gathering the required process knowledge might be to interview workers to formulize plans for several different products and ask for their reasoning in interviews [ 9 ]. Yet, this would be more cumbersome, as this would require additional staff for conducting the interviews, the interviews would have a limited time frame and would thus require focusing on the most important or difficult cases only [ 9 ].

Both approaches face two difficulties. First, they require the worker to work in a repetitive setting, which reduces internal motivation [ 12 ]. Second, the data would need to be digitalized, which would require human classification and domain expert knowledge, and additional computational overhead. 1.2.

Vision and approach

As a solution for these problems, we propose serious games as a method for extracting industrial process knowledge. Experts would playfully interact with a (simulated) production environment and thus share their experience and expertise with a system that captures all interactions. The knowledge captured digitally can then be used to train AI models for automation or decision support.

In our serious game, the worker is intended to play inside a gamified version of the shop floor, where all tools, machines and resources are available. As before, the player then gets prompted to manufacture certain products, while the game tracks his actions.

However, as this field is not yet researched, we conducted a proof-of-concept study, providing first insights on the pros and cons of our approach. To be able to reach more participants for the first proof-of-concept study, we realized a game for extracting cooking process knowledge instead of the specialized industrial use case. This allows us to gather extensive feedback more quickly, without the need for experts with their specific domain knowledge. The core idea should then be transferable to production use-cases, such as textile engineering, in the future.

Compared to previous approaches, this would have multiple advantages. One, the knowledge is immediately available, so the digitalization and unification would be simpler, faster, and more accurate. Two, serious games have shown an increase in motivation, which would favour the workers. The increased motivation could lead to increased productivity, benefiting the companies. Three, the time spent gathering the logs could be reduced, as all input will be stored in one place.

2. Related work

This chapter introduces the core concepts of our vision and relates these to existing research. 2.1.

Serious games

A Serious Games (SG) is a (often computermediated) game whose goal is not primarily entertainment, but that convey knowledge or behaviour change [ 13 ]. They usually use simplified abstractions of problems and are thus not necessarily complete [ 14 ]. In our case, we would build on the persuasive potential of games [ 15 ] to motivate people to share domain specific process knowledge. Note that SG differ from gamification, where unaltered activities are reinforced with game elements, such as timers, points, badges, or leaderboards [ 16 ]–[ 18 ].

Both gamification and SG have shown success in medical contexts, (e.g., reminding people to wash hands properly [ 19 ], [ 20 ]), personal education (e.g., increased learning of a new language [ 21 ] or to nudge students to learn efficiently [ 22 ].), but also in production (e.g., to convey knowledge and to study human behaviour in supply chains [ 23 ]).

2.2. Knowledge process mining extraction and

Knowledge Extraction (KE) is the act of gathering knowledge about a topic from structured sources, such as databases or XML, or unstructured sources, such as texts, images or—as in our case—games. The main goal is to create a ruleset or history for an AI to reason upon, to accurately predict solutions for the future. A very common approach is to create triplets, which are small information bits, linking multiple topics to each other. If enough triplets are created, one can follow this reasoning chain to create new information. This concept was the basis for the creation of the reasoner pellet [ 24 ]. KE is also used in medicine, either to provide data for dietary recommender systems [ 25 ] or to scrape patient information from clinical data [ 26 ].

While the above examples all focus on creating rulesets, Process Mining (PM) is working towards a unified process model [ 27 ]. This model can then be used to compare it with running work iterations or reasoned from. PM extracts information from event logs, which is a collection of activities, together with timestamps and process identifiers. PM defines a process as a theoretical series of activities (or actions of the worker), whereas a specific execution of this process is called a trace. Similar traces are grouped together, creating a variant, which in turn are used to create the model. PM also defines several disparity measurements between a variant and the model [ 28 ]. PM is widely used in business, as their production log is the ideal candidate to reason upon [ 27 ]. Computing a ruleset from a given dataset is difficult, as a wide variety of individual deviations as well as unification must be considered.

The combination of PM and gamification is promising, as they complement each other. This has been done in some cases, but not many in an industrial setting. For example, [ 29 ] used PM to classify data collected from a gamified experiment. We on the other hand would like to use gamification to create a better process log. In contrast, [ 30 ] created a gamified environment in a production setting, but without extracting or analyzing knowledge. Their evaluation showed mixed results, as tasks were completed faster but also failure rates increased.

As a reverse, [ 31 ] and [ 32 ] used gamification elements to facilitate learning in an industrial setting, either to teach lean manufacturing, or to identify warning indicators. While the authors used gamification in an industrial use-case, they focused on learning for the user, not extracting knowledge from them.

This overview highlights the missing research into combining PM and gamification. Both have previously shown benefits on their own, but only rarely together. Especially in the industrial usecase, where PM is widely used, the lack of a combined approach is glaring. 2.3.

Motivation

The major benefit for the workers would be higher hedonic motivation while sharing knowledge. Psychology divides motivation into intrinsic ("I work on this topic because it is fun.") and extrinsic ("I work on this topic because I get paid for it") motivations [ 33 ]. Here, intrinsic motivation is more important, as extrinsic motivation quickly degrades and tasks are not continued if the rewards decrease.

How can motivation be measured? Motivation can either be measured by using psychometric scales or by observing behaviour. Regarding the former, the Situational Motivation Scale (SIMS) is a validated scale that measures four dimensions of motivation, namely Intrinsic Motivation, Identified Regulation, External Regulation and Amotivation [ 34 ]. For the latter, the Free-Choice Measurement (FCM) can be used: Without any external control people can do a task or interact with a system. The time people invest is then an indicator of a persons’ motivation [ 35 ]. Combining both, SIMS and FCM, will provide richer reasoning behind the users’ behaviour. 2.4.

Research gap and objective

The extraction of process knowledge has been insufficiently solved so far. Serious games promise to motivate people to interact longer in a virtual environment and thus make capturing their process knowledge possible by logging their interactions. In this paper we investigate if process knowledge can be captured by means of a SG, whether a SG achieves better results than a control condition, and what role user diversity and motivation play. Our research is guided by the following hypotheses:

H0: Process knowledge can be captured by means of a SG.

As SG are often suggested as being more motivating, we compare the SG with a functionally equivalent control condition and postulate:

H1: Users of the serious game for knowledge harvesting report higher motivation than users of a control environment.

H2: User factors influence reported motivation after of the serious game

SG promise higher motivation and higher motivation goes hand in hand with higher performance. Therefore, the following two hypothesis address the

H3: A serious game captures more process knowledge compared to the control condition.

H4: A serious game provides more accurate process knowledge compared to the control condition.

H5: Higher motivation leads to more accurate process knowledge that can be captured.

Hypotheses H1 and H2 focus on the users’ motivation, whereas H3 and H4 address the benefit of KE by means of a SG. H5 connects both aspects, providing pointers for further research.

3. Implementation of conditions

To evaluate the feasibility of process KE by means of SG, we implemented a low-poly kitchen game using the Unity3D engine. We used WebGL to make the game accessible to participants using a browser, featuring keyboard and mouse input. It is designed as a top-down, fixed-perspective camera. Participants play a chef interacting with the different components of the kitchen. Figure 1 shows a screenshot of the game with the chef walking to the fridge.

The goal of the game is to extract the recipes for several dishes from the players by capturing their interactions in the virtual kitchen (i.e., to extract process knowledge in the cooking domain). To achieve this, a prompt displays only the name of the dish and the player is then given interaction opportunities to perform the steps he/she would take to cook the dish in real life.

Each interactable component in the game is modelled as either distinct cupboards, crates, or machinery. Cupboards hold container, i.e., pots and pans. Crates contain ingredients and machinery is e.g., an oven. Each container can hold an infinite amount of ingredients to reflect the different steps of a recipe, such as adding tomatoes. The container, and therefore the contained ingredients, can be cooked, baked and seasoned. The ingredients are divided into dairy products (milk, cheese, eggs), meats (fish, beef, minced beef), carbohydrates (noodles, bread) and vegetables (paprika, onions). Each category is contained in its own crate or inside a fridge. We choose these ingredients to allow for many possible recipes. Additionally, some of these ingredients can be cut into smaller pieces.

There are three distinct forms of interaction of increasing complexity: cutting ingredients, seasoning and cooking. Cutting ingredients will always result in the same outcome without any choice of the player. Seasoning recipes have a wider variety of choices, but it is generally understood to have only a small effect on the result. This is different to cooking, as—depending on the heat and time settings—it is possible to burn dishes in real life. To keep the complexity of the game low, burning dishes is not possible in the game. Figure 2 displays the user interface for interacting with the stove. The player can choose the heat level, as well as the duration and can see a preview of the current ingredients.

Stove

There are two kinds of recipe queries in the game: mandatory and free choice. Free choice recipes are not logged, and players can decide how many recipes they want to complete. Only the amount of completed free choice recipes will be used as a metric. Conversely, to complete an experiment each player must recreate the five mandatory dishes as recipes in the game. The recipes are green salad, omelettes, greek salad, burger and spaghetti bolognese. All interaction for these recipes is logged into a database, creating a process log. This allows a direct analysis of the resulting process models with the help of PM tools.

PM allows for either the recreation of a process model from a sufficient log, or conformance checking the log with a ground truth model. We have chosen the latter, as creating an accurate model would require hundreds of traces, which will not be feasible for early evolution of the concept. We have therefore created ground truth models for each of the mandatory recipes. The ground truth model for a burger is depicted in Figure 3. This model follows standard PM notation. Rounded rectangles represent different activities, + denotes an AND transition, whereas x denotes an XOR transition. Note that this model allows multiple vegetables by heaving a loop. + x

MincedMeat

Cook CUTSalad CUTTomato CUTBread x +

HandIn

We created an additional, functionally equivalent, drag-and-drop web interface as a control setting. This interface was intentionally designed in a bland, unenticing way to reflect the visuals of modern tools such as Excel. It does not include any form of gamification. All interactions and resources that are available in the cooking SG are also available in the control condition. Error! Reference source not found. depicts the interface.

4. Evaluation

To evaluate the general feasibility of our approach, we conducted a user study with the SG and a control group. The following sections present our experimental method, the sample, and the main results of the study. 4.1.

Method

The participants of our study were randomly assigned to either the SG or the control condition (game type as a between-subject factor). The control group is introduced to a bland drag and drop interface. Both groups have the same interaction possibilities and target recipes and were exclusively played on a computer. Participants were recruited from friends and the websites Positly and PollPool during May 2021. Due to the pandemic restrictions, they were able to choose their own place to partake in the experiment.

As independent variables, we collected the participants’ demographics using an online survey on Qualtrics, such as age, sex, as well as job type and -field. To further evaluate the influence of the effect of exploratory user factors, we further measured the participants’ attitudes towards technology [ 36 ] and their attitude towards games on 5-point Likert scales under the assumption that experienced players might evaluate the game differently than people who don’t enjoy playing games. Cronbach's α shows that both scales have a high internal consistency (gaming α=.916, attitude towards technology α=.911).

As dependent variables we measured a) the participants’ motivation after the interaction using the SIMS scale (intrinsic motivation α=.946, internal regulation α=.862, external regulation α=.837, amotivation α=.811), b) the number of process steps done for a recipe, and c) the quality of the recipes cooked by the participants measured by PM’s fitness measure. For the last two measures, log files captured the interactions with the system and thus the process steps while preparing the dishes. 4.2.

Description of the sample

Overall, 60 people participated in the study, 21 in the SG (34%) and 39 in the control condition (64%). Most of our participants were in the age range between 18–30 years and most of the participants were women (61%). In terms of their current employment, our sample was diverse, with participants working in technical and nontechnical domains (see Figure 6).

5. Results

In the following, the results of the experiment are presented in the order of the hypotheses.

Can process knowledge be captured by means of a serious game? First, we investigated whether process knowledge can be generated from the interaction logs of both the SG and the control condition and what the quality of the captured process knowledge is.

Across the five different recipes from the experiment, the participants performed on average 11 steps per recipe (see Figure 8). The resulting average fitness is .51 and thus satisfactory, with the fitness of the captured process model for the green salad being highest and for spaghetti being lowest).

Is the serious game more motivating than the control condition? To compare the reported Intrinsic Motivation between both conditions, we calculated a Mann-Whitney U (MW-U) test. Although the median Intrinsic Motivation appears lower for the SG condition (md=3.5) than for the control condition (md=4.7), this difference is not statistically significant (p=.223>.05). Therefore, H1 is not supported by the evidence.

Note the non-normal distribution, measured by a seven-point Likert scale, of intrinsic motivation for the SG, displayed in Figure 7. While the control group has a central peak at around 4.9, the SG version has two peaks (bimodal distribution) at 2 ("Didn't enjoy it") and 6 ("Did enjoy it").

Do individual user-factors influence motivation after interacting with the serious game? We surveyed six different user factors in this study: Gaming disposition, attitude towards technology, job field, highest degree, gender, and age. Neither job field, nor highest academic degree, nor the participants’ gender had any significant influence on intrinsic motivation (p’s≫.05).Error! Reference source not found. As both gaming and attitude towards technology are continuous measurements, we evaluated their relation to the intrinsic motivation with a linear regression. In neither of the games does Gaming disposition have a significant influence (serious game: p=.411, control: p=.086). On the other hand, attitude towards technology, has a significant influence on the SG version (p=.042<.05, est. β=-2.09, SE=.959, t=-2.18, R2=0.2). As this effect is negative, we conclude that participants with higher attitude towards technology found the SG less motivating.

Does the serious game capture more process data? Two measurements were analyzed to evaluate the validity of the H3. First, we compared the number of steps per recipe (see Figure 8). In the control condition, the participants contributed on average 16 recipe steps compared to 11 in the SG condition. A MW-U test showed that this difference is significant (p<.001). Consequently, H3 is refuted.

The second measurement is the Free-Choice Measurement. Figure 9 depicts the histogram for both versions. As both versions are non-normally distributed, we calculated a MW-U-test and there is no significant difference between both versions (p=.216) (n(C)=39, n(SG)=21, mdn(C)=0, mdn(SG)=0),. Therefore, the serious game does not provide more data compared to the control condition and H3 is discarded. Does the serious game provides more accurate process knowledge? As H3 evaluated the amount of data and not the quality thereof, we measured the difference in quality according to the models we provided.

To evaluate the accurateness of the participants’ recipes, the standard measurement in PM fitness was used. Figure 10 depicts the average fitness for each of the recipes in both versions. Here, the overall difference as measured by Welchs’ t-test is not significant (t(4)=-0.878, p=.406). Thus, the accurateness of the data acquired in the serous games is not higher compared to the control condition and H4 is not supported.

Does higher motivation of the participants yield more accurate process knowledge captured? Next, we analyse if the partcipants’ motivation relates to the accuracy of the captured process knowledge. We first consider the SG condition and then the control condition.

We compared the averaged fitness of all recipes from each participant with the participants’ SIMS. In the SG condition, no correlations between the averaged Fitness and Intrinsic Motivation (p=.252, R2=.068), Identified Regulation (p=.239, R2=.072), External Regulation (p=.720, R2=.007) or Amotivation (p=.204, R2=.083) from the SIMS scales were found. Thus, motivation was not linked to the accuracy of the captured process knowledge in the SG condition.

Contrary, there was a significant negative influence of both External Regulation (p=.009<.05, est. β=-2.06, R2=.185) and Amotivation (p=.009<.05, est. β=-2.48, R2=.185) on the average Fitness for the control condition , but no influence of Intrinsic Motivation (p=.234, R2=.058) and Identified Regulation (p=.324, R2=.022).

Thus, the findings suggest that motivation influences the accuracy of the captured process knowledge only in the control condition but not in the SG condition. Consequently, H5 is partially supported although a more thorough investigation with a larger sample size is needed.

6. Discussion

In this article, we presented the rationale for capturing process knowledge and a SG situated in a kitchen environment that aims at extracting recipes as one of the most common manifestations of process knowledge that most people have. The overall goal was to let the players create their own recipes for a set of dishes and compare the results with a ground truth. We wanted to analyze if a SGs approach provides two major benefits: Firstly, it should increase the motivation of the player, because it would be more interactive than the blander counterpart. This would have been a major benefit to the workers. Secondly, deriving from this increase in motivation, players should have created additional data, as well as have a higher accuracy of their recipes. This has been evaluated in an online experiment with a control group that used a functionally equivalent drag and drop interface for sharing recipes. Next, we discuss the findings of our experiment and provide pointers for further research.

First, our results indicate that we can extract peoples’ cooking knowledge for five common recipes in our study. The generated process logs were analyzed with PM metrics and achieved quite decent fitness. Thus, our SG approach for extracting process knowledge worked well.

However, in the end, none of our formulated research hypotheses that compared the SG against a conventional user interface could be validated. There was no significant difference in motivation (H1, as measured by the SIMS) between the playful SG and the rather dull control condition. As we targeted the SG towards the elderly workers, different user factors have been discussed (H2). Yet, there hasn't been a significant influence from age, gender, job field, or gaming disposition. Only attitude towards technology had a negative effect.

While there was a significant difference in the players’ intrinsic motivation, we could not yet identify the specific reasons for this effect. We found however that—independent of the experimental condition—older players reported a higher intrinsic motivation. This finding suggests that older participants might be more willing to share their experiences. A potential for KE and management that should be taped.

Additionally, the SG provided less additional data (H3) and no difference in data accuracy (H4). The only measurable difference between the two versions was the better usage of the cutting board, as nearly every player in the SG used it, while not even half of the control groups’ players used it. We attribute this to a more intuitive understanding and higher visual clarity of the cutting board compared to the text field in the control condition.

Due to its significant effort to program the SG compared to the conventional interface, we currently cannot recommend the SG in its current form to rise the workers’ motivation, or to increase the amount or quality of the extracted process knowledge.

As we were only able to show a negative effect of External Regulation and Amotivation on the control version (H5), we suggest evaluating this difference further. The lack of negative influence of these modifiers in the SGs is an interesting point for further investigations.

7. Limitations, outlook, implications

Of course, this study is not without limitations. The biggest limitation is certainly the small sample size, which limits the transferability and the consideration of user diversity effects. Also, we found that the steeper learning curve of the serious game led to more dropouts compared to the control condition and thus unequal group sizes. Nevertheless, the findings suggests that process knowledge can be captured digitally through serious games and that individual motivation, as a facet of user diversity, influences the result quantity and quality. This needs to be investigated and modelled in more detail in future studies. Ideally also under laboratory conditions and as a within-subject experiment, to mitigate the various biases of online survey.

A major downside of the online evaluation approach was the difficulty of learning the basic interaction with the game. Additional feedback provided by the participants centered around confusion about the game and its interactivity. While we provided a text-based tutorial, many participants didn't truly understand the SG, which lead to indecision and quitting the game. This difficulty resulted in a small sample size for the SG, which limits the overall validity of our findings. For further studies we thus need to flatten the learning curve, for example through appropriate tutorials, We also recommend a supplementary, qualitative experiment in which the participants get a live explanation and training session before being asked to provide the recipes. This could significantly reduce the participants’ confusion and might result in a significant change in the overall results. Of course, this would mean an even further increase in workload, compared to a drag-and-drop or Excel-based solution.

In summary, this study showed that process knowledge from the commonly known domain of cooking can be captured with a serious game, even if the consideration of motivational aspects revealed few surprises. While the digital capturing of cooking knowledge itself is only of marginal interest in specialized areas (e.g., to make cultural differences in the preparation of food measurable or for saving cultural heritage), the findings suggest the transferability of this concept to other domains and contexts. We postulate that this approach enables capturing process knowledge in areas of manufacturing that have been little digitised so far that may serve as data to increase automation and provide decision support [ 2 ], [ 10 ].

8. Acknowledgements

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy—EXC-2023 Internet of Production— 390621612. Anne Kathrin Schaar for immensely valuable feedback. We thank the anonymous reviewers for their immensely valuable feedback.

9. References

References

[1] M. Fakhar Manesh , M. M. Pellegrini , G. Marzi, and M. Dabic , “ Knowledge Management in the Fourth Industrial Revolution: Mapping the Literature and Scoping Future Avenues,” IEEE Transactions on Engineering Management , vol. 68 , no. 1 , pp. 289 - 300 , Feb. 2021 .

[2] P. Brauner et al., “A Computer Science Perspective on Digital Transformation in Production,” ACM Transactions on Internet of Things , vol. 3 , no. 2 , pp. 1 - 32 , May 2022 .

[3] C. El Morr and J. Subercaze , “ Knowledge Management in Healthcare,” in Handbook of Research on Developments in E-Health and Telemedicine , IGI Global , pp. 490 - 510 .

[4] P. Drucker , The Age of Discontinuity - Guidelines on Our Changing Society . New York, NY, USA: Heinemann, 1968 .

[5] F. Kimura , H. Ariyoshi , H. Ishikawa , Y. Naruko , and H. Yamato , “ Capturing Expert Knowledge for Supporting Design and Manufacturing of Injection Molds,” CIRP Annals , vol. 53 , no. 1 , pp. 147 - 150 , 2004 .

[6] P. Fantini , M. Pinzone , F. Sella , and M. Taisch , “ Collaborative Robots and New Product Introduction: Capturing and Transferring Human Expert Knowledge to the Operators,” in International Conference on Applied Human Factors and Ergonomics , 2017 , pp. 259 - 268 .

[7] M. Mansfield , V. Tamma , P. Goddard , and F. Coenen , “ Capturing Expert Knowledge for Building Enterprise SME Knowledge Graphs,” in Proceedings of the 11th on Knowledge Capture Conference , 2021 , pp. 129 - 136 .

[8] P. Welinder and P. Perona , “ Online crowdsourcing: Rating annotators and obtaining cost-effective labels,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops , 2010 , pp. 25 - 32 .

[9] R. R. Hoffmann , “ The Problem of Extracting the Knowledge of Experts from the Perspective of Experimental Psychology ,” AI Magazine , vol. 8 , no. 2 , p. 53 , 1987 .

[10] F. Brillowski , H. Dammers , H. Koch , K. Müller , L. Reinsch , and C. Greb , “ KnowHow Transfer and Production Support Systems to Cultivate the Internet of Production Within the Textile Industry ,” in International Conference on Intelligent Human Systems Integration (IHSI 2021 ), 2021 , pp. 309 - 315 .

[11] P. T. Kortum and A. Bangor , “ Usability Ratings for Everyday Products Measured With the System Usability Scale ,” International Journal of Human-Computer Interaction , vol. 29 , no. 2 , pp. 67 - 76 , Jan. 2013 .

[12] C. D. Fisherl , “Boredom at Work: A Neglected Concept,” Human Relations, vol. 46 , no. 3 , pp. 395 - 417 , Mar. 1993 .

[13] C. C. Abt , Serious Games . Madison Books , 1987 .

[14] D. Michael and S. Chen , Serious Games: Games That Educate, Train and Inform . Thomson Course Technology , 2006 .

[15] I. Bogost , Persuasive Games: The Expressive Power of Videogames . Cambridge, Massachusetts, USA: The MIT Press, 2007 .

[16] D. Djaouti , J. Alvarez , J.-P. Jessel , and O. Rampnoux , “Origins of Serious Games,” in Serious Games and Edutainment Applications , London: Springer London, 2011 , pp. 25 - 43 .

[17] J. Hamari , “ Transforming homo economicus into homo ludens: A field experiment on gamification in a utilitarian peer-to-peer trading service , ” Electronic Commerce Research and Applications , vol. 12 , no. 4 , pp. 236 - 245 , 2013 .

[18] J. Hamari , “Gamification,” in The Blackwell Encyclopedia of Sociology , Oxford, UK: John Wiley & Sons, Ltd, 2019 , pp. 1 - 3 .

[19] A. Higgins and M. M. Hannan , “ Improved hand hygiene technique and compliance in healthcare workers using gaming technology , ” Journal of Hospital Infection , vol. 84 , no. 1 , pp. 32 - 37 , May 2013 .

[20] P. Pereira , E. Duarte , F. Rebelo , and P. Noriega , “ A Review of Gamification for Health-Related Contexts ,” in International Conference of Design, User Experience, and Usability , 2014 , pp. 742 - 753 .

[21] W. L. Johnson , “ Serious Use of a Serious Game for Language Learning ,” International Journal of Artificial Intelligence in Education , vol. 20 , no. 10 , pp. 175 - 195 , 2010 .

[22] J. Simões , R. D. Redondo , and A. F. Vilas , “ A social gamification framework for a K-6 learning platform , ” Computers in Human Behavior , vol. 29 , no. 2 , pp. 345 - 353 , Mar. 2013 .

[23] P. Brauner , R. Philipsen , and M. Ziefle , “Projecting Efficacy and Use of Business Simulation Games in the Production Domain Using Technology Acceptance Models,” in Advances in Ergonomics of Manufacturing: Managing the Enterprise of the Future, C. Schlick and S . Trzcieliński, Eds. Springer International Publishing, 2016 , pp. 607 - 620 .

[24] E. Sirin , B. Parsia , B. C. Grau , A. Kalyanpur , and Y. Katz , “ Pellet: A practical OWL-DL reasoner , ” Journal of Web Semantics , vol. 5 , no. 2 , pp. 51 - 53 , Jun. 2007 .

[25] T. Eftimov , B. Koroušić Seljak , and P. Korošec , “ A rule-based named-entity recognition method for knowledge extraction of evidence-based dietary recommendations , ” PLOS ONE , vol. 12 , no. 6 , p. e0179488 , Jun . 2017 .

[26] G. K. Savova et al., “ Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications , ” Journal of the American Medical Informatics Association , vol. 17 , no. 5 , pp. 507 - 513 , Sep. 2010 .

[27] W. van der Aalst et al., “Business Process Mining: An Industrial Application,” Information Systems , vol. 32 , no. 5 , pp. 713 - 732 , Jul. 2007 .

[28] W. van der Aalst et al., “Process Mining Manifesto,” in Business Process Management Workshops, Springer, Berlin, Heidelberg, 2012 , pp. 169 - 194 .

[29] J. A. C. Hernández , M. P. Duarte , and J. M. Dodero , “ An architecture for skill assessment in serious games based on Event Sequence Analysis ,” in Proceedings of the 5th International Conference on Technological Ecosystems for Enhancing Multiculturality , 2017 , pp. 1 - 9 .

[30] O. Korn , M. Funk , and A. Schmidt , “ Towards a gamification of industrial production,” in Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems , 2015 , pp. 84 - 93 .

[31] M. Messaadia , A. Bufardi , J. Le Duigou , H. Szigeti , B. Eynard , and D. Kiritsis , “Applying Serious Games in Lean Manufacturing Training,” 2013 , pp. 558 - 565 .

[32] R. Patriarca , A. Falegnami , A. De Nicola , M. L. Villani , and N. Paltrinieri , “ Serious games for industrial safety: An approach for developing resilience early warning indicators , ” Safety Science , vol. 118 , pp. 316 - 331 , Oct. 2019 .

[33] R. Benabou and J. Tirole , “Intrinsic and Extrinsic Motivation ,” Review of Economic Studies , vol. 70 , no. 3 , pp. 489 - 520 , Jul. 2003 .

[34] F. Guay , R. J. Vallerand , and C. Blanchard , “ On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS),” Motivation and Emotion , vol. 24 , pp. 175 - 213 , 2000 .

[35] B. M. Wiechman and S. T. Gurland , “ What happens during the free-choice period? Evidence of a polarizing effect of extrinsic rewards on intrinsic motivation , ” Journal of Research in Personality , vol. 43 , no. 4 , pp. 716 - 719 , Aug. 2009 .

[36] T. Franke , C. Attig , and D. Wessel , “ A Personal Resource for Technology Interaction: Development and Validation of the Affinity for Technology Interaction (ATI) Scale ,” International Journal of Human-Computer Interaction , vol. 35 , no. 6 , pp. 456 - 467 , 2019 .