PERKS: Eliciting and Exploiting Procedural Knowledge in Industry 5.0 Ioan Toma1 , Umutcan Serles1,2 1 Onlim GmbH, Weintraubengasse 22, A-1020 Wien, Austria 2 University of Innsbruck, Technikerstrasse 21a, A-6020 Innsbruck, Austria Abstract Procedural Knowledge (PK) is essential for industrial operations but often challenging to manage. The PERKS project addresses this by developing AI-based tools and methodologies to capture, manage, and deliver PK throughout its lifecycle. By focusing on human-centric solutions and testing in diverse industrial settings, PERKS aims to improve operational efficiency and worker performance while enabling knowledge transfer across industries. The PERKS results are applied and tested in three industrial scenarios (white goods production plant, computer numerical control machines, microgrid testbed) providing different use cases in terms of PK complexity and industrial requirements. Keywords Procedural Knowledge, Knowledge Graphs, LLMs, Hybrid AI 1. Introduction PERKS “Eliciting and Exploiting Procedural Knowledge in Industry 5.0”’ is a Horizon Europe collabora- tive research project funded by the EU. It started in October 2023 and is expected to last 36 months. The PERKS project supports the holistic governance of industrial Procedural Knowledge (PK) in its entire life cycle, from elicitation to management and from access to exploitation [1]. PERKS bases its solutions on leading-edge AI (both symbolic and sub-symbolic) and data technologies, by advancing and integrating existing methodologies and tools in terms of readiness, flexibility and user acceptance. Besides AI and data, the third pillar of PERKS consists of people: the goal is to put industry workers at the centre, in line with the Industry 5.0 vision, to satisfy their concrete needs, to provide AI-powered digital tools to perform their tasks better and more easily, following a human-in-the-loop paradigm to enhance the technologies and the solutions. This paper presents the PERKS research project1 which aims to provide the support for eliciting and exploiting procedural knowledge in Industry 5.0. The reminder of this paper is organized as follows. Section 2 describes the main objectives of the project, Section 3 presents the project’s current status and Section 4 introduces the PERKS solution and details the current implementation of one of the PERKS use cases. Finally Section 5 reports on potential interests and collaborations with other projects. 2. Objectives PERKS objectives are standardizing the digital representation of procedures, investigating the best methods and technologies to elicit procedures from documents and people experience, designing solutions based on knowledge graphs [2] to manage, preserve and give access to procedures, and providing tools and systems to allow people to search and find procedural knowledge when and where they need it. Moreover, PERKS aims for a holistic governance throughout the entire life cycle, because knowledge evolves and changes over time. The rest of this section details the objectives of the PERKS project: RuleML+RR’24: Companion Proceedings of the 8th International Joint Conference on Rules and Reasoning, September 16–22, 2024, Bucharest, Romania Envelope-Open ioan.toma@onlim.com (I. Toma); umutcan.serles@onlim.com (U. Serles) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 https://perks-project.eu/ CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 1. AI-powered tools to assist industrial workers in managing and executing procedures. These tools will be designed to improve efficiency, productivity, and knowledge accessibility. The tools effectiveness will be measured through usability assessments, performance metrics, and data transparency evaluations. The developed tools will be integrated into real-world industrial settings to assess their impact. 2. Standardized ontology for representing procedural knowledge. This ontology will facilitate knowledge sharing and reuse across different industrial contexts. By aligning with existing standards and ensuring interoperability, the ontology will enable the development of AI-powered tools and services. PERKS will evaluate the ontology’s effectiveness through rigorous testing and validation in real-world industrial scenarios. 3. Improving AI and data technologies through human-in-the-loop collaboration. By involv- ing industry experts, PERKS aims to enhance the reliability, transparency, and accountability of AI systems. This approach will be measured by evaluating system performance, user satisfaction, and alignment with trust and safety standards. 4. Practical impact of the project’s solutions in real-world industrial settings. By applying the developed tools and methodologies to diverse use cases, PERKS aims to measure improvements in productivity, efficiency, and sustainability. The project will utilize established metrics and frameworks to assess economic, social, and environmental impacts. 3. Current Project Status The PERKS digital solution makes use of Artificial Intelligence to better manage the set of skills, information, and experiences related to “know-how” in industrial processes. The project started 10 months ago and involves a consortium of technical partners (Cefriel, DFKI, NTT Data, Onlim, Vienna University of Economics and Business) as well as industrial use case companies (BEKO Europe, Fagor Automation and Siemens). In these first months, the extensive discussions and collaborative workshops between the partners led to a set of clear and shared business and technical requirements. Those represent a blueprint for a large set of corporate scenarios beyond the boundaries of the project scope, as also testified by the initial market analysis carried out by International Data Corporation (IDC). The envisioned solution will represent procedural knowledge in a form that AI applications can easily leverage to support different stakeholders to manage, execute, audit and validate both the procedure definitions as well as their application on the field. In the next period, the PERKS project will build upon those foundations and will implement a first version of the digital solution to be deployed and tested in BEKO Europe, Fagor Automation and Siemens, to prove the effectiveness of the proposed approach to realize the PERKS vision. 4. PERKS Solution PERKS solution consists of various modules that enable AI-supported creation and consumption of procedural knowledge. In the following, we will first briefly introduce each module and give an indicative example of how a conversational agent can assist a user with the consumption of procedural knowledge. 4.1. Modules Figure 1 shows an overview of the PERKS solution [3]. The solution consists of 6 major modules: • Knowledge Extraction and Collection: The main responsibility of this module is to generate procedural knowledge from various sources with different nature with different levels of automa- tion. This includes automated extraction of procedural knowledge from unstructured text (e.g., PDF files), semi-automated generation from semi-structured data (e.g., JSON, XML, HTML files) and manual creation of procedural knowledge by domain experts. Figure 1: An overview of the PERKS solution and its components. • Human-in-the-loop (HITL) knowledge validation: This module enables domain experts to validate and approve generated knowledge [4]. This module is particularly important to ensure the correctness of (semi-)automatically generated knowledge. • Procedural knowledge auditing: This module provides a mechanism for domain experts to audit executed procedures to identify pitfalls and diagnose problems with the execution. • Procedural Knowledge Management System (PKMS): PKMS is the core module of the PERKS solution. It manages the lifecycle of procedural knowledge graph [5], including formal verification and hosting of created knowledge in a knowledge graph, versioning and querying of the procedures in the knowledge graph. • Conversational AI: This module provides the main interaction mechanism with the procedural knowledge for the end-users (e.g., operators in a factory). The conversations can be structured (e.g., state-based) based on the procedural knowledge stored in the knowledge graph, but can be also in a question answering setting where the questions are answered over unstructured text, including automatically generated FAQs. • Custom Applications: A toolbox of applications specifically designed to meet the unique requirements of a certain use case. 4.2. Indicative Prototype: Beko Europe Use Case Beko Europe manufactures home appliances in large factories. The manufacturing process is involved with many procedures, including LOTO (Lockout TagOut) procedures, which are series of steps that needs to be applied to take equipment used in manufacturing out of service in order to enable safe maintenance and servicing. LOTO procedures are typically more than just turning off a switch and needs to be carried by authorized personnel in a systematic manner. We prototyped an LLM-based conversational agent that can make use of the procedural knowledge stored in a knowledge graph with the LangChain framework2 . This prototype only implements the conversational AI module supported by a primitive representation of a procedure as a schema:HowTo instance3 . The agent uses the ReAct prompting technique[6], which allows an LLM agent to mimic human reasoning process and take certain actions based on thoughts and observations. 2 https://www.langchain.com/ 3 See the full schema here: https://semantify.it/ds/eh3AMgToN Listing 1: The zero-shot ReAct prompt for recursive procedure consumption 1 You a r e a b o t t h a t p r e s e n t s p r o c e d u r a l knowledge from a knowledge b a s e . Complete t h e o b j e c t i v e a s b e s t you can . You have a c c e s s t o t h e f o l l o w i n g t o o l s f o r you t o a c c e s s t o t h e knowledge b a s e : 2 3 { tools } 4 5 Use t h e f o l l o w i n g f o r m a t : 6 7 Q u e s t i o n : The i n p u t q u e s t i o n you must answer 8 Thought : You s h o u l d a l w a y s t h i n k a b o u t what t o do . 9 A c t i o n : t h e a c t i o n t o t a k e , s h o u l d be one o f [ { t o o l _ n a m e s } ] 10 Action Input : the input to the action 11 Observation : the r e s u l t of the action 12 Thought : you need t o t h i n k a b o u t what do w i t h t h e O b s e r v a t i o n 13 . . . ( t h i s Thought / A c t i o n / A c t i o n I n p u t / O b s e r v a t i o n s e q u e n c e can r e p e a t m u l t i p l e t i m e s ) 14 Thought : I now know t h e f i n a l answer 15 F i n a l Answer : t h e f i n a l answer t o t h e o r i g i n a l i n p u t q u e s t i o n 16 17 Here a r e some i n s t r u c t i o n s f o r you : 18 19 I f t h e q u e s t i o n a s k s f o r a p r o c e d u r e , t h e n l o o k f o r t h e p r o c e d u r e w i t h t h e G e t P r o c e d u r e t o o l i n your a c t i o n . The a c t i o n i n p u t i s t h e name o f t h e p r o c e d u r e . 20 21 start_function 22 The o b s e r v a t i o n O w i l l t e l l you a p r o c e d u r e c o n s i s t i n g o f z e r o o r more s t e p s . A s t e p i n O i s i n t h e form o f ” { { < u r i > } } _ { { position } } _ { { step text } } ” . 23 24 For each Step in O 25 26 − P r o d u c e a t h o u g h t a b o u t s e a r c h i n g f o r t h e s u b s t e p s o f S t e p . P r o d u c e an a c t i o n G e t P r o c e d u r e . P r o d u c e A c t i o n I n p u t i n t h e form o f ” { { u r i } } _ { { s t e p t e x t } } ” . 27 − The u s e r w i l l g i v e you a new O b s e r v a t i o n O1 . A s t e p i n O1 i s i n t h e form o f ” { { < u r i > } } _ { { p o s i t i o n } } _ { { s t e p t e x t } } ” . 28 − I f O1 i s ” There i s no s u b s t e p . ” , p r o d u c e an a c t i o n and a c t i o n i n p u t w i t h t h e n e x t S t e p i n O 29 − I f O1 i s s o m e t h i n g e l s e , a p p l y t h e i n s t r u c t i o n s between s t a r t _ f u n c t i o n and e n d _ f u n c t i o n t o O1 . 30 31 − P r o d u c e a t h o u g h t a b o u t knowing t h e f i n a l answer when t h e s u b s t e p s o f a l l s t e p s and t h e i r s u b s t e p s a r e c h e c k e d r e c u r s i v e l y w i t h t h e l o o p a bov e and o b s e r v a t i o n f o r e a c h p r o d u c e d ” There i s no s u b s t e p . ” . 32 end_function 33 34 Apply t h e i n s t r u c t i o n s between s t a r t _ f u n c t i o n and e n d _ f u n c t i o n t o t h e q u e s t i o n . 35 36 P r e s e n t F i n a l Answer by u s i n g t h e i n f o r m a t i o n i n e a c h O b s e r v a t i o n . Orde r t h e s t e p s by p o s i t i o n . 37 Begin ! 38 39 Question : { input } Listing 1 shows the prompt used for the prototype. Line 1 is an introductory prompt to tell the agent its task. Line 3 is a place holder for available tools. These typically functions that generate observations for the agent based on some external source. In our case, we use only one tool called GetProcedure, which runs a single SPARQL query with some given parameters to retrieve the (sub)steps of a procedure. Lines 7-15 presents the expected output from the agent in an abstract way. Given a question, the agent is expected to produce a series of thought-action-observation patterns until it reaches to final answer. The lines 17-32 define a recursive function including an exit condition in natural language. This way, the agent knows until when it should produce the thought-action-observation patterns. The lines 34-36 calls the defined function and provide some formatting instructions. Line 39 is a place holder for the input. Listing 2: An excerpt from the ReAct prompting outcome for querying a LOTO procedure Thought : I need t o r e t r i e v e t h e p r o c e d u r e f o r t u r n i n g o f f p n e u m a t i c s . Action : GetProcedure Action Input : pneumatics O b s e r v a t i o n : < h t t p s : / / o n l i m . com / e n t i t y / N e c 0 f a 4 7 5 9 5 4 d 4 9 2 a b b 7 2 2 a d a b c d e 6 3 d 1 >_1 − D i s c o n n e c t i o n P r o c e s s < h t t p s : / / o n l i m . com / e n t i t y / N 8 6 8 4 a 7 7 c 3 3 2 a 4 0 b f a b 4 9 0 9 f 8 1 8 9 f f f 3 2 >_2 − V e r i f i c a t i o n I now have t h e f i r s t s t e p o f t h e p r o c e d u r e , which i s t h e d i s c o n n e c t i o n p r o c e s s . Now I need t o s e a r c h f o r t h e s u b s t e p s o f t h i s step . Action : GetProcedure A c t i o n I n p u t : < h t t p s : / / o n l i m . com / e n t i t y / N e c 0 f a 4 7 5 9 5 4 d 4 9 2 a b b 7 2 2 a d a b c d e 6 3 d 1 > _ D i s c o n n e c t i o n P r o c e s s O b s e r v a t i o n : < h t t p s : / / o n l i m . com / e n t i t y / N e 2 e 0 4 f d 1 4 2 f e 4 5 7 4 9 b 5 3 d 8 6 7 c 7 e 5 c 3 f 7 >_1 − R o t a t e b a l l v a l v e A−1 90 d e g r e e s from open ( v e r t i c a l ) to closed ( horizontal ) position ... I have f o u n d t h e s u b s t e p s f o r t h e d i s c o n n e c t i o n p r o c e s s . Now I need t o s e a r c h f o r t h e s u b s t e p s o f e a c h s u b s t e p . Action : GetProcedure A c t i o n I n p u t : < h t t p s : / / o n l i m . com / e n t i t y / N e 2 e 0 4 f d 1 4 2 f e 4 5 7 4 9 b 5 3 d 8 6 7 c 7 e 5 c 3 f 7 > _ R o t a t e b a l l v a l v e A−1 90 d e g r e e s from open ( v e r t i c a l ) to closed ( horizontal ) position .... O b s e r v a t i o n : There i s no s u b s t e p . Thought : I now know t h e f i n a l answer . F i n a l Answer : 1. Disconnection Process : − R o t a t e b a l l v a l v e A−1 90 d e g r e e s from open ( v e r t i c a l ) t o c l o s e d ( h o r i z o n t a l ) p o s i t i o n − Apply ed b l o c k t h e r e d p a d l o c k ed t h e r e d TAG − D i s c h a r g e t h e r e s i d u a l a i r p r e s e n t i n t h e c i r c u i t u s i n g t h e gun c o n n e c t e d t o t h e T un der v a l v e A−1 2. Verification : − Check t h a t no more a i r comes o u t o f t h e gun and t h a t t h e p r e s s u r e on t h e p r e s s u r e g a u g e s on t h e machine h a s d r o p p e d t o zero Listing 2 shows an excerpt from a run of the agent with the input ”turn off pneumatics”. The agent creates an initial thought about retrieving the procedure for turning of the pneumatics. Following this thought, it takes an action to call the GetProcedure function. GetProcedure function parses the input and uses the name as a parameter to the SPARQL query to retrieve the steps of procedure that contains the name ”pneumatics”. The result of this function produces an observation. An observation contains the URI of the returned step, its position and its name. Due to its instructions, the agent produces thoughts actions and observations for each returned procedure step recursively, until the SPARQL query returns no results, leading to a depth-first traversal of the procedure. Finally, it presents the answer, the entire solution. The advantage of this approach is that logic of the agent is mostly in the prompt, which gives the developers flexibility. The recursion logic does not have to be hard-coded in the application. Only the GetProcedure function needs to be implemented to call a single SPARQL query whenever a new procedure step is needed. The LLM agent decides when the procedure retrieval process is over. 5. Collaboration with other projects PERKS actively seeks collaboration with projects operating in similar or complementary industrial domains to enrich its procedural knowledge ontology and refine its technological solutions. By engaging with partners from diverse sectors, the project aims to identify additional use cases and requirements that can be incorporated into the ontology’s structure and functionality. This collaborative approach will enhance the ontology’s adaptability and applicability across various industrial contexts, ultimately strengthening the project’s outcomes. Furthermore, collaboration with other projects offers the op- portunity to leverage existing tools and methodologies, avoiding redundant efforts and accelerating development. By sharing knowledge and resources, PERKS can build upon the strengths of partner projects to create more robust and comprehensive solutions. Acknowledgments This work is funded by the HORIZON Innovation Action 101120323 PERKS “Eliciting and Exploiting Procedural Knowledge in Industry 5.0”. References [1] A. Rula, G. R. Calegari, A. Azzini, D. Bucci, I. Baroni, I. Celino, Eliciting and curating procedural knowledge in industry: Challenges and opportunities, in: Proceedings of the Third Conference on Digital Curation Technologies (Qurator 2022), Berlin, Germany, Sept. 19th-23rd, 2022, volume 3234, 2022. [2] D. Fensel, U. Simsek, K. Angele, E. Huaman, E. Kärle, O. Panasiuk, I. Toma, J. Umbrich, A. Wahler, Knowledge graphs: Methodology, tools and selected use cases, Knowledge Graphs (2020). URL: https://api.semanticscholar.org/CorpusID:210975360. [3] M. Hulea, M. Scrocca, V. Carriero, N. Pokaratsiri, S. Ostermann, U. Serles, I. Toma, J. Xavier Parreira, E. Flitz, L. Waltersdorfer, et al., Deliverable D4.1 Reference Architecture for a Procedural Knowledge Management System, Technical Report, PERK Project, 2024. [4] S. Tsaneva, M. Sabou, Enhancing human-in-the-loop ontology curation results through task design, ACM J. Data Inf. Qual. 16 (2024) 4:1–4:25. [5] U. Simsek, K. Angele, E. Kärle, J. Opdenplatz, D. Sommer, J. Umbrich, D. Fensel, Knowledge graph lifecycle: Building and maintaining knowledge graphs., in: KGCW@ ESWC, 2021. [6] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, React: Synergizing reasoning and acting in language models, arXiv preprint arXiv:2210.03629 (2022).