This paper addresses the i4Q project vision, including stakeholders' requirements and expectations, aiming to present the digital technologies, as well as a multi-dimensional benchmarking instrument that supports the i4Q design and development. It also sets clear specifications that drive the creation of i4Q. It analyses the current systems of the demonstration scenarios, to establish the starting point (Key Performance Indicators' (KPIs)) for the implementation of their industrial use cases and to understand how, data reliability and manufacturing quality, are impacted by i4Q. Finally, it focuses on the most suitable KPIs and identifies the most relevant regulation and trustworthy systems for data management in the i4Q Solutions.
Quality control in Industry 4.0 is embedded in the production line. Smart sensors register and transmit the data collected from the manufacturing line and use these to take the necessary decisions and finally improve manufacturing processes. Smart factories with high levels of digitalization will be a key element for the new form of industrial production based on Industry 4.0 initiatives. The challenge is the transformation of the cost-based competitive advantages into those that rely on sustainable, high-value-added production. In order to address this challenge, it is necessary to enable manufacturing companies to achieve superior product quality with highly efficient, and smart production processes. A successful smart factory needs to manage data-related processes along the entire data life cycle, including data collection, storage, distribution, analysis, use, and deletion, to ensure continuous high data quality.
i4Q [1] Reliable Industrial Data Services (RIDS) aim to support the complete flow of industrial data, starting from the data collection to data analysis, simulation and prediction. It provides solutions to ensure data quality, security and trustworthiness, such as blockchain-based data services and distributed storage. The i4Q Project will develop a set of solutions to improve the quality of manufactured products aiming at zero-defect manufacturing, hence pushing forward the concept of a smart, fully digitized factory [1].
This workshop paper aims to provide an overview to the target audience, perform a multidimensional assessment of current technologies for quality in manufacturing, establish benchmarks and capture the needs from industry to be satisfied by the project outputs.
A multi-dimensional benchmarking instrument was developed, in order to support the i4Q design and development. The state-of-the-art of emerging and promising digital technologies (e.g., Blockchain, Hyperledger, fog/edge computing, data analytics, big data, machine learning, IIoT, digital twin, etc.) was characterized. Each technology was described and analyzed, according to its state, maturity, tools, EU project solutions, benchmarking and assessment, application of i4Q solutions. The benchmarking evaluation framework was composed of different dimensions divided into different criteria to assess the various technologies that were analyzed and selected as relevant to fulfill the development objectives of the i4Q solutions. These five dimensions were the following: general, technological, business model, informational, social.
The criteria that have also been defined to be part of this evaluation were the following: capability/features, cost, coverage, development-friendliness, generality, integration, interoperability, learning curve, legal compliance, maturity, need for data traceability, need for quality data, performance, relevance, risk, scalability, security, social preferences, support, traceability, training and documentation.
Results were used to identify the best cases – the cases in which the technologies and tools meet the most criteria, as well as which of the solutions could be used by most tools – establishing benchmarks for each of the dimensions. More specifically, results showed that Relevance and Capability are the best-fulfilled criteria from the general criteria, while integration is the best fulfilled from the technological. Data Analytics, Machine Learning, and Big Data are the three technologies that meet most criteria.
Last but not least, Python libraries seem to be an appropriate data analytics/visualization relatedtool for the development of 8 of the 19 solutions. In the case of the digital technology of machine learning, Python seems to be the most appropriate tool to develop different solutions. Additionally, the most important big data related-tool is Tensorflow.
According to the ISO 22400 [2], a KPI is a quantifiable level of achieving a critical objective. This section, describes the methodology used to define, implement and visualize the KPIs. The ISO 22400 automation systems and integration KPIs for manufacturing operations management is taken as a reference document.
The KPIs will serve to quantitatively evaluate the results obtained by setting up i4Q-based solutions. The definition of the KPIs and its measurement will enable to compare the performance between the AsIs business processes and the ToBe business processes. The ToBe business processes are the set of activities performed with a set of resources, including i4Q solutions, to realize an objective within a specified timeline. Moreover, performance measures will allow establishing the starting point (KPIs baseline values) for the implementation of the industrial use cases.
In order to characterize and define the KPIs a top-down methodology has been considered to formalize the objectives to be achieved in each of the ToBe business process, and the objectives are converted/mapped into a set of KPIs. A good KPI has certain criteria which ensure its usefulness in achieving various goals in the manufacturing operation: aligned, balanced, standardized, valid, quantifiable, accurate, timely, predictive, actionable, traceable, relevant, correct, complete, unambiguous, documented, comparable, understandable and inexpensive. The KPIs are named as KPIxyk, and measure the achievement of the objectives Oxyk, where k is the KPI/objective formulated to measure/achieve the business process y in pilot x. The following structure identifies KPI descriptive elements: • Name (ID): Name of the KPI (user defined unique identification of the KPI). • Description: A brief description of the KPIxyk • Objective: Objectives to be realized with use of performance indicators determined • Unite of measure: The basic unit or dimension in which the KPIxyk is expressed • Data source: The source or sources from which the pilot is going to obtain the data needed to calculate the mathematical formula of the KPIxyk • Mathematical formula: The mathematical formula of the KPIxyk specified in terms of elements i.e. KPIxyk = algth(granu, w, pva) • Measurement timing: KPIxyk can be calculated either in real-time - after each new data acquisition event; on demand - after a specific data selection request; or periodically - done at a certain interval, e.g. once per day • Evaluation timing: KPIxyk evaluation frequency can coincide with the measurement timing • Trend: Is the information about the improvement direction of the x KPIxyk, higher is better or lower is better • Range: Specifies the upper and lower logical limits of the KPIxyk • Responsible for measurement: Responsible is the group typically measuring this KPIxyk. • Audience: Audience is the user group typically using this KPIxyk, i.e. operator, manager, etc. • Decision: Decision to be taken when the KPIxyk is out of the limits • KPI value: Number result of the KPIxyk mathematical formula • Data: Number results of the different data to be computed in the KPIxyk value (mathematical formula) • KPI measurement datetime: dd:hh
A KPI dashboard will enable to understand how, data reliability and manufacturing quality, will be impacted by i4Q. The KPI dashboard will be exploited for the evaluation of i4Q pilots.
Focused on establishing the KPIs baseline values of the current situation, and AsIs scenarios, of the pilots participating in i4Q project, the KPIs Dashboard represents KPIs information in graphical form to monitor in an intuitive way over time. The KPIs dashboard represents metrics and measures used to check the performance of i4Q Solutions, i4Q will use Dashboards that use charts and graphs to show the evolution of KPIs over time; allowing i4Q Pilots to easily see trends and be alerted to KPIs that have values out of minimum and maximum values. In this regard, the companies determine the improvements that are expected in their business processes after the implementation of i4Q-based solutions.
To gather requirements from industrial partners and software developers, an iterative procedure was established that is aligned with the standards and guidelines ISO/IEC/IEEE 29148 [3], ISO/IEC/IEEE 12207 [4], ISO/IEC/IEEE 15288 [5], VDI 2221 [6, 7] and VDI 2206 [8, 9]. Also, a model-based systems engineering (MBSE) [10] approach, similar to the function-based systems engineering [11], is applied to handle the complexity of the requirements and functional structures as well as their connections.
Pilot requirements are elicited and listed along the pilot’s business processes and rated with their priority and difficulty. Then, these requirements are refined and structured in requirement diagrams. Solution requirements are defined by the solution developers to specify interfaces and ensure interoperability of the i4Q solutions. The solution requirement lists include the needs for a single solution as well as the general requirements of the complete set of RIDS. To capture all dependencies and relations in-between the requirements and identify gaps, the system modeling language SysML [12] is used to model the mappings and structures according to [11, 13].
Function Structure Diagrams (FSD) [14] are prepared for every i4Q solution to describe the input, data flow, functionalities and output of every solution. Together with the requirements diagrams and lists, these FSDs are transferred to a MBSE software which creates diagrams for both perspectives: from the pilot's point of view, all requirements are represented in a tree-like structure, from higherlevel requirements to lower-level and more precise requirements, to which the appropriate solution function that should fulfill the requirement is then assigned. From the solution perspective, the functional architecture is derived from the FSD and all associated requirements are mapped as accurately as possible to the lowest level functions (Figure 1).
To enable project partners to plan the next steps of the software development an overall evaluation of the requirements and functionalities mapping is provided. This evaluation identifies possible highrisk solutions. The functional specifications which are described by the mapping diagrams are evaluated according to introduced KPIs, namely completeness, number of interfaces, precision and requirements origin.
Some core solutions have been identified which have many interfaces to other i4Q solutions and provide functionalities for requirements from all pilots: e.g., the data repository and data integration and transformation as well as trusted networks solutions which fulfill basic needs such as secure data acquisition, transformation and storage. In some cases, the level of abstraction of the solutions is high which results in less requirements from end-user side: e.g., blockchain, security handler and process qualification. On the other hand, the application of the analytical solutions such as infrastructure monitoring, quality diagnosis and digital twin are clearly described by user requirements since the functionalities seem well-known and detailed in the area of quality control tools in production lines.
With the steady increase of demand and use of software applications in most industrial sectors, the trustworthiness of software systems is receiving increased attention. Due to the high complexity of modern software systems for industrial use, most functions and algorithms implemented in software solutions remain hidden from the user in a black box. These software systems are only accepted and used to their full potential if users trust the system. Therefore, a sufficient level of trustworthiness has to be established, verified, and certified. Additionally, with the broader use of artificial intelligence (AI) new regulations are developed by law makers to ensure that new intelligent software systems are compliant to existing and future laws and regulations. 5.1.
Alongside the requirements and functional specifications driven by the needs of the end-users, due attention is to be directed towards the legal provisions which need to be complied with. In relation to ensuring trustworthiness there is an important upcoming piece of EU legislation – the AI Act. The latter is recently presented [15] by the European Commission and already sparked a wide discussion in terms of its influence on technology providers [16].
Although the AI Act is yet to be finalized, adopted and enforced, the proposed provisions are analyzed to elicit key requirements, which would be the legal indicators of a trustworthy AI, which could enter the EU digital single marker. These are namely [17]: • Data and data governance: This requirement mainly refers to the data sets, which will be used to train and validate the AI prior to hitting the EU digital single marker. Therefore, it would be expected that the used data sets are always relevant, representative, free of errors and complete. • Transparency for users: The promotion of human-centric policies at EU level is to be continued by the practical implementation of this requirement. It would call the AI service providers to disclose information to its users about the purpose, the characteristics, capacities and constraints of the respective AI-service, alongside information of its functionalities and maintenance. • Human oversight: One of the central requirements for a trustworthy AI is the principle of keeping a human in the loop. The human oversight exercised by at least two humans is a guarantee that risks associated with the use of the respective AI service, namely risks related to human right, security or health would be minimized. This is especially relevant to those AIsystems that the AI Act classifies as high-risk1. • Accuracy, robustness and cybersecurity: In view of the purpose and functionality of any AI system, due attention should be paid to achieving corresponding level of precision, firmness and cybersecurity. This is to be verified by sharing the metrics with the users. Furthermore, the existence of substitute plans and cybersecurity detection and management system are another dimensions of this requirement. • Technical documentation and record keeping: Last but not least, similarly to requirements popularized by the General Data Protection Regulation regime, the upcoming AI act would pose obligations to AI-system developers to be able at all times to demonstrate their compliance with the abovementioned requirements.
All these requirements are an important ingredient the legislator has previewed in order to provide a guarantee that the AI-based services available to EU citizens are of trustworthy nature. To this end, there’re duly considered in the i4Q context. 5.2.
To achieve trustworthiness within the future i4Q solutions in a first step key characteristics of trustworthy systems are identified. These key characteristics are collected from scientific publications and existing standards by conducting a systematic literature review. The result of this review is a ranked list of 21 key characteristics and attributes. Due to similar features of the found 21 key characteristics and attributes are condensed to five main key characteristics: • Safety – the ability of the system to operate without harmful states, • Reliability – the ability of the system to deliver services as specified, • Availability – the ability of the system to deliver services when requested, • Resilience – the ability of the system to transform, renew and recover in timely response to events, • Security – the ability of the system to remain protected against accidental or deliberate attacks.
Based on the main characteristics to achieve a trustworthy system which are also described in the British Standard BS 10754-1:2018 [18] and the approach suggested by [19], an i4Q Trustworthy System Framework is developed containing three main elements: • the Trustworthy Pillars from the defined key characteristics, • the 14Q Core Services – representing all i4Q solutions clustered in layers, • and the Environment containing all elements which have an impact on trustworthiness. 1 High-risk AI systems are designed in order to be exploited a safety component of certain products, or are themselves products, that are covered by the legal provisions set out in Annex II of the AI Act.
In regard to the procedure described in the British Standard BS 10754-1:2018 [18], Trustworthy Levels (TL) for all i4Q solutions are defined and requirements are elicited. These requirements are communicated to pilot providers, representing the end users of i4Q solutions, via surveys for rating of importance. In a next step all solution developers are asked to which degree their future solutions cover the requirements. Based on the results a Trustworthy Score for each solution is calculated for the current state of the solutions. Potential risks are identified and highlighted. This Trustworthy Score provides a KPI for future validation of achieved trustworthiness of i4Q solutions.
This paper presented a multi-dimensional benchmarking instrument, which was used in order to fulfill the development objectives of the i4Q solutions. Results showed that the technologies that meet the most criteria are: Data Analytics, Machine Learning, and Big Data. In order to compare the performance between the AsIs and the ToBe business processes, a KPIs dashboard was used, which consisted of metrics and measures to check the performance of the solutions. That way, industries determine the improvements that are expected in their business processes after the implementation of i4Q solutions.
In order to gather the necessary requirements from the industrial partners and the software developers, an iterative procedure was established, aligned with standards and guidelines described in Section 4 of this paper. Pilot requirements are elicited and listed along the pilot’s business processes and rated with their priority and difficulty. The system modeling language SysML [12] is used to model the mappings and structures. Function Structure Diagrams (FSD) [14] was also used for every i4Q solution to describe the input, data flow, functionalities and output of every solution.
Last but not least, the trustworthiness of the AI-based services available to EU citizens is guaranteed and the legislation is being previewed. Although the AI Act is yet to be finalized, adopted, and enforced, the proposed provisions are analyzed to elicit key requirements which would be the legal indicators of a trustworthy AI which could enter the EU digital single marker.
This work is funded by the European Union's Horizon 2020 research and innovation program under Grant Agreement No: 958205.
[1] i4Q, Homepage, 2021. URL: https://www.i4q-project.eu/ [2] ISO, ISO 22400-2:2014, Automation systems and integration - Key performance indicators (KPIs) for manufacturing operations management - Part 2: Definitions and descriptions, 2014.
URL: https://www.iso.org/standard/54497.html [3] ISO, ISO/IEC/ IEEE 29148:2018, Systems and software engineering - Life cycle processes
Requirements engineering, 2018. URL: https://www.iso.org/standard/72089.html [4] ISO, ISO/IEC/IEEE 12207:2017, Systems and software engineering - Software life cycle processes, 2017. URL: https://www.iso.org/standard/63712.html [5] ISO, ISO/IEC/IEEE 15288:2015, Systems and software engineering - System life cycle processes, 2015. URL: https://www.iso.org/standard/63711.html#:~:text=ISO%2FIEC%2FIEEE%2015288%3A2015%2 0establishes%20a%20common%20framework,hierarchy%20of%20a%20system's%20structure. [6] VDI, VDI 2221-1:2019-11, Design of technical products and systems, Part 1: Model of product design, 2019. URL: https://www.vdi.de/richtlinien/details/vdi-2221-blatt-1-design-of-technicalproducts-and-systems-model-of-product-design [7] VDI, VDI 2221-2:2019-11, Design of technical products and systems, Part 2: Configuration of individual product design processes, 2019. URL: https://www.vdi.de/richtlinien/details/vdi-2221blatt-2-design-of-technical-products-and-systems-configuration-of-individual-product-designprocesses [8] VDI, VDI 2206:2004-06, Design methodology for mechatronic systems, 2004. URL: https://www.beuth.de/de/technische-regel/vdi-2206/73296956 [9] VDI, VDI 2206:2020-09, Development of cyber-physical mechatronic systems (CPMS), 2020.
URL: https://www.beuth.de/de/technische-regel-entwurf/vdi-vde-2206/327616828 [10] MBSEworks.com, What is MBSE? – What You Need to Know, 2022. URL: https://mbseworks.com/ [11] INCOSE, INCOSE Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities, Wiley, New Jersey, 2015. [12] SysML.org, SysML Open Source Project – What is SysML? Who created SysML?, 2021. URL: https://sysml.org/ [13] R. Zafirov, Modellbildung und Spezifikation, in: M. Eigner, D. Roubanov, R. Zafirov (Eds.),
Modellbasierte Virtuelle Produktentwicklung, Springer, Berlin, 2014, pp. 77-96. [14] B. Bender, K. Gericke (Eds.), Pahl/Beitz Konstruktionslehre. Methoden und Anwendung erfolgreicher Produktentwicklung, Springer, Berlin, 2021. [15] European Commission, Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts, 2021. URL: https://eur- lex.europa.eu/legalcontent/EN/TXT/?uri=CELEX%3A52021PC0206 [16] European Digital SME Alliance, European AI Act – How will this regulation affect SMEs?, 2021. URL: https://www.digitalsme.eu/european-ai-act-how-will-this-regulation-affect- smes/ [17] E. Gaumond, Artificial Intelligence Act: What Is The European Approach For AI?, 2021. URL: https://www.lawfareblog.com/artificial-intelligence-act-what-european- approach-ai [18] British Standards Institution, BS 10754-1:2018, Information technology - Systems trustworthiness - Part 1: Governance and management specification, 2018. URL: https://www.en-standard.eu/bs-10754-1-2018-information-technology-systems-trustworthinessgovernance-and-management-specification/ [19] R. P. J. C. Bose, K. Singi, V. Kaulgud, K. K. Phokela, S. Podder, Framework for Trustworthy Software Development, in: 34th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW), IEEE, New York, 2019, pp. 45–48. doi: 10.1109/ASEW.2019.0002.