This paper deals with the standard on data quality for artificial intelligence (AI) with the extension of the concept of data to datasets. It presents the evolution from SQuaRE born in traditional environment towards AI considering the new characteristics of datasets essential for machine learning (ML) and AI applications.
The document aims to promote a common language and a culture of data quality. AI systems, in particular machine learning, require data, knowledge and practical experiences to be effective. Systems can be aided by the provision of high-quality data and datasets, including algorithms to process them.
The emergence of AI systems has necessitated a broad reassessment of data quality. This paper discusses the growing importance of data quality when considering AI systems, highlighting the definitions provided in ISO/IEC 5259 series and several data-related standards. The management involved can adopt quality models for data and provide new and adequate processes, in order to guarantee the expected results.
This paper discusses data quality standards in artificial intelligence (AI) systems. Building on traditional data quality models such as ISO/IEC 25012 and ISO/IEC 25024, it proposes how these standards are evolving to meet also the unique demands of AI and machine learning (ML) applications.
Since 2008, the most complete standard on data quality
product is considered the ISO/IEC 25012 “Data quality
model” [
For this reason, ISO/IEC 25012, and ISO/IEC 25024,
developed by SC7/WG6, have been imported into
ISO/IEC 5259-2 “Data quality measures” [
Why do we talking about data quality standards? Because standards play an important role in offering better products, harmonizing terms and processes, facilitating communication, enabling synergies, increasing interoperability of systems and data exchange, with common semantics and with shared metadata, contributing to the reduction of costs and incidents.
It must be considered that data is not a digital artefact, as often happens with software, but above all represents the fact itself and the criterion of truth for each of its aspects. Data is therefore reality, the interpretable representation of what exists. They are the elements on which decisions, improvements and developments can be based. Data, which follows standard rules, are not a convention, but provide information and knowledge of reality. To describe the concept of data quality, standard 25012 defined 15 characteristics, inherent and system dependent. Standard 25024 has defined 63 measures to support the quantification of the level of quality achieved.
Thus, it is happening that AI perspective strengthens data and data quality putting their features at the center of interest, but also requiring the extension of quality and measures to datasets, adding 9 characteristics and some measures.
For AI systems, the qualitative aspects listed below are encounter with new qualitative characteristics: representativeness (with respect to the population to which the AI is intended, refer without discrimination), identifiability or de-identifiability, balance, similarity, diversity, effectiveness, relevance, timeliness, auditability, as defined in 5259-2 for machine learning. The various aspects mentioned highlight the need for specific legislative governance, processes, coordination of products and technical approaches that include training, management preparation, new processes and dissemination of product quality principles for the end user. Data governance will need to ensure, among other things, transparency, stakeholder collaboration and accountability in the different areas of data management.
Progress must be made quickly due to the pervasive impact of AI, considering data and information as an aspect of knowledge, to be explored and recognized from different points of view: a) Constraints : laws, legal and technical quality requirements, human rights, ethics, quality models, methodologies, governance, social impacts, monitoring quality in use; b) Development: management of systems that process input and output data , assessment of data and software quality, taking into account cybersecurity, risk classes and automation; c) Technologies: cloud, big data, quantum systems, neural networks, language management, generative models, machine learning , robots, devices, IOT, sensors, hardware, etc..
For each of these points of view, it is reiterated that data, information and knowledge are essential topics to achieve trustworthiness in the new technology.
•
•
•
To better understand the scope of the models developed
by SC7/WG6 in ISO/IEC 25000 family [
The fifteen data quality characteristics defined are classified in the following: - Inherent: accuracy, completeness, consistency, credibility, currentness; - Inherent and system dependent: accessibility, compliance, confidentiality, efficiency, precision, traceability, understandability; - System dependent: availability, portability, recoverability.
ISO/IEC 25024 specifies 63 measures to quantify the data quality level in relation to characteristics. 5259-2 describes 42 measures imported from 25024, some adopted as is, some adapted for artificial intelligence. 3.2. Datasets quality The new quality characteristics od datasets defined in 5259-2 can be divided according to the following point of view: Technical: Auditability Legal: Identifiability Reality oriented: Balance, Diversity, Effectiveness, Relevance, Representativenes, Similarity, Timeliness In addition, the aspects of provenance and randomness are considered in 5259-2 as examples of essential factors for preparation of datasets to reinforce the credibility characteristic.
Regarding the quality characteristics of datasets in 52592, measures to quantify quality are also reported. Names of the measures described in the standard for each characteristics are related to: audited records, resolution balance, balance of images, label proportion balance, category size diversity, category size effectiveness, label effectiveness, identifiability ratio, relevance data in a record, representativeness ratio, sample similarity, timeliness of data items.
Regarding the aspects of balance, representativeness and
risks of bias (unwanted discrimination of datasets), other
examples are also available from literature [
Measurement is a set of operations having the object of determining a value of a measure. The most relevant measures in the contest of data quality are related to the level of quality achieved (threshold) both for individual data and for the group of data in a dataset.
The adoption of the data quality model is essential for verification and evaluation. When following some specific steps to design, control and manage data quality, develop and use AI systems, it is necessary to take into account a detailed data life cycle for different purposes (e.g. data governance, data acquisition, format normalization, data bias, sources integration, storage, maintenance, dismissioning).
Therefore, entire data quality models can be considered an analytical guide for data preparation and not only a useful instrument for evaluation and assessment of a single product.
While data quality concepts and models currently follow a predominantly deterministic orientation in 5259-2, functional measures to quantify datasets quality levels can be extended in the future also to non-deterministic approaches introducing statistical probability methods.
Data quality models can be considered a fundamental topic for AI that links new technology to various existing international standards. For a general operational view we should also consider other standards to be added to those mentioned, such as at least the following, wihich mention SQuaRE standards and data models in the text and mentioned in the bibliography: ISO/IEC 42001:2023 on AI management system, ISO/IEC 25059:2023 on quality model for AI system, ISO/IEC TS 12791:2024 on treatment of unwanted bias, ISO/IEC TR 24027:2021 on bias in AI systems.
ISO/IEC 42001 specifies requirements and provides
guidance for establishing, implementing, maintaining
and continually improving an AI management system in
the context of an organization. This document is
intended for use by an organization that provides or uses
products or services that use AI systems. The document
is intended to help the organization develop, provide or
use AI systems responsibly in pursuing its objectives
and meeting applicable requirements, stakeholder
obligations related and expectations araising from them,
paying attention to data quality in the Annex B [
ISO/IEC TR 24027 addresses bias in relation to AI
systems, particularly with respect to AI-aided
decisionmaking. Measurement techniques and methods for
assessing bias are described, with the aim to address and
treat bias-related vulnerabilities. All phases of AI system
lifecycle are in the scope, including, but not limited to,
data collection, training, continual learning, design,
testing, evaluation and implementation [
A global view of data and datasets quality is described in
the technical report CEN CLC TR 18115 “Data
Governance and quality for AI within the European
context” [
The report is consistent with the EU AI Act
Regulation[
The title of this paper is just a tip of the iceberg. The topic of data and the whole AI is huge. It is evolving rapidly. The adoption of quality models for data and datasets and datasets and the measurement activities are important for a complete quality assurance avoiding bias and discrimination, supporting useful AI for an efficient decision support system, avoiding bias and discrimination. In addition, it must be considered as a complement to quality of software to manage the data, including features that are also important such as safety and transparency. The propagation of SQuaRE quality models in many standards confirms the value of 25000 family. Likewise, the adoption of a managed quality system in the organization developing AI systems is becoming mandatory, paving the way for conformity assessment and quality certification. It is important to underline the consistency of standards and compliance with the law, improving the efficiency of AI systems production, respecting ethical aspects, human centric vision, inclusiveness and confiming trustworthiness.
ISO/IEC 25012:2008 Software engineering Software product quality Requirements and Evaluation (SQuaRE) - Data quality model ISO/IEC 25024:2015 Systems and software engineering – Systems and software quality