ISO/IEC 25000 quality measures for A.I.: a geometrical approach Andrea Trenta UNINFO UNI CT 504 Technical Committee System and Software Engineering Italy andrea.trenta@dataqualitylab.it Abstract— In a previous paper [13] we discussed ISO/IEC 2. Transform image i (i=1,..M) in a (n2x1) column 25000 application when new quality measures are defined. In vector Γi {Γ1, Γ2,… ΓM}; the present paper: 1 - some quality issues in A.I. are identified, 3. Compute the “average face” Ψ = ∑𝑀 1 Γ𝑖 𝑀 - then known solutions are recalled and - new quality measures for A.I. are proposed and subtract Ψ to each image and obtain new vectors {Φ1, Φ2,… ΦM} Keywords: data quality, eigenvalue, A.I., AI, image 4. Build the matrix (n2xM) A = [Φ1, Φ2,… ΦM] and recognition, training, ISO, ISO/IEC 25024 1 𝑇 compute the covariance matrix (n2xn2) C = ∑𝑀 1 Φ𝑖 Φ𝑖 = 𝑀 1 I. INTRODUCTION 𝑀 AAT In this paper new ISO/IEC 25000 quality measures for 5. Compute M eigenvalues λi (i=1,..M) of matrix ATA dataset used in some A.I. applications are proposed based and then eigenvectors of AAT on [7] and [12]. Furthermore, some considerations are developed about the possible specification and extension of 6. Sort the eigenvalues of C in descending order the method to any kind of dataset. 7. Choose a number N of eigenvalues, starting from the II. Quality issues in A.I. biggest, in order to represent 95% of their sum η and keep them; the other n2-N eigenvalues will not be considered In this paper, the term A.I. is used for simplicity even when referring to Machine Learning. 8. Calculate the images dataset as a linear combination of the N eigenvectors defined at step 7 Further steps are defined [7],[12],[13] for face recognition, that is out of the scope of this paper Figure 1 Definitions [14] Firstly, we consider the A.I. application face recognition, well-known both for the solutions and for the open issues point of view. Among the open issues there is how to understand whether the training dataset is “optimal”. To this end, we will explore the measure of completeness characteristic [6] of a set of images supposed to be a training dataset. Note that the measures proposed neither correspond to the measure of the whole A.I. system output results nor to its behavior observation, as it is a purely static measure of the input, although it could be used together with other ones, to evaluate the overall system quality. The basis of our analysis is the calculation of “eigenfaces” [7], according Karhunen-Loève transformation (PCA) with the following steps: Figure 2 MR2 Face database [11] 1. Collect M images with nxn grayscale pixels of faces with similar dimension, light condition, shot, etc. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). III. PROPOSAL (e.g. we expect different M values for rotated or else non- Intuitively, if we want to measure the completeness of 1 rotated images, for personal names or else tags,…); so, in an image dataset, we try to answer questions like: general, having M>N does not mean that the space is complete, in other words, that every image can be a. how many similar images are there represented, e.g. it could be a new M+1th face that cannot b. how strong is the similarity of some images be an acceptable4 linear combination of existing N images (e.g. a bald face is missing from the dataset of fig.2). If in Here the proposed measures for (a) and (b): this case the new M+1th image is added to the training dataset, that corresponds to an “enforced learning”. A. as some dimensions are eliminated in step 7 Therefore, some distinction should be made between a above 2 , we can measure N\M “PCA space dimension machine with a “unsupervised learning” and a machine with against dataset space dimension” “reinforced learning” when evaluating measurements values (A) and (B) over a training dataset. B. the more a dataset is orthogonal, the less its images As bias is critical for the learning dataset quality, the are similar to each other; as a measure of it, we can measures (A) and (B) are suggested also for bias 5 consider the product of N eigenvalues λ1*λ2*...*λN, that is measurement, when defining bias as the modification of an also the “Determinant of reduced eigenvalues matrix”, ideal fully orthogonal and normalized dataset 6. that in turn is the volume of the hyper-parallelepiped that this matrix represents. V. CONCLUSION The measures (A) and (B) appear belonging to data To sum up, with this proposal we reframe the issue of quality completeness characteristic [6]. Applying the finding an effective data quality (for completeness) process described in [8], the measures (A) and (B) can be measurement function into a well-known geometrical defined as ISO 25000 conforming measures; they can be calculation. considered in SC7 WG6 and SC42 work in progress on A.I., too. IV. FURTHER STUDIES The steps 1-8 above were proven to be effective in face REFERENCES recognition and are potentially applicable to any dataset. To [1] ISO/IEC 25010:2011 Systems and Software engineering - Systems do this, the vectorization step 2 above shall be applied to and software Quality Requirements and Evaluation (SQuaRE) - any attribute(s) of the dataset: as images were vectorized System and software quality models pixel by pixel3, similar operation could be performed e.g. [2] ISO/IEC 25012:2008 Systems and Software engineering - Systems for char strings, taking into account possible different and software Quality Requirements and Evaluation (SQuaRE) - lengths that require a further step of normalization. Data quality model [3] ISO/IEC 25020:2019, Systems and Software engineering - Systems Further studies are needed to apply the method also to and software Quality Requirements and Evaluation (SQuaRE) - images rotated and translated, that is the most frequent case Quality measurement framework. in A.I. applications (fig.3). [4] ISO/IEC 25022:2016, Systems and Software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Measurement of quality in use. [5] ISO/IEC 25023:2016, Systems and Software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Measurement of system and software product quality. [6] ISO/IEC 25024:2015, Systems and Software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Measurement of data quality [7] Turk, M.A., Pentland, A.P.: Face recognition using eigenfaces. In: Computer Vision and Pattern Recognition, 1991. Proceedings CVPR'91. IEEE Computer Society Conference on. pp. 586-591. IEEE (1991) [8] D. Natale, A. Trenta: Examples of practical use of ISO/IEC 25000 Proceedings APSEC IWESQ 2019 (CEUR-WS.org, ISSN 1613- Figura 3 Dataset trial MPEG-CDVA (Compact Descriptor for 0073) Video Analysis)– [9] [10] [9] Compact Descriptors for Video Analysis: the Emerging MPEG For the case of a “unsupervised learning”, care should be Standard 2017 Ling-Yu Duan, Vijay Chandrasekhar, Shiqi Wang, taken in generating the appropriate (i.e. minimum) number Yihang Lou, Jie Lin, Yan Bai, Tiejun Huang, Alex Chichung Kot, Fellow, IEEE and Wen Gao, Fellow, IEEE of M images, as M appears depending on the kind of dataset 1 4 It is intended the data quality characteristic “completeness” i.e. the projection of the M+1th image in the space of faces has see [2], [6] an euclidean distance with respect to the other faces below a 2 threshold; in the case the M+1th image is a copy, there is one N