Metrics for Identifying Bias in Datasets

Abstract

Nowadays automated decision-making systems are pervasively used and more often, they are used for taking important decisions in sensitive areas such as the granting of a bank overdraft, the susceptibility of an individual to a virus infection, or even the likelihood of repeating a crime. The widespread use of these systems raises a growing ethical concern about the risk of a potential discriminatory impact. In particular, machine-learning systems trained on unbalanced data could rise to systematic discriminations in the real world. One of the most important challenges is to determine metrics capable of detecting when an unbalanced training dataset may lead to discriminatory behaviour of the model built on it. In this paper, we propose an approach based on the notion of data completeness using two diferent metrics: one based on the combinations of the values of the dataset, which will be our benchmark, and the second using frame theory, widely used among others for quality measures of control systems. It is important to remark that the use of metrics cannot be a substitute for a broader design that must take into account the columns that could lead to the presence of bias in the data. The line of research does not end with these activities but aims to continue the path towards a standardised register of measures.

Keywords

• eol>data quality
• iso 25000
• bias
• Artificial Intelligence
• AI
• machine learning
• ML
• standard
• square

1. Introduction

ing data [14][18][19]. The idea of the present study is to verify the balance of the data even before the onset Recently, machine learning has become more and more of discriminatory problems by the prediction algorithm popular in diferent sectors [ 1 ][ 2 ][ 3 ][ 4 ][ 5 ], thanks to both through an index that measures the degree of balance of the possibility of having large amounts of data avail- the data. At the same time the proposed metric can be able [ 6 ][ 7 ] and the possibility of increasingly eficient considered within the framework of the measures of the implementations [ 8 ][ 9 ][ 10 ][ 11 ]. ISO 25000 series of standards [20][21]. In the following,

The world around us is gradually being enriched with reference will be made to the data quality “characterisfunctionalities that seem “normal” to us, but which are tics” (e.g. completeness, accuracy) defined in the ISO/IEC actually the result of sophisticated learning algorithms 25024 standard [22]. This article will also use the termibelonging to the field of AI [ 12 ]. This may seem rational nology of relational algebra: the term column identifies and neutral, but sometimes, it can lead to harmful situa- the field or attribute of the dataset (intensional level), tions such as discrimination [ 13 ] or to worse predictions while the term tuple or row identifies an instance of the than what could be achieved with balanced data [14]. Eq- dataset (extensional level). uity and discrimination risks arise mainly due to disproportionate data sets: learning algorithms build the model from training data, so such a disproportion can lead to 2. The proposed methodology conclusions that are out of line with reality [15][16][17].

On the other hand, in some situations it is dificult to Let us consider a dataset containing distinct instances of have homogeneous and proportional data because if the an entity, in one-to-many relation with other entities. The population is unbalanced, the sample used for learning classic example is the entity Person that can belong to difwill also be unbalanced too. In literature there are dif- ferent categories or classes describing gender, ethnicity, ferent techniques that allow to fill the dataset with miss- income bracket and religion (Fig. 1).

A simple, but sometimes ineficient, way of realising this concept is through the introduction of as many columns as there are concepts linked by the set relation since it is a one-to-many relation. In a relational database, these relationships are realised through foreign keys or columns with discrete domain controlled through row constraints. For the problem at hand, some of these columns are special, i.e. they may contain an identifier (primary key), a protected attribute (e.g., gender or relibelong to the same categorising classes constitutes a constraint for all the other instances of the dataset. That is, one must ensure that one has the same number of replicas of distinct class combinations for distinct instances.

Instead, in the second case it is suficient to have at least one combination of distinct classes among those possible for each instance. For simplicity, but without loss of generality of the procedure, we will explore the minimum completeness of the dataset, then we will reduce the dataset to just the columns () by removing duplicate rows. We will use the Python language to explicate the calculation formulas and make the mathematical logic implied less abstract. The Python language has the pandas library, which makes it possible to carry out analysis and data manipulation in a fast, powerful, flexible and easy-to-use manner. Through the DataFrame class it is possible to load data frames from a simple csv file: We will use two diferent metrics to measure the degree of completeness: one is based on combinatorial calculus and the other on frame theory. gion, or any other personal characteristic that should not be object of discrimination [23]), a target variable (usually denoted by Y) and an output of a prediction (usually denoted by R) if it has already been computed. However, the approach proposed is agnostic with respect to this classification because for our purposes we are interested in evaluating those columns that assume values in finite and discrete intervals, which we will call categorical with respect to the row data. This characteristic will allow us to consider the set of their values as the digits constituting a number in a variable base numbering system. 2.1. Combinatorial metric The idea of the present study is based on the principle that a learning system provides predictions consistent with the data with which it has been trained. Therefore, if it is fed with non-homogeneous data it will provide unbalanced and discriminatory predictions with respect to reality. For this reason, the methodology we propose starts with the analysis phase of the reality of interest and of the dataset, an activity that must be carried out even before starting the pre-training phase in line with previous studies where bias measures in automated decision making systems were proposed [24][25][ 26 ][22]. In particular, during this phase, it is necessary to identify all the independent columns that define whether the instance belongs to a class or category. Suppose we have a structured dataset as follows: The ideal value of minimum completeness for the combinatorial metric is when in the dataset there is at least one instance that belongs to each distinct combination of categories. The absence of some combination could create the lack of information that we do not want to exist.

To calculate the total number of distinct combinations we need to calculate the product of the distinct replicas per single category (()).

k= l e n ( d f [ ' CS0 ' ] . u n i q u e ( ) ∗ l e n ( d f [ ' CS1 ' ] . u n i q u e ( ) ∗ . . . ∗ l e n ( d f [ ' CSm−1 ' ] . u n i q u e ( ) = {0, 1, ..., − 1}

(1) Indicating with the set S the positions of the columns categorising the instances, functionally independent of the other columns in the dataset: ⊆ { 0, 1, ..., − 1}, () = , ≤

(2) we can analyze the new dataset consisting of the columns () with ∈ [0, − 1]. Having said that, we can decide to use two diferent notions of completeness: maximum or minimum. In the first case the presence in the dataset of a greater number of distinct instances that

On the other hand, in the dataset we only have the characterising columns so we can derive the true number of distinct instances in order to determine how far the data in our possession deviates from the ideal case.

l e n ( d f . d r o p _ d u p l i c a t e s ( ) ) / ( l e n ( d f [ ' CS0 ' ] . u n i q u e ( ) ∗ l e n ( d f [ ' CS1 ' ] . u n i q u e ( ) ∗ . . . ∗ l e n ( d f [ ' CSm−1 ' ] . u n i q u e ( ) ) The value for maximum completeness is calculated from the maximum number of duplicates of the same combinations of characterising columns. For this reason it is necessary to maintain in the dataset in addition to the columns () a discriminating identification field of the rows with the same values in these columns. To determine the potential total, once the maximum number of M= d f . g r o u p b y ( [ ' CS0 ' , . . . , ' CSm−1 ' ] ) .

2.2. Metric based on frame theory

as a finite collection of vectors What we have been described till now is just a combinatorial calculations exercise, however we would like to have a more sophisticated calculation model. For this reason, we have found in Frame theory, which finds its natural application in the field of signals and control systems theory, a promising calculation system [ 27 ][ 28 ]. In a ifnite-dimensional Hilbert space, of dimension and with an inner product ⟨· , ·⟩ , a frame of is defined

if there exist two constants and Θ , with 0 < ⩽ Θ , such that:

()∈ ⊂ ‖‖ ⩽ ∑︁ |⟨, ⟩ |2 ⩽ Θ ‖‖

(3) A frame is said to be tight if = Θ . So we can say that, a frame is tight when its vectors are as spread out in space as possible. A notable result that we will exploit in the following is that whenever a sequence ()∈ ⊂ of vectors constitute a frame, the constants and Θ are the smallest and the largest eigenvalues respectively of the corresponding frame operator. Because the frame operator is self adjoint, a frame is tight if and only if its frame operator is an appropriate multiple of the identity operator. If we consider now as row vectors the component tuples of the original dataset, reduced to the columns (), we represent the new dataset with the matrix ( × ): = ⎢ . ⎥ ⎡1⎤ ⎢2⎥ ⎢⎣ .. ⎥⎦

Before calculating the Gramnian matrix, the values must be centred with respect to the mean row, in the following way: = − ⎡1⎤

1 ⎢ ⎥ ⎢⎢ .. ⎥⎥ · =1 ⎣ . ⎦

1 ∑︁ 1 The Gramian matrix is obtained by performing the matrix product:

= · duplications (M) has been determined, it is necessary to At this point, we can calculate the eigenvalues, which extend this multiplication factor to all other classes.

represent for each column a measure of variance of the (4) (5) (6) values within that column.

Λ = ( 1, ..., ) With reference to the diferent categories of frames [ 29 ], we recall that for some of them it is possible to expect results on the values and multiplicity of the eigenvalues [ 30 ][ 31 ]. In the following we derive considerations about the tightness of the frame by analysing the distribution of the eigenvalues. The ideal case is when the frame is tight and the eigenvectors are uniformly distributed with respect to the vector space. In this case all the eigenvalues assume the same value, and an easy way to check this is when the diference between the maximum and minimum eigenvalues is zero: ∆ =

(Λ) − (Λ) = 0 However, this index cannot be adopted since it does not have a predefined range of value and it would not allow comparisons between datasets. In this regard several measures of homogeneity and balance of a distribution have been proposed in literature (e.g., Shannon, Gini-Simpson, Theil, etc.). In particular we will use the Gini-Simpson index for categorical data which we will normalise to obtain values in the range [ 0, 1 ]: columns. Moreover, since the method is based on the position of the eigenvectors in the vector space, it is sensitive with respect to their Euclidean distance. Distance depending on dataset values. Therefore, in order to apply the methodology correctly, the following algorithm should be followed: 1. analyse the reality of interest and the dataset by selecting the columns that classify the rows and that are independent of each other (()); 2. mapping the domain of column values that classify the rows into an ordinal set of natural numbers N by means of a transformation function that matches each value in the domain (numeric, alphanumeric or null) to a natural progressive number while preserving the ordering of the values; 3. construct the matrix from by subtracting given to the Python interpreter. This heuristic approach the mean vector; allow us to understand the limitations of the adopted 4. calculate the matrix and its eigenvalues; metrics and their strengths. To make the case more real5. calculate the heterogeneity index of the eigenval- istic we used a dataset with 5 categorising columns on a ues; domain based on 5 discrete values (0..4), so = 5 and 6. analyse the results obtained and return to the ini- = 5. The categorising columns in our simulation tial point if any error situations arise (dependency are , , , and (Table 1).

between columns, incorrect mapping,...).

The Fig. 2 shows the two diferent algorithms needed to calculate the metrics described in this article.

3. Case Study

3.1. Designing

Due to the high sensitivity of the frame-based method to domain values, we preferred to test its efectiveness using a dataset with synthetic data. This allowed us to start from an ideal model. Knowing in advance the presence of bias in the dataset will allow us to verify the goodness of performance of the two metrics, without having the influence of any side efects. We will use a dataset with categorising columns and assume that we have the same number of values in the categories that is to say . Computing all possible arrangements with repetitions is a simple task [ 32 ]. In fact, it is suficient to compose all the numbers ranging from 0 to − 1 expressing them in terms of digits of the base . In our study, we used a spreadsheet to automatically and easily create the various datasets, data sources that were then

With this assumption the distinct rows of the dataset are = 55 = 3,125. The methodology can be replicated with any value of and . The method remains valid also in the more general case in which the columns belong to categories with diferent range. In fact, in the latter case it will be suficient to adopt a variable base numbering system ( ). In the case study we will consider the case of minimum completeness, since maximum completeness can be easily extended by the latter.

The matrix corresponds to the rows in Table 1:

The idea is to start from the complete schema by introducing, step by step, distortion efects to verify the behaviour of the two metrics. The combinatorial metric can be applied directly to the set of columns reduced from the original dataset without any transformation (Fig.5 2). Indeed, the advantage of this method is that it acts directly on the values regardless of their nature (datatype). The first index can be calculated using the following instructions: l e n ( d f . d r o p _ d u p l i c a t e s ( ) ) / ( l e n ( d f [ ' A ' ] . u n i q u e ( ) ∗ l e n ( d f [ ' B ' ] . u n i q u e ( ) ∗ An index closer to unity means a better (minimum) completeness of the data set, while more it tends towards zero, greater is the incompleteness of the data An index closer to unity means a better (minimum) completeness of the data set, while more it tends towards zero, greater is the incompleteness of the data. The second metric requires the Python library numpy.linag to calculate the eigenvalues of the Gramnian matrix. Once the eigenvalues have been calculated, the Gini-Simpson index can be determined: from numpy import l i n a l g a s LA d f = ( d f . d r o p _ d u p l i c a t e s ( ) ) d f = d f . s u b t r a c t ( d f . mean ( ) ) G=np . d o t ( d f . T , d f ) w , v = LA . e i g ( G ) # G i n i − S i m p s o n I n d e x (w . s i z e / ( w . s i z e − 1 ) ) ∗

( 1 − ( (w/w . sum ( ) ) ∗ ∗ 2 ) . sum ( ) )

3.2. Results

3.2.1. Case 1: no bias With reference to Table 1, the presence of all possible digits in the 5 positions guarantees that there is no combination of the categories that is not represented, so the data set contains exhaustively all the allowed combinations [ 33 ]. It is interesting to verify that the value of the two metrics does not change with respect to the position of the rows and columns, so that a random shufle of them does not alter the final value of the two metrics.

This demonstrates the invariance of the two metrics with respect to the position of the rows and columns. Indeed, we cannot make any assumptions in advance about the sorting (horizontal or vertical) of the data. 3. deleting a large portion of the values in a column: e.g. the first 615 rows out of 625 that have in column the value zero.

As we can see from the results obtained (Table 2) it might be surprising that the combinatorial metric in case 1) returns the ideal value one: because the calculation is based on the values present in the dataset thus the result is that the domain is complete. The question is whether the bias measure should take into account all possible values of the categorising domains or not. If we want to extend the data domain to the all the possible (but not present) values we should increase the denominator of the index formula and the expected value should be 0.80.

In cases 2) and 3) the combinatorial metric is consistent with the level of bias in the data: low bias corresponds to a value close to the ideal, while higher bias corresponds to a greater deviation from the optimum. The metric using the Gini-Simpson index applied to the eigenvalues of the W matrix identifies case 2) as the best situation, which represents an almost complete domain. While case 1) is correctly identified as the one with the highest bias in the data.

In this case the combinatorial index will change from 1 First, we are interested to know what will be the be- to 0.2 because the contribution of a column is reduced haviour of the two metrics if a subinterval of values is and the interval is reduced by one fifth ( 3,125/5 = 625) missing in the data set. This circumstance could lead too. The metric based on the Gini-Simpson index applied to discrimination in a category. We will consider three to the eigenvalues is not afected by functional dependendiferent situations in which the rows of Table 1 are re- cies since the eigenvalues are the same. Therefore, the duced: index is equal to unity. We observe the same behaviour of the two metrics if the functional dependence is between several columns, such as the case where depends on the other columns ( → , , , ) according to the relation: = ( + + + ) 5

(13) 3.2.4. Case 4: diferent ordinal scales This paragraph demonstrates the necessity of the mapping phase (Fig. 2) in order to use correctly frame theory.

In fact, since the method is based on the concept of Euclidean distance, it requires integer, homogeneous and comparable values. This is not directly applicable to real datasets since it is not possible to make a priori assumptions about the data domain of the columns. Having said that, we will study the behaviour of the two metrics in three diferent random transformations of the initial dataset (Table 1): 2. the same transformation 1 applied to a single

column; 3. a diferent transformation ( , , , , ) per individual column (Table 3).

people pose important ethical and social challenges. It is important to be able to anticipate the possible emergence of discriminatory behaviour which, in certain situations, 1. an uniform transformation 1 applied to all could even violate fundamental human rights. For this columns that transforms discrete values into reason, we propose to integrate the common data curaother discrete values that are not necessarily adja- tion pipeline in ML with a preliminary analysis of intrincent and have a diferent Euclidean distance from sic properties of the dataset that could anticipate bias their initial positions: problems downwards the chain. In this paper, building on top of previous work in measures of disproportions in ⎪⎧ 0 → 44 datasets, we propose a measurement process composed ⎪⎨⎪⎪ 1 → 11 of two complementary metrics: combinatorial and Gini 1 : 2 → 5 on eigenvalues. We show a case study and report on ⎪⎪⎪ 3 → 111 their strengths and limitations. These metrics open up ⎪⎩ 4 → 9999 new scenarios for possible future developments in the

mitigation of bias since the learning phase. The combinatorial metric is invariant with respect to the distance of the values since it counts the distinct shapes present in each column. So, in all the three cases the dataset is complete and has no bias (value one). Conversely, the metric based on frame theory is sensitive to distances so, except in the first case where the transformation acts consistently and uniformly on all values, in the others it detects a presumed bias (Table 4). This demonstrates the need to reposition the values on an equidistant ordinal scale in case the dataset does not have this feature in the category data.

4. Conclusion

The problem of biased outputs of machine learning systems in domains that impact the rights and freedom of

5. Acknowledgments

We would like to thank Prof. Marco Torchiano for the interesting discussion on the research articles and to the President of the Italian National Body UNI CT 504 Commission, Dr. Domenico Natale, for his foundational contributions in the field of data quality.

References

[1] S. A. Ebiaredoh-Mienye , E. Esenogho , T. G. Swart, Artificial neural network technique for improving prediction of credit card default: A stacked sparse autoencoder approach , International Journal of Electrical and Computer Engineering 11 ( 2021 ) 4392 - 4402 . doi: 10 .11591/ijece.v11i5. pp4392 - 4402 .

[2] G. Saranya , A. Pravin , A comprehensive study on disease risk predictions in machine learning , International Journal of Electrical and Computer Engineering 10 ( 2020 ) 4217 - 4225 . doi: 10 .11591/ ijece.v10i4. pp4217 - 4225 .

[3] A. Ramachandran , A. Ramesh , A. Sukhlecha , A. Pandey , A. Karuppiah , Machine learning algorithms for fall detection using kinematic and heart rate parameters-a comprehensive analysis , IAES International Journal of Artificial Intelligence 9 ( 2020 ) 772 - 780 . doi: 10 .11591/ijai.v9. i4 . pp772 - 780 .

[4] G. Capizzi , G. Lo Sciuto , C. Napoli , E. Tramon- cil of Europe, Directorate General of Democracy tana, A multithread nested neural network archi- ( 2018 ). URL: https://pure.uva.nl/ws/files/42473478/ tecture to model surface plasmon polaritons prop- 32226549.pdf . agation, Micromachines 7 ( 2016 ). doi: 10 .3390/ [14] A. Watson , Reducing ai bias with synthetic mi7070110 . data, 2020 . URL: https://towardsdatascience.com/

[5] G. Lo Sciuto , G. Capizzi, S. Coco , R. Shikler , Geomet- reducing - ai -bias-with-synthetic-data-7bddc39f290d. ric shape optimization of organic solar cells for efi- [15] C. O'Neil , Weapons of Math Destruction: How Big ciency enhancement by neural networks, Lecture Data Increases Inequality and Threatens DemocNotes in Mechanical Engineering ( 2017 ) 789 - 796 . racy (reprintedition ed.)., Broadway Books , New doi:10.1007/978-3- 319 -45781-9_ 79 . York, 2017 .

[6] A. Boukhalfa , N. Hmina , H. Chaoui , Parallel pro- [16] S. Barocas , A. D. Selbst , Big data's disparate impact, cessing using big data and machine learning tech- 2016 . URL: http://dx.doi.org/10.2139/ssrn.2477899. niques for intrusion detection , IAES International [17] J. Angwin , J. Larson , S. Mattu , L. Kirchner, Journal of Artificial Intelligence 9 ( 2020 ) 553 - 560 . ProPublica, Machine bias - there's software doi:10 .11591/ijai.v9. i3 . pp553 - 560 . used across the country to predict future

[7] N. Seman , N. Razmi , Machine learning-based criminals. and it's biased against blacks., technique for big data sentiments extraction , 2016 . URL: https://www.propublica.org/article/ IAES International Journal of Artificial Intelli- machine-bias-risk-assessments-in-criminal-sentencing . gence 9 ( 2020 ) 473 - 479 . doi: 10 .11591/ijai.v9. [18] G. Menardi , N. Torelli , Training and assessing clasi3 . pp473 - 479 . sification rules with imbalanced data, Data Mining

[8] G. C. Cardarilli , L. D. Nunzio , R. Fazzolari , and Knowledge Discovery 28 ( 2012 ) 92 - 122 . URL: M. Panella , M. Re , A. Rosato , S. Span , A paral- https://doi.org/10.1007/s10618-012-0295-5. doi:10. lel hardware implementation for 2-d hierarchical 1007/s10618-012-0295-5. clustering based on fuzzy logic , IEEE Transactions [19] G. Capizzi , C. Napoli , L. Paternò , An innovative on Circuits and Systems II: Express Briefs 68 ( 2021 ) hybrid neuro-wavelet method for reconstruction 1428-1432 . doi: 10 .1109/TCSII. 2020 . 3032660 . of missing data in astronomical photometric sur-

[9] G. De Magistris , S. Russo , P. Roma, J. Starczewski, veys, in: L. Rutkowski , M. Korytkowski , R. Scherer , C. Napoli , An explainable fake news detector based R . Tadeusiewicz, L. A. Zadeh , J. M. Zurada (Eds.), on named entity recognition and stance classifica- Artificial Intelligence and Soft Computing , Springer tion applied to covid-19, Information (Switzerland) Berlin Heidelberg, Berlin, Heidelberg, 2012 , pp. 21 - 13 ( 2022 ). doi: 10 .3390/info13030137. 29 .

[10] A. Simonetta , M. C. Paoletti , M. Muratore , [20] A. Trenta , ISO/IEC 25000 Quality Measures for A new approach for designing of computer A.I.: A Geometrical Approach , in: IWESQ@ APSEC, architectures using multi-value logic , Inter- 2020 . URL: http://ceur-ws. org/ Vol- 2800 /paper-05. national Journal on Advanced Science, Engi- pdf . neering and Information Technology 11 ( 2021 ) [21] A. Trenta , D. Natale , Examples of practical use of 1440-1446 . URL: http://ijaseit.insightsociety. ISO/IEC 25000, in: IWESQ@APSEC, 2019 . URL: org/index.php?option=com_content&view= http://ceur-ws. org/ Vol- 2545 /paper-02.pdf . article&id=9&Itemid=1&article_id=15778. [22] ISO/IEC 25024:2015, Systems and Software engidoi:10.18517/ijaseit.11.4.15778. neering - Systems and software Quality Require-

[11] R. Brociek , G. De Magistris , F. Cardia , F. Coppa, ments and Evaluation (SQuaRE) - Measurement S. Russo , Contagion prevention of covid- of data quality, 2015 . URL: https://www.iso.org/ 19 by means of touch detection for retail standard/35749.html . stores, volume 3092 , 2021 , p. 89 - 94 . URL: [23] Charter of fundamental rights of the european https://www .scopus.com/inward/record.uri? union, 2000 . URL: https://www.europarl.europa.eu/ eid= 2 - s2 . 0 - 85126138443 &partnerID= 40 &md5= charter/pdf/text_en.pdf . 043ee2df651270a6ac0434a39cef92f5, cited by: 0 . [24] M. Mecati , F. E. Cannavò , A. Vetrò , M. Torchi-

[12] S. Spanò , G. C. Cardarilli , L. Di Nunzio , R. Fazzo- ano, Identifying risks in datasets for autolari, D. Giardino, M. Matta , A. Nannarelli , M. Re, mated decision-making, in: International ConAn eficient hardware implementation of rein- ference on Electronic Government , Springer, 2020 , forcement learning: The q-learning algorithm , pp. 332 - 344 . doi:https://doi.org/10.1007/ IEEE Access 7 ( 2019 ) 186340 - 186351 . doi: 10 .1109/ 978-3- 030 -57599-1_ 25 . ACCESS . 2019 . 2961174 . [25] E. Beretta , A. Vetrò , B. Lepri , J. D. Martin , De-

[13] F. Z. Borgesius , et al., Discrimination, artificial in- tecting discriminatory risk through data annotatelligence, and algorithmic decision-making, Coun- tion based on bayesian inferences , in: ACM Conference on Fairness, Accountability, and Transparency (pp. 794 - 804 ), 2021 , pp. 794 - 804 . URL: https://doi.org/10.1145/3442188.3445940.

[26] E. Beretta , A. Vetrò , B. Lepri , J. D. Martin , Ethical and socially-aware data labels. in annual international symposium on information management and big data , in: Annual International Symposium on Information Management and Big Data (pp. 320 - 327 ). Springer, Cham., 2018 . URL: https://link.springer. com/chapter/10.1007% 2F978 - 3 - 030 -11680-4_ 30 .

[27] K. MishalAssifP. , M.R. Sherif , D. Chatterjee , Measure of quality of finite-dimensional linear systems: A frame-theoretic view , Syst. Control. Lett . 151 ( 2021 ) 104911 .

[28] O. Christensen , An Introduction to Frames and Riesz Bases, Birkhäuser/Springer, 2016 .

[29] J. Kovacevic , A. Chebira , An introduction to frames , Foundations and Trends in Signal Processing 2 ( 2008 ) 1 - 94 . doi: 10 .1561/2000000006.

[30] M. Fickus , D. G. Mixon , Tables of the existence of equiangular tight frames , 2016 . arXiv: 1504 . 00253 .

[31] E. Tsiligianni , L. Kondi , A. Katsaggelos , Approximate equiangular tight frames for compressed sensing and cdma applications , Eurasip Journal on Advances in Signal Processing 2017 ( 2017 ). doi:10.1186/s13634-017-0501-0 , publisher Copyright: © 2017 , The Author(s) . Copyright: Copyright 2017 Elsevier B.V. , All rights reserved .

[32] A. Simonetta , M. C. Paoletti , Designing digital circuits in multi-valued logic , International Journal on Advanced Science, Engineering and Information Technology 8 ( 2018 ) 1166 - 1172 . URL: http://ijaseit.insightsociety. org/index.php ?option=com_content&view= article&id=9&Itemid=1&article_id=5966 . doi: 10 .18517/ijaseit.8.4.5966.

[33] UNI/TS 11725: 2018 , System and software engineering - Guidelines for the measurement of data quality, 2018 . URL: http://store.uni.com/catalogo/ uni-ts-11725- 2018 .